You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "bzablocki (via GitHub)" <gi...@apache.org> on 2023/06/28 10:29:55 UTC
[GitHub] [beam] bzablocki opened a new pull request, #27284: Yaml API: Day Zero tutorial notebook
bzablocki opened a new pull request, #27284:
URL: https://github.com/apache/beam/pull/27284
I've created the first version of Getting Started with Yaml notebooks.
The target group are users who don't have a lot of experience with coding and/or beam; and want to quickly get started with writing pipelines in beam.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amotley commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "amotley (via GitHub)" <gi...@apache.org>.
amotley commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1247125196
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
Review Comment:
I vaguely recall there are some Beam development resources that talk about environment setup, you can check there to see if there's any sections on installing apache-beam
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1611165187
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1247769458
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input file into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "Let’s see how our data looks like."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## First pipeline\n",
+ "We’ll start with creating a pipeline which loads the data, filters out valid messages leaving spam, and saves only valid lines to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " input: SmsData\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " input: SplitLine\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: KeepSpam\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "In this example, each transformation contains the 'input' key, but if the pipeline is linear, such as ours, we can let the inputs be implicit by designating the pipeline as a `chain` type.\n"
Review Comment:
Ok, I'll do that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amotley commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "amotley (via GitHub)" <gi...@apache.org>.
amotley commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1300289009
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,582 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK. You can use any text editor.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and downloads some text files from Cloud Storage to your local file system. We'll use these text files as input to the pipelines."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('wget -O data/kinglear.txt https://storage.googleapis.com/dataflow-samples/shakespeare/kinglear.txt')\n",
+ "run('wget -O data/SMSSpamCollection.csv https://storage.googleapis.com/apache-beam-samples/SMSSpamCollection/SMSSpamCollection')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with two datasets. We'll use `kinglear.txt` for the first example, and `SMSSpamCollection.csv` for the second and third examples.\n",
+ "Let's first take a look at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file that contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file that contains 5,574 rows of SMS messages labeled either spam or not-spam (\"ham\"). Each row contains two columns separated by a tab character:\n",
+ "1. `Column 1`: The label, either `ham` or `spam`\n",
+ "2. `Column 2`: The SMS message as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: Word count\n",
+ "This example is a version of the [WordCount](https://beam.apache.org/get-started/wordcount-example/)). It reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
Review Comment:
extra paren
```suggestion
"This example is a version of the [WordCount](https://beam.apache.org/get-started/wordcount-example/). It reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1432949967
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,680 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs the elements for debugging purposes.\n",
+ "\n",
+ "The `LogForTesting` transform lets us log the data when developing a pipeline. Remember, it is not advised to use this transform in production.\n",
+ "\n",
+ "Let's define the pipeline and save it to a file:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We can verify the contents of this file by running:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "! cat pipelines/pipeline-01.yaml"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Now, we can execute the yaml pipeline by passing this file as an argument to the following command:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here we use Python and `apache_beam` package to execute the pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for that in the future.\n",
+ "\n",
+ "If you scroll through the output logs, you'll find entries such as:\n",
+ "```\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=1, firstname='Reeba', age=58, country='Belgium', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=2, firstname='Maud', age=45, country='Spain', profession='firefighter')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=3, firstname='Meg', age=11, country='France', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=4, firstname='Rani', age=53, country='Spain', profession='doctor')\n",
+ "```\n",
+ "This is a representation of records from the input dataset."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's add a transform - `Filter`. To use this transform you need to specify the 'keep' condition and a language your condition is written in. Below you'll find an example with a condition written in Python.\n",
+ "This pipeline will filter out records containing people that are younger than 18 years old. The only records left to the next transform will be records representing adults. Verify this by scrolling to the bottom of the output logs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Similarly, we can create a condition in other languages, for example SQL. In this example we filter out people that are younger than 18 and have a profession other than being 'unemployed'."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: sql\n",
+ " keep: \"age >= 18 or (age < 18 and profession = 'unemployed')\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Another useful transform is `MapToFields`. This transform lets us manipulate fields of a record. For example, we can add a field to our records, which is a boolean field specifying if the person is adult or not."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam will try to infer the types involved in the mappings, but sometimes this is not possible. In these cases we can explicitly denote the expected output type, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult:\n",
+ " expression: \"age >= 18\"\n",
+ " output_type: boolean\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "When the `append` field is specified, one can `drop` fields as well, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " drop:\n",
+ " - age\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-03.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-03.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can also create simple UDFs (User Defined Functions) using Python or other languages. In the example below we add a field `random_number` which value is a random number not bigger than the age of the person."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " random_number:\n",
+ " callable: |\n",
+ " import random\n",
+ " def my_mapping(row):\n",
+ " return random.randrange(row.age)\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-04.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-04.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam YAML has EXPERIMENTAL ability to do aggregations to group and combine values across records. The is accomplished via the `Combine` transform type. Currently `Combine` needs to be in the `yaml_experimental_features` option (see the bottom of the pipeline) to use this transform.\n",
+ "\n",
+ "In this example we'll aggregate our records based on the `is_adult` classification. We'll calculate an average age for each of the groups."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: Combine\n",
+ " config:\n",
+ " group_by: is_adult\n",
+ " combine:\n",
+ " total:\n",
+ " value: age\n",
+ " fn: mean\n",
+ " - type: LogForTesting\n",
+ "options:\n",
+ " yaml_experimental_features: Combine\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-combine-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-combine-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If all was executed correctly, you should see the following lines at the bottom of the output log:\n",
+ "```\n",
+ "INFO:root:Result(is_adult=True, total=40.674418604651166)\n",
+ "INFO:root:Result(is_adult=False, total=11.714285714285714\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "All the previous pipelines were linear - output of one transform was an input to the next transform. This is also known as a `chain` pipeline. This is designated in the top-level pipeline configuration, for example:\n",
+ "```\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " ...\n",
+ "```\n",
+ "In YAML we can also create nonlinear pipelines. To do this, we need to specify `type: composite` (or omit this line at all, this is the default mode) and specify the `name` and `input` in each of the following transforms.\n",
Review Comment:
```suggestion
"In YAML we can also create nonlinear pipelines. To do this, we can specify `type: composite`, or omit this line completely (this is the default pipeline type). In these pipelines, we must specify the `input` in each of the transforms that take the output of previous transforms. This `input` is the name, or collection of names, of the transform(s) that feed into the receiving transform.\n",
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,680 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs the elements for debugging purposes.\n",
+ "\n",
+ "The `LogForTesting` transform lets us log the data when developing a pipeline. Remember, it is not advised to use this transform in production.\n",
+ "\n",
+ "Let's define the pipeline and save it to a file:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We can verify the contents of this file by running:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "! cat pipelines/pipeline-01.yaml"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Now, we can execute the yaml pipeline by passing this file as an argument to the following command:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here we use Python and `apache_beam` package to execute the pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for that in the future.\n",
+ "\n",
+ "If you scroll through the output logs, you'll find entries such as:\n",
+ "```\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=1, firstname='Reeba', age=58, country='Belgium', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=2, firstname='Maud', age=45, country='Spain', profession='firefighter')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=3, firstname='Meg', age=11, country='France', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=4, firstname='Rani', age=53, country='Spain', profession='doctor')\n",
+ "```\n",
+ "This is a representation of records from the input dataset."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's add a transform - `Filter`. To use this transform you need to specify the 'keep' condition and a language your condition is written in. Below you'll find an example with a condition written in Python.\n",
+ "This pipeline will filter out records containing people that are younger than 18 years old. The only records left to the next transform will be records representing adults. Verify this by scrolling to the bottom of the output logs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Similarly, we can create a condition in other languages, for example SQL. In this example we filter out people that are younger than 18 and have a profession other than being 'unemployed'."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: sql\n",
+ " keep: \"age >= 18 or (age < 18 and profession = 'unemployed')\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Another useful transform is `MapToFields`. This transform lets us manipulate fields of a record. For example, we can add a field to our records, which is a boolean field specifying if the person is adult or not."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam will try to infer the types involved in the mappings, but sometimes this is not possible. In these cases we can explicitly denote the expected output type, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult:\n",
+ " expression: \"age >= 18\"\n",
+ " output_type: boolean\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "When the `append` field is specified, one can `drop` fields as well, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " drop:\n",
+ " - age\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-03.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-03.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can also create simple UDFs (User Defined Functions) using Python or other languages. In the example below we add a field `random_number` which value is a random number not bigger than the age of the person."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " random_number:\n",
+ " callable: |\n",
+ " import random\n",
+ " def my_mapping(row):\n",
+ " return random.randrange(row.age)\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-04.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-04.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam YAML has EXPERIMENTAL ability to do aggregations to group and combine values across records. The is accomplished via the `Combine` transform type. Currently `Combine` needs to be in the `yaml_experimental_features` option (see the bottom of the pipeline) to use this transform.\n",
+ "\n",
+ "In this example we'll aggregate our records based on the `is_adult` classification. We'll calculate an average age for each of the groups."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: Combine\n",
+ " config:\n",
+ " group_by: is_adult\n",
+ " combine:\n",
+ " total:\n",
+ " value: age\n",
+ " fn: mean\n",
+ " - type: LogForTesting\n",
+ "options:\n",
+ " yaml_experimental_features: Combine\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-combine-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-combine-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If all was executed correctly, you should see the following lines at the bottom of the output log:\n",
+ "```\n",
+ "INFO:root:Result(is_adult=True, total=40.674418604651166)\n",
+ "INFO:root:Result(is_adult=False, total=11.714285714285714\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "All the previous pipelines were linear - output of one transform was an input to the next transform. This is also known as a `chain` pipeline. This is designated in the top-level pipeline configuration, for example:\n",
+ "```\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " ...\n",
+ "```\n",
+ "In YAML we can also create nonlinear pipelines. To do this, we need to specify `type: composite` (or omit this line at all, this is the default mode) and specify the `name` and `input` in each of the following transforms.\n",
+ "The specification below will create the following pipeline:\n",
+ "```\n",
+ " +----> Doctors -----------> SaveDoctors\n",
+ "InputData ---+\n",
+ " +----> OtherProfessions --> SaveOtherProfessions\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: composite\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " name: InputData\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " name: Doctors\n",
+ " input: InputData\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"profession == 'doctor'\"\n",
+ " - type: Filter\n",
+ " name: OtherProfessions\n",
+ " input: InputData\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"profession != 'doctor'\"\n",
+ " - type: WriteToCsv\n",
+ " name: SaveDoctors\n",
+ " input: Doctors\n",
+ " config:\n",
+ " path: results/doctors\n",
+ " - type: WriteToCsv\n",
+ " name: SaveOtherProfessions\n",
+ " input: OtherProfessions\n",
+ " config:\n",
+ " path: results/other-professions\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-nonlinear-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-nonlinear-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The output are 2 files: `results/doctors-00000-of-00001` and `results/other-professions-00000-of-00001`. Let's see their contents:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head results/doctors-00000-of-00001"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head results/other-professions-00000-of-00001"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "Congratulations! You've just run Apache Beam pipelines using YAML.\n",
+ "\n",
+ "For all the available operations visit the documentation: # todo(yaml) add url\n",
+ "\n",
+ "For a list of available transforms, visit # todo(yaml) add url\n",
+ "\n",
+ "To run your pipeline in Dataflow, you'll need to set up your Google Cloud and run the pipeline with the `DataflowRunner`. For more information, follow https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#run-on-dataflow"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [],
+ "metadata": {
+ "collapsed": false
+ }
+ }
Review Comment:
nit: Do we need this last blank code block?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1977273050
I updated the PR with documentation and a short explanation on why a pipeline with a transform from an expansion service (Filter-sql) logs output in a different format than a simple pipeline without an expansion service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1286423365
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
Review Comment:
OK, we can keep this for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1298165472
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,581 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('wget -O data/kinglear.txt https://storage.googleapis.com/dataflow-samples/shakespeare/kinglear.txt')\n",
+ "run('wget -O data/SMSSpamCollection.csv https://storage.googleapis.com/apache-beam-samples/SMSSpamCollection/SMSSpamCollection')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using CombinePerKey transform with the 'sum' function as a combine function,\n",
+ " # we'll calculate the occurrence of each word.\n",
+ " - type: CombinePerKey\n",
+ " config:\n",
+ " combine_fn: sum\n",
+ " name: GroupAndSum\n",
+ " input: PairWordsWith1\n",
+ "\n",
+ " # Format results - each record should be represented as 'word: count'.\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyMap\n",
+ " name: FormatResults\n",
+ " input: GroupAndSum\n",
+ " config:\n",
+ " fn: \"lambda word_count_tuple: f'{word_count_tuple[0]}: {word_count_tuple[1]}'\"\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: FormatResults\n",
+ " config:\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Each pipeline specification has to start with a `pipeline` key on the first line.\n",
+ "Following, there is a list of transforms, such as the first one:\n",
+ "```\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "```\n",
+ "This one reads an input file for futher processing.\n",
+ "Remember that the indentation is important here - it specifies object hierarchy.\n",
+ "YAML also supports comments - everything after the `#` is always treated as a comment. Use them to improve readability.\n",
+ "\n",
+ "Each operation has to specify the `type` descriptor and other fields, such as `name` and other transform-specific parameters.\n",
+ "Take a look at the documentation for a list of available transforms and their parameters. # todo(yaml) add link\n",
+ "\n",
+ "To link two operations, use the `input` field, just like in the third operation:\n",
+ "```\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "```\n",
+ "The `input` refers to the name of another transform, the previous one in this case.\n",
+ "This particular operation takes `fn` (stands for function) as an argument. Currently only Python functions are supported.\n",
+ "For more complicated functions, you can take advantage of YAML's multiline feature, as you can see in the second operation:\n",
+ "```\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "```\n",
+ "Here we had to import Python's regex package - `re`. To do that, we indicated a multiline string with `|` character and wrote the function in 2 lines.\n",
+ "\n",
+ "Let's run the pipeline executing the Python entry-point script (`apache_beam.yaml.main`) with our pipeline file as an argument:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-01.yaml')"
Review Comment:
Hi Veronica! Sorry, I think you need to run `pip install -e .` in the sdks/python directory in the same python virtual environment as the notebook is running.
This will not be necessary once the changes that this notebook relies on are released to the Python's central repository PyPi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1880635506
Hi @Polber, do you think we can merge it now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "yardenbm (via GitHub)" <gi...@apache.org>.
yardenbm commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1430139281
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,581 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('wget -O data/kinglear.txt https://storage.googleapis.com/dataflow-samples/shakespeare/kinglear.txt')\n",
+ "run('wget -O data/SMSSpamCollection.csv https://storage.googleapis.com/apache-beam-samples/SMSSpamCollection/SMSSpamCollection')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using CombinePerKey transform with the 'sum' function as a combine function,\n",
+ " # we'll calculate the occurrence of each word.\n",
+ " - type: CombinePerKey\n",
+ " config:\n",
+ " combine_fn: sum\n",
+ " name: GroupAndSum\n",
+ " input: PairWordsWith1\n",
+ "\n",
+ " # Format results - each record should be represented as 'word: count'.\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyMap\n",
+ " name: FormatResults\n",
+ " input: GroupAndSum\n",
+ " config:\n",
+ " fn: \"lambda word_count_tuple: f'{word_count_tuple[0]}: {word_count_tuple[1]}'\"\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: FormatResults\n",
+ " config:\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Each pipeline specification has to start with a `pipeline` key on the first line.\n",
+ "Following, there is a list of transforms, such as the first one:\n",
+ "```\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "```\n",
+ "This one reads an input file for futher processing.\n",
+ "Remember that the indentation is important here - it specifies object hierarchy.\n",
+ "YAML also supports comments - everything after the `#` is always treated as a comment. Use them to improve readability.\n",
+ "\n",
+ "Each operation has to specify the `type` descriptor and other fields, such as `name` and other transform-specific parameters.\n",
+ "Take a look at the documentation for a list of available transforms and their parameters. # todo(yaml) add link\n",
+ "\n",
+ "To link two operations, use the `input` field, just like in the third operation:\n",
+ "```\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "```\n",
+ "The `input` refers to the name of another transform, the previous one in this case.\n",
+ "This particular operation takes `fn` (stands for function) as an argument. Currently only Python functions are supported.\n",
+ "For more complicated functions, you can take advantage of YAML's multiline feature, as you can see in the second operation:\n",
+ "```\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "```\n",
+ "Here we had to import Python's regex package - `re`. To do that, we indicated a multiline string with `|` character and wrote the function in 2 lines.\n",
+ "\n",
+ "Let's run the pipeline executing the Python entry-point script (`apache_beam.yaml.main`) with our pipeline file as an argument:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-01.yaml')"
Review Comment:
Should this be functional by now? :)
As a developer with minimal experience with pipelines i've also tried using the notebook (without changes whatsoever) and got an exception with the validator in yaml_transform.py:
`jsonschema.exceptions.ValidationError: [.......] is not of type 'object' at line unknown`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "yardenbm (via GitHub)" <gi...@apache.org>.
yardenbm commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1430139281
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,581 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('wget -O data/kinglear.txt https://storage.googleapis.com/dataflow-samples/shakespeare/kinglear.txt')\n",
+ "run('wget -O data/SMSSpamCollection.csv https://storage.googleapis.com/apache-beam-samples/SMSSpamCollection/SMSSpamCollection')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using CombinePerKey transform with the 'sum' function as a combine function,\n",
+ " # we'll calculate the occurrence of each word.\n",
+ " - type: CombinePerKey\n",
+ " config:\n",
+ " combine_fn: sum\n",
+ " name: GroupAndSum\n",
+ " input: PairWordsWith1\n",
+ "\n",
+ " # Format results - each record should be represented as 'word: count'.\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyMap\n",
+ " name: FormatResults\n",
+ " input: GroupAndSum\n",
+ " config:\n",
+ " fn: \"lambda word_count_tuple: f'{word_count_tuple[0]}: {word_count_tuple[1]}'\"\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: FormatResults\n",
+ " config:\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Each pipeline specification has to start with a `pipeline` key on the first line.\n",
+ "Following, there is a list of transforms, such as the first one:\n",
+ "```\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "```\n",
+ "This one reads an input file for futher processing.\n",
+ "Remember that the indentation is important here - it specifies object hierarchy.\n",
+ "YAML also supports comments - everything after the `#` is always treated as a comment. Use them to improve readability.\n",
+ "\n",
+ "Each operation has to specify the `type` descriptor and other fields, such as `name` and other transform-specific parameters.\n",
+ "Take a look at the documentation for a list of available transforms and their parameters. # todo(yaml) add link\n",
+ "\n",
+ "To link two operations, use the `input` field, just like in the third operation:\n",
+ "```\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "```\n",
+ "The `input` refers to the name of another transform, the previous one in this case.\n",
+ "This particular operation takes `fn` (stands for function) as an argument. Currently only Python functions are supported.\n",
+ "For more complicated functions, you can take advantage of YAML's multiline feature, as you can see in the second operation:\n",
+ "```\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "```\n",
+ "Here we had to import Python's regex package - `re`. To do that, we indicated a multiline string with `|` character and wrote the function in 2 lines.\n",
+ "\n",
+ "Let's run the pipeline executing the Python entry-point script (`apache_beam.yaml.main`) with our pipeline file as an argument:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-01.yaml')"
Review Comment:
Should this be functional by now? :)
As a developer with minimal experience in pipelines i've also tried using the notebook (without changes whatsoever) and got an exception with the validator in yaml_transform.py:
`jsonschema.exceptions.ValidationError: [.......] is not of type 'object' at line unknown`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1976932158
Ping! Seems like this is ready to merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1432603044
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -415,6 +415,10 @@ def _parse_window_spec(spec):
'Flatten': Flatten,
'WindowInto': WindowInto,
'GroupByKey': beam.GroupByKey,
+ 'SumPerKey': lambda: beam.CombinePerKey(sum),
Review Comment:
The new version of the notebook doesn't use this transform. Closing this comment.
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1279522396
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
Review Comment:
I was considering this. I went for a custom function `save_to_file()` because Jupyter notebooks in some IDEs (like Intellij) will not highlight the text nicely when a `%%writefile` is used. Take a look at the screenshot:
On the other hand, I checked for the same in Google Colab and the highlighting there is absent, which makes it look better.
I'd still opt for my custom function, but let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1611179894
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/27284?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#27284](https://app.codecov.io/gh/apache/beam/pull/27284?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (dee90b2) into [master](https://app.codecov.io/gh/apache/beam/commit/a4281d8f35cf26bc62444f06b48d0019085e8e64?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (a4281d8) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #27284 +/- ##
=======================================
Coverage 71.49% 71.50%
=======================================
Files 858 858
Lines 104715 104750 +35
=======================================
+ Hits 74865 74897 +32
- Misses 28302 28305 +3
Partials 1548 1548
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `80.81% <ø> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://app.codecov.io/gh/apache/beam/pull/27284?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/yaml/yaml\_provider.py](https://app.codecov.io/gh/apache/beam/pull/27284?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0veWFtbC95YW1sX3Byb3ZpZGVyLnB5) | `68.07% <ø> (ø)` | |
... and [8 files with indirect coverage changes](https://app.codecov.io/gh/apache/beam/pull/27284/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1284568439
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
Review Comment:
Raw jupyter will not highlight the `%%writefile` nor a String version, so I'd still opt for my solution, given it is better in Intellij and it doesn't matter for the raw Jupyter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1247768969
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,646 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": 134,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> pip install --quiet apache-beam\n",
+ "\n",
+ ">> mkdir -p data\n",
+ "\n",
+ ">> gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt\n",
+ "Copying gs://dataflow-samples/shakespeare/kinglear.txt...\r\n",
+ "/ [1 files][153.6 KiB/153.6 KiB] \r\n",
+ "Operation completed over 1 objects/153.6 KiB. \r\n",
+ "\n",
+ ">> gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv\n",
+ "Copying gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection...\r\n",
+ "/ [1 files][466.7 KiB/466.7 KiB] \r\n",
+ "Operation completed over 1 objects/466.7 KiB. \r\n",
+ "\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 135,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/kinglear.txt\n",
+ "\tKING LEAR\r\n",
+ "\r\n",
+ "\r\n",
+ "\tDRAMATIS PERSONAE\r\n",
+ "\r\n",
+ "\r\n",
+ "LEAR\tking of Britain (KING LEAR:)\r\n",
+ "\r\n",
+ "KING OF FRANCE:\r\n",
+ "\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 136,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/SMSSpamCollection.csv\n",
+ "ham\tGo until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...\r\n",
+ "ham\tOk lar... Joking wif u oni...\r\n",
+ "spam\tFree entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's\r\n",
+ "ham\tU dun say so early hor... U c already then say...\r\n",
+ "ham\tNah I don't think he goes to usf, he lives around here though\r\n",
+ "spam\tFreeMsg Hey there darling it's been 3 week's now and no word back! I'd like some fun you up for it still? Tb ok! XxX std chgs to send, £1.50 to rcv\r\n",
+ "ham\tEven my brother is not like to speak with me. They treat me like aids patent.\r\n",
+ "ham\tAs per your request 'Melle Melle (Oru Minnaminunginte Nurungu Vettam)' has been set as your callertune for all Callers. Press *9 to copy your friends Callertune\r\n",
+ "spam\tWINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.\r\n",
+ "spam\tHad your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030\r\n",
+ "\n",
+ ">> wc -l data/SMSSpamCollection.csv\n",
+ "5574 data/SMSSpamCollection.csv\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
Review Comment:
That's a valid point, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1279531606
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
Review Comment:
Sounds good, is someone already assigned to it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] svetakvsundhar commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "svetakvsundhar (via GitHub)" <gi...@apache.org>.
svetakvsundhar commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1259728368
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
Review Comment:
Filed https://github.com/apache/beam/issues/27450
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] svetakvsundhar commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "svetakvsundhar (via GitHub)" <gi...@apache.org>.
svetakvsundhar commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1245569245
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
Review Comment:
```suggestion
"While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate (although https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input file into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "Let’s see how our data looks like."
Review Comment:
```suggestion
"Let’s see how our data looks."
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input file into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "Let’s see how our data looks like."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## First pipeline\n",
+ "We’ll start with creating a pipeline which loads the data, filters out valid messages leaving spam, and saves only valid lines to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " input: SmsData\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " input: SplitLine\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: KeepSpam\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "In this example, each transformation contains the 'input' key, but if the pipeline is linear, such as ours, we can let the inputs be implicit by designating the pipeline as a `chain` type.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01-chain.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To run the pipeline locally, using a DirectRunner, you need to run the yaml's main python script, passing the `pipeline-01-chain.yaml` (or `pipeline-01.yaml`) file as an input:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-01-chain.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's verify the results and see the content of the output file."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/result-pipeline-01-00000-of-00001.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "If everything went well, you should see only spam messages from our input dataset. Congratulations, onto the next example!\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Count words in spam messages, select top 10 popular words and write results to a file\n",
+ "\n",
+ "We'd like to write a pipeline which counts words occurring in spam messages, selects the most popular ones and writes the result to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ "\n",
+ " transforms:\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " # Split each line into an array, where the first element is message label (ham or spam) and the second is the content of the message.\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " # Select only the rows that contain spam messages, based on the label.\n",
+ " - type: PyFilter\n",
+ " name: SpamMessages\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings)\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line[1])\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairLoweredWordsWith1\n",
+ " fn: 'lambda word: (word.lower(), 1)'\n",
+ "\n",
+ " # Using SumPerKey transform, we'll calculate the occurence of each word.\n",
+ " - type: SumPerKey\n",
+ " name: GroupAndSum\n",
+ "\n",
+ " # Select 10 most popular words. Input format to this step is a tuple (word: count),\n",
+ " # so we provide the count (row[1]) as the key to compare the numbers\n",
+ " - type: TopNLargest\n",
+ " name: Largest\n",
+ " n: 10\n",
+ " key: 'lambda row: row[1]'\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " file_path_prefix: \"data/result-pipeline-02\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-02.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's run the pipeline:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-02.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To view the output:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/result-pipeline-02-00000-of-00001.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Summary\n",
+ "Congratulations! You've just ran yaml pipelines in Apache Beam.\n",
Review Comment:
```suggestion
"Congratulations! You've just run Apache Beam pipelines using YAML.\n",
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
Review Comment:
```suggestion
"# Try Apache Beam - YAML\n",
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
Review Comment:
```suggestion
"Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* the pipeline, but we envision various runners (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input file into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "Let’s see how our data looks like."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## First pipeline\n",
+ "We’ll start with creating a pipeline which loads the data, filters out valid messages leaving spam, and saves only valid lines to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " input: SmsData\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " input: SplitLine\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: KeepSpam\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "In this example, each transformation contains the 'input' key, but if the pipeline is linear, such as ours, we can let the inputs be implicit by designating the pipeline as a `chain` type.\n"
Review Comment:
Add a sentence here on what a `chain` type is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amotley commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "amotley (via GitHub)" <gi...@apache.org>.
amotley commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1245700773
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
Review Comment:
Can we add a landing page (similar to https://beam.apache.org/get-started/try-apache-beam/) which jumps to this collab page?
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
Review Comment:
```suggestion
"Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
```
Let's say "YAML syntax" instead of "declarative syntax". There have been some discussions on branding and right now we're not sure declarative is the right description.
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
Review Comment:
```suggestion
"In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input file into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "Let’s see how our data looks like."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## First pipeline\n",
+ "We’ll start with creating a pipeline which loads the data, filters out valid messages leaving spam, and saves only valid lines to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " input: SmsData\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " input: SplitLine\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: KeepSpam\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "In this example, each transformation contains the 'input' key, but if the pipeline is linear, such as ours, we can let the inputs be implicit by designating the pipeline as a `chain` type.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01-chain.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To run the pipeline locally, using a DirectRunner, you need to run the yaml's main python script, passing the `pipeline-01-chain.yaml` (or `pipeline-01.yaml`) file as an input:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-01-chain.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's verify the results and see the content of the output file."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/result-pipeline-01-00000-of-00001.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "If everything went well, you should see only spam messages from our input dataset. Congratulations, onto the next example!\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Count words in spam messages, select top 10 popular words and write results to a file\n",
Review Comment:
something like, "Example 2: <description>"
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
Review Comment:
```suggestion
"While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
Review Comment:
```suggestion
"Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
```
Let's remove this since we don't have these capabilities yet
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
Review Comment:
Can we add a pointer to some troubleshooting help if this doesn't work?
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
Review Comment:
It would be good to get someone from the Tech Writer team to review too. You can check w/ rszper@.
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
Review Comment:
Let's add a disclaimer somewhere this is is still EXPERIMENTAL
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input file into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "Let’s see how our data looks like."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## First pipeline\n",
Review Comment:
Maybe something like "Example 1: <description of what pipeline does>"
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input file into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "Let’s see how our data looks like."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## First pipeline\n",
+ "We’ll start with creating a pipeline which loads the data, filters out valid messages leaving spam, and saves only valid lines to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " input: SmsData\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " input: SplitLine\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: KeepSpam\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "In this example, each transformation contains the 'input' key, but if the pipeline is linear, such as ours, we can let the inputs be implicit by designating the pipeline as a `chain` type.\n"
Review Comment:
Can we remove 'chain' from this tutorial? It could be unnecessary overhead for new YAML users and we can cover in the advanced section of the documentation instead.
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input file into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "Let’s see how our data looks like."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## First pipeline\n",
+ "We’ll start with creating a pipeline which loads the data, filters out valid messages leaving spam, and saves only valid lines to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " input: SmsData\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " input: SplitLine\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: KeepSpam\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "In this example, each transformation contains the 'input' key, but if the pipeline is linear, such as ours, we can let the inputs be implicit by designating the pipeline as a `chain` type.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01-chain.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To run the pipeline locally, using a DirectRunner, you need to run the yaml's main python script, passing the `pipeline-01-chain.yaml` (or `pipeline-01.yaml`) file as an input:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-01-chain.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's verify the results and see the content of the output file."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/result-pipeline-01-00000-of-00001.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "If everything went well, you should see only spam messages from our input dataset. Congratulations, onto the next example!\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Count words in spam messages, select top 10 popular words and write results to a file\n",
+ "\n",
+ "We'd like to write a pipeline which counts words occurring in spam messages, selects the most popular ones and writes the result to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ "\n",
+ " transforms:\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " # Split each line into an array, where the first element is message label (ham or spam) and the second is the content of the message.\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " # Select only the rows that contain spam messages, based on the label.\n",
+ " - type: PyFilter\n",
+ " name: SpamMessages\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings)\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line[1])\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairLoweredWordsWith1\n",
+ " fn: 'lambda word: (word.lower(), 1)'\n",
+ "\n",
+ " # Using SumPerKey transform, we'll calculate the occurence of each word.\n",
+ " - type: SumPerKey\n",
+ " name: GroupAndSum\n",
+ "\n",
+ " # Select 10 most popular words. Input format to this step is a tuple (word: count),\n",
+ " # so we provide the count (row[1]) as the key to compare the numbers\n",
+ " - type: TopNLargest\n",
+ " name: Largest\n",
+ " n: 10\n",
+ " key: 'lambda row: row[1]'\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " file_path_prefix: \"data/result-pipeline-02\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-02.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's run the pipeline:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-02.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To view the output:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/result-pipeline-02-00000-of-00001.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Summary\n",
Review Comment:
If it's not too much extra work, it may be good to include a WordCount example so that folks can compare word count of YAML vs different SDKS (the other SDK quick starts use word count).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1478388159
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,671 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
Review Comment:
I already considered that approach and listed some benefits of using my custom method 'save_to_file'. See my thoughts here: https://github.com/apache/beam/pull/27284#discussion_r1276610298.
If you still think that the built-in `%%writefile` approach is better, let me know and I'll change it. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1511512821
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,671 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
Review Comment:
@Polber could you respond to this comment, please? I'm not sure which implementation to go with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1863210366
## Test Results
15 tests - 267 15 [:heavy_check_mark:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v2.11.0/README.md#the-symbols "passed tests") - 264 1h 30m 25s [:stopwatch:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v2.11.0/README.md#the-symbols "duration of all tests") - 1d 6h 9m 17s
12 suites - 35 0 [:zzz:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v2.11.0/README.md#the-symbols "skipped / disabled tests") - 3
12 files - 35 0 [:x:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v2.11.0/README.md#the-symbols "failed tests") ± 0
Results for commit d4d2b4d7. ± Comparison against base commit 5c7ec723.
<details>
<summary>This pull request <b>removes</b> 282 and <b>adds</b> 15 tests. <i>Note that renamed tests count towards both.</i></summary>
```
org.apache.beam.runners.dataflow.DataflowRunnerTest ‑ testBatchGroupIntoBatchesOverrideBytes
org.apache.beam.runners.dataflow.DataflowRunnerTest ‑ testBatchGroupIntoBatchesOverrideCount
org.apache.beam.sdk.PipelineTest ‑ testIdentityTransform
org.apache.beam.sdk.PipelineTest ‑ testMultipleApply
org.apache.beam.sdk.PipelineTest ‑ testTupleInjectionTransform
org.apache.beam.sdk.PipelineTest ‑ testTupleProjectionTransform
org.apache.beam.sdk.SdkHarnessEnvironmentTest ‑ testJammAgentAvailable
org.apache.beam.sdk.SdkHarnessEnvironmentTest ‑ testLogging
org.apache.beam.sdk.SdkHarnessEnvironmentTest ‑ testTlsAvailable
org.apache.beam.sdk.io.CountingSourceTest ‑ testBoundedSourceSplits
…
```
```
org.apache.beam.examples.complete.AutoCompleteIT ‑ testE2EAutoComplete
org.apache.beam.examples.complete.TfIdfIT ‑ testE2ETfIdf
org.apache.beam.examples.complete.TopWikipediaSessionsIT ‑ testE2ETopWikiPages
org.apache.beam.examples.complete.TrafficMaxLaneFlowIT ‑ testE2ETrafficMaxLaneFlow
org.apache.beam.examples.complete.TrafficRoutesIT ‑ testE2ETrafficRoutes
org.apache.beam.examples.cookbook.BigQueryTornadoesIT ‑ testE2EBigQueryTornadoesWithExport
org.apache.beam.examples.cookbook.BigQueryTornadoesIT ‑ testE2EBigQueryTornadoesWithExportUsingQuery
org.apache.beam.examples.cookbook.BigQueryTornadoesIT ‑ testE2eBigQueryTornadoesWithStorageApi
org.apache.beam.examples.cookbook.BigQueryTornadoesIT ‑ testE2eBigQueryTornadoesWithStorageApiUsingQuery
org.apache.beam.examples.cookbook.CombinePerKeyExamplesIT ‑ testE2ECombinePerKey
…
```
</details>
[test-results]:data:application/gzip;base64,H4sIAFbUgWUC/12Myw7CIBBFf6Vh7YJOeag/Y2AoycS2GAor47+LaGjr7p5zk/NknqZxZdeuh1PH1kxpI5ejSRSWwlKALKacqd4NbmtG/DN3ehTDm/CGpoMYYwzxZ2JeWvGzD8Gv2HqVd7nK+xqGeaZUgDnhwAqnlTEaB636Mx9QKbDALb8geItKKvDs9QaylxQfBAEAAA==
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amotley commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "amotley (via GitHub)" <gi...@apache.org>.
amotley commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1247123481
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
Review Comment:
Tech Writers can probably help answer this question too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1300123614
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -415,6 +415,10 @@ def _parse_window_spec(spec):
'Flatten': Flatten,
'WindowInto': WindowInto,
'GroupByKey': beam.GroupByKey,
+ 'SumPerKey': lambda: beam.CombinePerKey(sum),
Review Comment:
Two things:
1) I wanted to implement the CombineGlobally transform that would take a TopN combiner, as you suggested. For some reason I can't get it to work... That's my code for defining the transform in the dictionary in the yaml_provider.py:
```python
'CombineGlobally': lambda fn: beam.CombineGlobally(
fn=python_callable.PythonCallableWithSource(fn)),
```
Then in the yaml pipeline I would use it like this:
```
- type: CombineGlobally
name: MostPopular
input: XXX
fn: "beam.combiners.TopCombineFn(1)"
```
but that results in error:
```
self = <apache_beam.utils.python_callable.PythonCallableWithSource object at 0x7f9802d93b20>
args = ([3, 4],), kwargs = {}
def __call__(self, *args, **kwargs):
> return self._callable(*args, **kwargs)
E TypeError: 'TopCombineFn' object is not callable
```
Do you have a suggestion how to implement it otherwise?
2) I still think that we should have high level transforms that are popular, like 'TopN', 'Largest', 'Smallest', etc available in the global scope. If the yaml api is aimed at less experienced developer, they will have a hard time creating these transforms with callable arguments.
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -415,6 +415,10 @@ def _parse_window_spec(spec):
'Flatten': Flatten,
'WindowInto': WindowInto,
'GroupByKey': beam.GroupByKey,
+ 'SumPerKey': lambda: beam.CombinePerKey(sum),
Review Comment:
Two things:
1) I wanted to implement the CombineGlobally transform that would take a TopN combiner, as you suggested. For some reason I can't get it to work... That's my code for defining the transform in the dictionary in the yaml_provider.py:
```python
'CombineGlobally': lambda fn: beam.CombineGlobally(
fn=python_callable.PythonCallableWithSource(fn)),
```
Then in the yaml pipeline I would use it like this:
```
- type: CombineGlobally
name: MostPopular
input: XXX
fn: "beam.combiners.TopCombineFn(1)"
```
but that results in error:
```
self = <apache_beam.utils.python_callable.PythonCallableWithSource object at 0x7f9802d93b20>
args = ([3, 4],), kwargs = {}
def __call__(self, *args, **kwargs):
> return self._callable(*args, **kwargs)
E TypeError: 'TopCombineFn' object is not callable
```
Do you have a suggestion how to implement it otherwise?
2) I still think that we should have high level transforms that are popular, like 'TopN', 'Largest', 'Smallest', etc available in the global scope. If the yaml api is aimed at less experienced developer, they will have a hard time creating these transforms with callable arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1286422548
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -415,6 +415,10 @@ def _parse_window_spec(spec):
'Flatten': Flatten,
'WindowInto': WindowInto,
'GroupByKey': beam.GroupByKey,
+ 'SumPerKey': lambda: beam.CombinePerKey(sum),
Review Comment:
CombineGlobally would take a TopN combiner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1329917081
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -415,6 +415,10 @@ def _parse_window_spec(spec):
'Flatten': Flatten,
'WindowInto': WindowInto,
'GroupByKey': beam.GroupByKey,
+ 'SumPerKey': lambda: beam.CombinePerKey(sum),
Review Comment:
Hi @amotley and @robertwb, could you take a look at this unresolved issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1259486927
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
Review Comment:
Me or @svetakvsundhar will work on this once this PR is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1279549145
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -415,6 +415,10 @@ def _parse_window_spec(spec):
'Flatten': Flatten,
'WindowInto': WindowInto,
'GroupByKey': beam.GroupByKey,
+ 'SumPerKey': lambda: beam.CombinePerKey(sum),
Review Comment:
Fair, I agree that a CombinePerKey with a callable argument is better. Implementing this.
What would you say about the TopNLargest? It seems to me that we should map all (or at least most) of the functions that the Python SDK offers. Maybe we should automate it just like you did with the [io transforms](https://github.com/apache/beam/blob/22b30adc848de3605e7ab1dc1ff4f190909a2a9f/sdks/python/apache_beam/yaml/yaml_provider.py#L388)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amotley commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "amotley (via GitHub)" <gi...@apache.org>.
amotley commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1247127090
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,646 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": 134,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> pip install --quiet apache-beam\n",
+ "\n",
+ ">> mkdir -p data\n",
+ "\n",
+ ">> gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt\n",
+ "Copying gs://dataflow-samples/shakespeare/kinglear.txt...\r\n",
+ "/ [1 files][153.6 KiB/153.6 KiB] \r\n",
+ "Operation completed over 1 objects/153.6 KiB. \r\n",
+ "\n",
+ ">> gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv\n",
+ "Copying gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection...\r\n",
+ "/ [1 files][466.7 KiB/466.7 KiB] \r\n",
+ "Operation completed over 1 objects/466.7 KiB. \r\n",
+ "\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 135,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/kinglear.txt\n",
+ "\tKING LEAR\r\n",
+ "\r\n",
+ "\r\n",
+ "\tDRAMATIS PERSONAE\r\n",
+ "\r\n",
+ "\r\n",
+ "LEAR\tking of Britain (KING LEAR:)\r\n",
+ "\r\n",
+ "KING OF FRANCE:\r\n",
+ "\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 136,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/SMSSpamCollection.csv\n",
+ "ham\tGo until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...\r\n",
+ "ham\tOk lar... Joking wif u oni...\r\n",
+ "spam\tFree entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's\r\n",
+ "ham\tU dun say so early hor... U c already then say...\r\n",
+ "ham\tNah I don't think he goes to usf, he lives around here though\r\n",
+ "spam\tFreeMsg Hey there darling it's been 3 week's now and no word back! I'd like some fun you up for it still? Tb ok! XxX std chgs to send, £1.50 to rcv\r\n",
+ "ham\tEven my brother is not like to speak with me. They treat me like aids patent.\r\n",
+ "ham\tAs per your request 'Melle Melle (Oru Minnaminunginte Nurungu Vettam)' has been set as your callertune for all Callers. Press *9 to copy your friends Callertune\r\n",
+ "spam\tWINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.\r\n",
+ "spam\tHad your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030\r\n",
+ "\n",
+ ">> wc -l data/SMSSpamCollection.csv\n",
+ "5574 data/SMSSpamCollection.csv\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `yaml` file specifying our pipeline."
Review Comment:
```suggestion
"To start, we'll create a `YAML` file specifying our pipeline."
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,646 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": 134,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> pip install --quiet apache-beam\n",
+ "\n",
+ ">> mkdir -p data\n",
+ "\n",
+ ">> gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt\n",
+ "Copying gs://dataflow-samples/shakespeare/kinglear.txt...\r\n",
+ "/ [1 files][153.6 KiB/153.6 KiB] \r\n",
+ "Operation completed over 1 objects/153.6 KiB. \r\n",
+ "\n",
+ ">> gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv\n",
+ "Copying gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection...\r\n",
+ "/ [1 files][466.7 KiB/466.7 KiB] \r\n",
+ "Operation completed over 1 objects/466.7 KiB. \r\n",
+ "\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 135,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/kinglear.txt\n",
+ "\tKING LEAR\r\n",
+ "\r\n",
+ "\r\n",
+ "\tDRAMATIS PERSONAE\r\n",
+ "\r\n",
+ "\r\n",
+ "LEAR\tking of Britain (KING LEAR:)\r\n",
+ "\r\n",
+ "KING OF FRANCE:\r\n",
+ "\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 136,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/SMSSpamCollection.csv\n",
+ "ham\tGo until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...\r\n",
+ "ham\tOk lar... Joking wif u oni...\r\n",
+ "spam\tFree entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's\r\n",
+ "ham\tU dun say so early hor... U c already then say...\r\n",
+ "ham\tNah I don't think he goes to usf, he lives around here though\r\n",
+ "spam\tFreeMsg Hey there darling it's been 3 week's now and no word back! I'd like some fun you up for it still? Tb ok! XxX std chgs to send, £1.50 to rcv\r\n",
+ "ham\tEven my brother is not like to speak with me. They treat me like aids patent.\r\n",
+ "ham\tAs per your request 'Melle Melle (Oru Minnaminunginte Nurungu Vettam)' has been set as your callertune for all Callers. Press *9 to copy your friends Callertune\r\n",
+ "spam\tWINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.\r\n",
+ "spam\tHad your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030\r\n",
+ "\n",
+ ">> wc -l data/SMSSpamCollection.csv\n",
+ "5574 data/SMSSpamCollection.csv\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 153,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ "\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using SumPerKey transform, we'll calculate the occurrence of each word.\n",
+ " - type: SumPerKey\n",
+ " name: GroupAndSum\n",
+ "\n",
+ " # Format results - each record should be represented as 'word: count'.\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyMap\n",
+ " name: FormatResults\n",
+ " fn: \"lambda word_count_tuple: f'{word_count_tuple[0]}: {word_count_tuple[1]}'\"\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's tun the pipeline executing the Python script with the pipeline file as an argument:"
Review Comment:
```suggestion
"Let's run the pipeline executing the Python script with the pipeline file as an argument:"
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,646 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": 134,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> pip install --quiet apache-beam\n",
+ "\n",
+ ">> mkdir -p data\n",
+ "\n",
+ ">> gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt\n",
+ "Copying gs://dataflow-samples/shakespeare/kinglear.txt...\r\n",
+ "/ [1 files][153.6 KiB/153.6 KiB] \r\n",
+ "Operation completed over 1 objects/153.6 KiB. \r\n",
+ "\n",
+ ">> gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv\n",
+ "Copying gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection...\r\n",
+ "/ [1 files][466.7 KiB/466.7 KiB] \r\n",
+ "Operation completed over 1 objects/466.7 KiB. \r\n",
+ "\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 135,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/kinglear.txt\n",
+ "\tKING LEAR\r\n",
+ "\r\n",
+ "\r\n",
+ "\tDRAMATIS PERSONAE\r\n",
+ "\r\n",
+ "\r\n",
+ "LEAR\tking of Britain (KING LEAR:)\r\n",
+ "\r\n",
+ "KING OF FRANCE:\r\n",
+ "\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 136,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/SMSSpamCollection.csv\n",
+ "ham\tGo until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...\r\n",
+ "ham\tOk lar... Joking wif u oni...\r\n",
+ "spam\tFree entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's\r\n",
+ "ham\tU dun say so early hor... U c already then say...\r\n",
+ "ham\tNah I don't think he goes to usf, he lives around here though\r\n",
+ "spam\tFreeMsg Hey there darling it's been 3 week's now and no word back! I'd like some fun you up for it still? Tb ok! XxX std chgs to send, £1.50 to rcv\r\n",
+ "ham\tEven my brother is not like to speak with me. They treat me like aids patent.\r\n",
+ "ham\tAs per your request 'Melle Melle (Oru Minnaminunginte Nurungu Vettam)' has been set as your callertune for all Callers. Press *9 to copy your friends Callertune\r\n",
+ "spam\tWINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.\r\n",
+ "spam\tHad your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030\r\n",
+ "\n",
+ ">> wc -l data/SMSSpamCollection.csv\n",
+ "5574 data/SMSSpamCollection.csv\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 153,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ "\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using SumPerKey transform, we'll calculate the occurrence of each word.\n",
+ " - type: SumPerKey\n",
+ " name: GroupAndSum\n",
+ "\n",
+ " # Format results - each record should be represented as 'word: count'.\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyMap\n",
+ " name: FormatResults\n",
+ " fn: \"lambda word_count_tuple: f'{word_count_tuple[0]}: {word_count_tuple[1]}'\"\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's tun the pipeline executing the Python script with the pipeline file as an argument:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 154,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-01.yaml\n",
+ "Building pipeline...\r\n",
+ "INFO:apache_beam.yaml.yaml_transform:Expanding \"InputText\" at line 7 \r\n",
+ "INFO:apache_beam.yaml.yaml_transform:Expanding \"FindWords\" at line 12 \r\n",
+ "INFO:apache_beam.yaml.yaml_transform:Expanding \"PairWordsWith1\" at line 19 \r\n",
+ "INFO:apache_beam.yaml.yaml_transform:Expanding \"GroupAndSum\" at line 24 \r\n",
+ "INFO:apache_beam.yaml.yaml_transform:Expanding \"FormatResults\" at line 28 \r\n",
+ "INFO:apache_beam.yaml.yaml_transform:Expanding \"SaveToText\" at line 33 \r\n",
+ "Running pipeline...\r\n",
+ "WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: -*-of-%(num_shards)05d\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's inspect the data. Each line contains a word and an associated count."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 155,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/result-pipeline-01-00000-of-00001.txt\n",
+ "KING: 243\r\n",
+ "LEAR: 236\r\n",
+ "DRAMATIS: 1\r\n",
+ "PERSONAE: 1\r\n",
+ "king: 65\r\n",
+ "of: 447\r\n",
+ "Britain: 2\r\n",
+ "OF: 15\r\n",
+ "FRANCE: 10\r\n",
+ "DUKE: 3\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/result-pipeline-01-00000-of-00001.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 2: load data, filter unwanted lines and save results to a text file.\n",
+ "In this example, we'll create a pipeline which loads the data, filters out valid messages leaving spam, and saves only valid lines to a file.\n",
+ "Please note that this time we didn't specify the top-level type as `chain`. This allows for more flexibility when creating a pipeline, but requires us to specify the `input` explicitly for each transform.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 138,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ "\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " input: SmsData\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " input: SplitLine\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: KeepSpam\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-02.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Since this pipeline is linear, we can rewrite it to a `chain` and drop the `input`s in transforms."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " file_path_prefix: \"data/result-pipeline-02\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-02-chain.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To run the pipeline locally, using a DirectRunner, you need to run the yaml's main python script, passing the `pipeline-02-chain.yaml` (or `pipeline-02.yaml`) file as an input:"
Review Comment:
```suggestion
"To run the pipeline locally, using a DirectRunner, you need to run the YAML's main python script, passing the `pipeline-02-chain.yaml` (or `pipeline-02.yaml`) file as an input:"
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input file into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "Let’s see how our data looks like."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## First pipeline\n",
+ "We’ll start with creating a pipeline which loads the data, filters out valid messages leaving spam, and saves only valid lines to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " input: SmsData\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " input: SplitLine\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: KeepSpam\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "In this example, each transformation contains the 'input' key, but if the pipeline is linear, such as ours, we can let the inputs be implicit by designating the pipeline as a `chain` type.\n"
Review Comment:
I would rather keep the inputs and remove "chain" entirely since it's more advanced syntax.
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,646 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": 134,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> pip install --quiet apache-beam\n",
+ "\n",
+ ">> mkdir -p data\n",
+ "\n",
+ ">> gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt\n",
+ "Copying gs://dataflow-samples/shakespeare/kinglear.txt...\r\n",
+ "/ [1 files][153.6 KiB/153.6 KiB] \r\n",
+ "Operation completed over 1 objects/153.6 KiB. \r\n",
+ "\n",
+ ">> gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv\n",
+ "Copying gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection...\r\n",
+ "/ [1 files][466.7 KiB/466.7 KiB] \r\n",
+ "Operation completed over 1 objects/466.7 KiB. \r\n",
+ "\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 135,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/kinglear.txt\n",
+ "\tKING LEAR\r\n",
+ "\r\n",
+ "\r\n",
+ "\tDRAMATIS PERSONAE\r\n",
+ "\r\n",
+ "\r\n",
+ "LEAR\tking of Britain (KING LEAR:)\r\n",
+ "\r\n",
+ "KING OF FRANCE:\r\n",
+ "\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 136,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/SMSSpamCollection.csv\n",
+ "ham\tGo until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...\r\n",
+ "ham\tOk lar... Joking wif u oni...\r\n",
+ "spam\tFree entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's\r\n",
+ "ham\tU dun say so early hor... U c already then say...\r\n",
+ "ham\tNah I don't think he goes to usf, he lives around here though\r\n",
+ "spam\tFreeMsg Hey there darling it's been 3 week's now and no word back! I'd like some fun you up for it still? Tb ok! XxX std chgs to send, £1.50 to rcv\r\n",
+ "ham\tEven my brother is not like to speak with me. They treat me like aids patent.\r\n",
+ "ham\tAs per your request 'Melle Melle (Oru Minnaminunginte Nurungu Vettam)' has been set as your callertune for all Callers. Press *9 to copy your friends Callertune\r\n",
+ "spam\tWINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.\r\n",
+ "spam\tHad your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030\r\n",
+ "\n",
+ ">> wc -l data/SMSSpamCollection.csv\n",
+ "5574 data/SMSSpamCollection.csv\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
Review Comment:
Here it may be good to do really basic syntax introduction, similar to "structure" section in https://s.apache.org/beam-yaml-pipelines-syntax
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,646 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": 134,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> pip install --quiet apache-beam\n",
+ "\n",
+ ">> mkdir -p data\n",
+ "\n",
+ ">> gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt\n",
+ "Copying gs://dataflow-samples/shakespeare/kinglear.txt...\r\n",
+ "/ [1 files][153.6 KiB/153.6 KiB] \r\n",
+ "Operation completed over 1 objects/153.6 KiB. \r\n",
+ "\n",
+ ">> gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv\n",
+ "Copying gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection...\r\n",
+ "/ [1 files][466.7 KiB/466.7 KiB] \r\n",
+ "Operation completed over 1 objects/466.7 KiB. \r\n",
+ "\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 135,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/kinglear.txt\n",
+ "\tKING LEAR\r\n",
+ "\r\n",
+ "\r\n",
+ "\tDRAMATIS PERSONAE\r\n",
+ "\r\n",
+ "\r\n",
+ "LEAR\tking of Britain (KING LEAR:)\r\n",
+ "\r\n",
+ "KING OF FRANCE:\r\n",
+ "\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 136,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/SMSSpamCollection.csv\n",
+ "ham\tGo until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...\r\n",
+ "ham\tOk lar... Joking wif u oni...\r\n",
+ "spam\tFree entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's\r\n",
+ "ham\tU dun say so early hor... U c already then say...\r\n",
+ "ham\tNah I don't think he goes to usf, he lives around here though\r\n",
+ "spam\tFreeMsg Hey there darling it's been 3 week's now and no word back! I'd like some fun you up for it still? Tb ok! XxX std chgs to send, £1.50 to rcv\r\n",
+ "ham\tEven my brother is not like to speak with me. They treat me like aids patent.\r\n",
+ "ham\tAs per your request 'Melle Melle (Oru Minnaminunginte Nurungu Vettam)' has been set as your callertune for all Callers. Press *9 to copy your friends Callertune\r\n",
+ "spam\tWINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.\r\n",
+ "spam\tHad your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030\r\n",
+ "\n",
+ ">> wc -l data/SMSSpamCollection.csv\n",
+ "5574 data/SMSSpamCollection.csv\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `yaml` file specifying our pipeline."
Review Comment:
Or maybe .yaml?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1247771492
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,646 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": 134,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> pip install --quiet apache-beam\n",
+ "\n",
+ ">> mkdir -p data\n",
+ "\n",
+ ">> gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt\n",
+ "Copying gs://dataflow-samples/shakespeare/kinglear.txt...\r\n",
+ "/ [1 files][153.6 KiB/153.6 KiB] \r\n",
+ "Operation completed over 1 objects/153.6 KiB. \r\n",
+ "\n",
+ ">> gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv\n",
+ "Copying gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection...\r\n",
+ "/ [1 files][466.7 KiB/466.7 KiB] \r\n",
+ "Operation completed over 1 objects/466.7 KiB. \r\n",
+ "\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 135,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/kinglear.txt\n",
+ "\tKING LEAR\r\n",
+ "\r\n",
+ "\r\n",
+ "\tDRAMATIS PERSONAE\r\n",
+ "\r\n",
+ "\r\n",
+ "LEAR\tking of Britain (KING LEAR:)\r\n",
+ "\r\n",
+ "KING OF FRANCE:\r\n",
+ "\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 136,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/SMSSpamCollection.csv\n",
+ "ham\tGo until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...\r\n",
+ "ham\tOk lar... Joking wif u oni...\r\n",
+ "spam\tFree entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's\r\n",
+ "ham\tU dun say so early hor... U c already then say...\r\n",
+ "ham\tNah I don't think he goes to usf, he lives around here though\r\n",
+ "spam\tFreeMsg Hey there darling it's been 3 week's now and no word back! I'd like some fun you up for it still? Tb ok! XxX std chgs to send, £1.50 to rcv\r\n",
+ "ham\tEven my brother is not like to speak with me. They treat me like aids patent.\r\n",
+ "ham\tAs per your request 'Melle Melle (Oru Minnaminunginte Nurungu Vettam)' has been set as your callertune for all Callers. Press *9 to copy your friends Callertune\r\n",
+ "spam\tWINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.\r\n",
+ "spam\tHad your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030\r\n",
+ "\n",
+ ">> wc -l data/SMSSpamCollection.csv\n",
+ "5574 data/SMSSpamCollection.csv\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `yaml` file specifying our pipeline."
Review Comment:
I'd go for .yaml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1630500305
Hi @robertwb, could you take a look at this PR? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1301495823
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,582 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK. You can use any text editor.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and downloads some text files from Cloud Storage to your local file system. We'll use these text files as input to the pipelines."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('wget -O data/kinglear.txt https://storage.googleapis.com/dataflow-samples/shakespeare/kinglear.txt')\n",
+ "run('wget -O data/SMSSpamCollection.csv https://storage.googleapis.com/apache-beam-samples/SMSSpamCollection/SMSSpamCollection')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with two datasets. We'll use `kinglear.txt` for the first example, and `SMSSpamCollection.csv` for the second and third examples.\n",
+ "Let's first take a look at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file that contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file that contains 5,574 rows of SMS messages labeled either spam or not-spam (\"ham\"). Each row contains two columns separated by a tab character:\n",
+ "1. `Column 1`: The label, either `ham` or `spam`\n",
+ "2. `Column 2`: The SMS message as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: Word count\n",
+ "This example is a version of the [WordCount](https://beam.apache.org/get-started/wordcount-example/)). It reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using CombinePerKey transform with the 'sum' function as a combine function,\n",
+ " # we'll calculate the occurrence of each word.\n",
+ " - type: CombinePerKey\n",
+ " config:\n",
+ " combine_fn: sum\n",
+ " name: GroupAndSum\n",
+ " input: PairWordsWith1\n",
+ "\n",
+ " # Format results - each record should be represented as 'word: count'.\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyMap\n",
+ " name: FormatResults\n",
+ " input: GroupAndSum\n",
+ " config:\n",
+ " fn: \"lambda word_count_tuple: f'{word_count_tuple[0]}: {word_count_tuple[1]}'\"\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: FormatResults\n",
+ " config:\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Each pipeline specification must start with a `pipeline` key on the first line.\n",
+ "The `pipeline` keyh is followed by a list of transforms. For example, the first transform reads the input file:\n",
Review Comment:
thanks! I fixed it along with a few other typos.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1432579864
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,629 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some of the available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs to elements for debugging purposes. The `LogForTesting` transform lets us see the data when developing a pipeline. Remember, it is not advised to use this transform in production."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml"
Review Comment:
Very good suggestion, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1977092183
Thanks for the ping, I'll submit an updated PR today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1513144263
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,671 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
Review Comment:
I left a comment in the thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1529348432
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,671 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
Review Comment:
Decided to go with the '%%writefile'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "amotley (via GitHub)" <gi...@apache.org>.
amotley commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1344502754
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,582 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK. You can use any text editor.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and downloads some text files from Cloud Storage to your local file system. We'll use these text files as input to the pipelines."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('wget -O data/kinglear.txt https://storage.googleapis.com/dataflow-samples/shakespeare/kinglear.txt')\n",
+ "run('wget -O data/SMSSpamCollection.csv https://storage.googleapis.com/apache-beam-samples/SMSSpamCollection/SMSSpamCollection')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with two datasets. We'll use `kinglear.txt` for the first example, and `SMSSpamCollection.csv` for the second and third examples.\n",
+ "Let's first take a look at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file that contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file that contains 5,574 rows of SMS messages labeled either spam or not-spam (\"ham\"). Each row contains two columns separated by a tab character:\n",
+ "1. `Column 1`: The label, either `ham` or `spam`\n",
+ "2. `Column 2`: The SMS message as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: Word count\n",
+ "This example is a version of the [WordCount](https://beam.apache.org/get-started/wordcount-example/)). It reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using CombinePerKey transform with the 'sum' function as a combine function,\n",
+ " # we'll calculate the occurrence of each word.\n",
+ " - type: CombinePerKey\n",
+ " config:\n",
+ " combine_fn: sum\n",
+ " name: GroupAndSum\n",
+ " input: PairWordsWith1\n",
+ "\n",
+ " # Format results - each record should be represented as 'word: count'.\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyMap\n",
+ " name: FormatResults\n",
+ " input: GroupAndSum\n",
+ " config:\n",
+ " fn: \"lambda word_count_tuple: f'{word_count_tuple[0]}: {word_count_tuple[1]}'\"\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: FormatResults\n",
+ " config:\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Each pipeline specification must start with a `pipeline` key on the first line.\n",
+ "The `pipeline` keyh is followed by a list of transforms. For example, the first transform reads the input file:\n",
+ "```\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "```\n",
+ "Note: The indentation is important, because it specifies object hierarchy.\n",
+ "YAML supports comments. Everything after the `#` is always treated as a comment. Use them to improve readability.\n",
+ "\n",
+ "Each operation must specify the `type` descriptor and other fields, such as `name` and other transform-specific parameters.\n",
+ "For a list of available transforms and their parameters, see the YAML API documentation. # todo(yaml) add link\n",
Review Comment:
I would lean toward leaving and creating tickets. You can use the new YAML issue tags that Jeff created: https://bugdashboard.corp.google.com/app/tree;dashboardId=551658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-2005133600
@Polber I converted saving the file to the built-in Jupyter's '%%writefile'. PR is ready for the final review/merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1431922620
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,629 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some of the available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs to elements for debugging purposes. The `LogForTesting` transform lets us see the data when developing a pipeline. Remember, it is not advised to use this transform in production."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml"
Review Comment:
I think in these execution steps (or maybe just the first example), it might be nice to call out the fact it is being saved to a file before executing. Something like
Here we create a sample yaml pipeline and save to a file
```
pipeline = '''
pipeline:
type: chain
transforms:
- type: ReadFromCsv
config:
path: data/people.csv
- type: LogForTesting
'''
save_to_file(pipeline, 'pipelines/pipeline-01.yaml')
```
We can verify the contents of this file by running
```
! cat pipelines/pipeline-01.yaml
```
Now, we can execute the yaml pipeline by passing it as an argument to the following command
```
! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml
```
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
Review Comment:
Could you rebase on main and make sure if there are any changes in this file, they are relevant to this PR and not removing new code
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,629 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some of the available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs to elements for debugging purposes. The `LogForTesting` transform lets us see the data when developing a pipeline. Remember, it is not advised to use this transform in production."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If you scroll through the logs, you'll find entries such as:\n",
+ "```\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=1, firstname='Reeba', age=58, country='Belgium', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=2, firstname='Maud', age=45, country='Spain', profession='firefighter')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=3, firstname='Meg', age=11, country='France', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=4, firstname='Rani', age=53, country='Spain', profession='doctor')\n",
+ "```\n",
+ "This is a representation of records from the input dataset."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's add a transform - `Filter`. To use this transform you need to specify the 'keep' condition and a language your condition is written in. Below you'll find an example with a condition written in Python.\n",
+ "This pipeline will filter out records containing people that are younger than 18 years old. The only records left to the next transform will be records representing adults. Verify this by scrolling to the bottom of the output logs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Similarly, we can create a condition in other languages, for example SQL. In this example we filter out people that are younger than 18 and have a profession other than being 'unemployed'."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: sql\n",
+ " keep: \"age >= 18 or (age < 18 and profession = 'unemployed')\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Another useful transform is `MapToFields`. This transform lets us manipulate fields of a record. For example, we can add a field to our records, which is a boolean field specifying if the person is adult or not."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam will try to infer the types involved in the mappings, but sometimes this is not possible. In these cases we can explicitly denote the expected output type, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult:\n",
+ " expression: \"age >= 18\"\n",
+ " output_type: boolean\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "When the `append` field is specified, one can `drop` fields as well, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " drop:\n",
+ " - age\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-03.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-03.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can also create simple UDFs (User Defined Functions) using Python or other languages. In the example below we add a field `random_number` which value is a random number not bigger than the age of the person."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " random_number:\n",
+ " callable: |\n",
+ " import random\n",
+ " def my_mapping(row):\n",
+ " return random.randrange(row.age)\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-04.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-04.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam YAML has EXPERIMENTAL ability to do aggregations to group and combine values across records. The is accomplished via the `Combine` transform type. Currently `Combine` needs to be in the `yaml_experimental_features` option (see the bottom of the pipeline) to use this transform.\n",
+ "\n",
+ "In this example we'll aggregate our records based on the `is_adult` classification. We'll calculate an average age for each of the groups."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:K\n",
Review Comment:
I assume typo
```suggestion
" fields:\n",
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,629 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some of the available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs to elements for debugging purposes. The `LogForTesting` transform lets us see the data when developing a pipeline. Remember, it is not advised to use this transform in production."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If you scroll through the logs, you'll find entries such as:\n",
+ "```\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=1, firstname='Reeba', age=58, country='Belgium', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=2, firstname='Maud', age=45, country='Spain', profession='firefighter')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=3, firstname='Meg', age=11, country='France', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=4, firstname='Rani', age=53, country='Spain', profession='doctor')\n",
+ "```\n",
+ "This is a representation of records from the input dataset."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's add a transform - `Filter`. To use this transform you need to specify the 'keep' condition and a language your condition is written in. Below you'll find an example with a condition written in Python.\n",
+ "This pipeline will filter out records containing people that are younger than 18 years old. The only records left to the next transform will be records representing adults. Verify this by scrolling to the bottom of the output logs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Similarly, we can create a condition in other languages, for example SQL. In this example we filter out people that are younger than 18 and have a profession other than being 'unemployed'."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: sql\n",
+ " keep: \"age >= 18 or (age < 18 and profession = 'unemployed')\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Another useful transform is `MapToFields`. This transform lets us manipulate fields of a record. For example, we can add a field to our records, which is a boolean field specifying if the person is adult or not."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam will try to infer the types involved in the mappings, but sometimes this is not possible. In these cases we can explicitly denote the expected output type, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult:\n",
+ " expression: \"age >= 18\"\n",
+ " output_type: boolean\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "When the `append` field is specified, one can `drop` fields as well, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " drop:\n",
+ " - age\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-03.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-03.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can also create simple UDFs (User Defined Functions) using Python or other languages. In the example below we add a field `random_number` which value is a random number not bigger than the age of the person."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " random_number:\n",
+ " callable: |\n",
+ " import random\n",
+ " def my_mapping(row):\n",
+ " return random.randrange(row.age)\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-04.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-04.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam YAML has EXPERIMENTAL ability to do aggregations to group and combine values across records. The is accomplished via the `Combine` transform type. Currently `Combine` needs to be in the `yaml_experimental_features` option (see the bottom of the pipeline) to use this transform.\n",
+ "\n",
+ "In this example we'll aggregate our records based on the `is_adult` classification. We'll calculate an average age for each of the groups."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:K\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: Combine\n",
+ " config:\n",
+ " group_by: is_adult\n",
+ " combine:\n",
+ " total:\n",
+ " value: age\n",
+ " fn:\n",
+ " type: mean\n",
Review Comment:
nit: This can be simplified.
```suggestion
" - type: Combine\n",
" config:\n",
" group_by: is_adult\n",
" combine:\n",
" total:\n",
" value: age\n",
" fn: mean\n",
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,629 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some of the available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs to elements for debugging purposes. The `LogForTesting` transform lets us see the data when developing a pipeline. Remember, it is not advised to use this transform in production."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If you scroll through the logs, you'll find entries such as:\n",
+ "```\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=1, firstname='Reeba', age=58, country='Belgium', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=2, firstname='Maud', age=45, country='Spain', profession='firefighter')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=3, firstname='Meg', age=11, country='France', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=4, firstname='Rani', age=53, country='Spain', profession='doctor')\n",
+ "```\n",
+ "This is a representation of records from the input dataset."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's add a transform - `Filter`. To use this transform you need to specify the 'keep' condition and a language your condition is written in. Below you'll find an example with a condition written in Python.\n",
+ "This pipeline will filter out records containing people that are younger than 18 years old. The only records left to the next transform will be records representing adults. Verify this by scrolling to the bottom of the output logs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Similarly, we can create a condition in other languages, for example SQL. In this example we filter out people that are younger than 18 and have a profession other than being 'unemployed'."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: sql\n",
+ " keep: \"age >= 18 or (age < 18 and profession = 'unemployed')\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Another useful transform is `MapToFields`. This transform lets us manipulate fields of a record. For example, we can add a field to our records, which is a boolean field specifying if the person is adult or not."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam will try to infer the types involved in the mappings, but sometimes this is not possible. In these cases we can explicitly denote the expected output type, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult:\n",
+ " expression: \"age >= 18\"\n",
+ " output_type: boolean\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "When the `append` field is specified, one can `drop` fields as well, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " drop:\n",
+ " - age\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-03.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-03.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can also create simple UDFs (User Defined Functions) using Python or other languages. In the example below we add a field `random_number` which value is a random number not bigger than the age of the person."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " random_number:\n",
+ " callable: |\n",
+ " import random\n",
+ " def my_mapping(row):\n",
+ " return random.randrange(row.age)\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-04.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-04.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam YAML has EXPERIMENTAL ability to do aggregations to group and combine values across records. The is accomplished via the `Combine` transform type. Currently `Combine` needs to be in the `yaml_experimental_features` option (see the bottom of the pipeline) to use this transform.\n",
+ "\n",
+ "In this example we'll aggregate our records based on the `is_adult` classification. We'll calculate an average age for each of the groups."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:K\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: Combine\n",
+ " config:\n",
+ " group_by: is_adult\n",
+ " combine:\n",
+ " total:\n",
+ " value: age\n",
+ " fn:\n",
+ " type: mean\n",
+ " - type: LogForTesting\n",
+ "options:\n",
+ " yaml_experimental_features: Combine\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-combine-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-combine-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If all was executed correctly, you should see the following lines at the bottom of the output log:\n",
+ "```\n",
+ "INFO:root:Result(is_adult=True, total=40.674418604651166)\n",
+ "INFO:root:Result(is_adult=False, total=11.714285714285714\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "All the previous pipelines were linear - output of one transform was an input to the next transform. In YAML we can also create nonlinear pipelines. To do this, we need to specify `type: composite` (or omit this line at all, this is the default mode) and specify the `name` and `input` in each of the following transforms.\n",
Review Comment:
Nit: I would point out the fact the last several examples used `chain`
All the previous pipelines were linear - output of one transform was an input to the next transform. This is also known as a `chain` pipeline. This is designated in the top-level pipeline configuration, for example:
```
pipeline:
type: chain
transforms:
...
```
In YAML we can also create nonlinear pipelines. To do this, we need to specify `type: composite` (or omit this line at all, this is the default mode) and specify the `name` and `input` in each of the following transforms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1432483211
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
Review Comment:
yes of course, sorry, the merge apparently went wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1432604187
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,629 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some of the available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs to elements for debugging purposes. The `LogForTesting` transform lets us see the data when developing a pipeline. Remember, it is not advised to use this transform in production."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If you scroll through the logs, you'll find entries such as:\n",
+ "```\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=1, firstname='Reeba', age=58, country='Belgium', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=2, firstname='Maud', age=45, country='Spain', profession='firefighter')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=3, firstname='Meg', age=11, country='France', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=4, firstname='Rani', age=53, country='Spain', profession='doctor')\n",
+ "```\n",
+ "This is a representation of records from the input dataset."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's add a transform - `Filter`. To use this transform you need to specify the 'keep' condition and a language your condition is written in. Below you'll find an example with a condition written in Python.\n",
+ "This pipeline will filter out records containing people that are younger than 18 years old. The only records left to the next transform will be records representing adults. Verify this by scrolling to the bottom of the output logs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Similarly, we can create a condition in other languages, for example SQL. In this example we filter out people that are younger than 18 and have a profession other than being 'unemployed'."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: sql\n",
+ " keep: \"age >= 18 or (age < 18 and profession = 'unemployed')\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Another useful transform is `MapToFields`. This transform lets us manipulate fields of a record. For example, we can add a field to our records, which is a boolean field specifying if the person is adult or not."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam will try to infer the types involved in the mappings, but sometimes this is not possible. In these cases we can explicitly denote the expected output type, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult:\n",
+ " expression: \"age >= 18\"\n",
+ " output_type: boolean\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "When the `append` field is specified, one can `drop` fields as well, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " drop:\n",
+ " - age\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-03.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-03.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can also create simple UDFs (User Defined Functions) using Python or other languages. In the example below we add a field `random_number` which value is a random number not bigger than the age of the person."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " random_number:\n",
+ " callable: |\n",
+ " import random\n",
+ " def my_mapping(row):\n",
+ " return random.randrange(row.age)\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-04.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-04.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam YAML has EXPERIMENTAL ability to do aggregations to group and combine values across records. The is accomplished via the `Combine` transform type. Currently `Combine` needs to be in the `yaml_experimental_features` option (see the bottom of the pipeline) to use this transform.\n",
+ "\n",
+ "In this example we'll aggregate our records based on the `is_adult` classification. We'll calculate an average age for each of the groups."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:K\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: Combine\n",
+ " config:\n",
+ " group_by: is_adult\n",
+ " combine:\n",
+ " total:\n",
+ " value: age\n",
+ " fn:\n",
+ " type: mean\n",
+ " - type: LogForTesting\n",
+ "options:\n",
+ " yaml_experimental_features: Combine\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-combine-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-combine-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If all was executed correctly, you should see the following lines at the bottom of the output log:\n",
+ "```\n",
+ "INFO:root:Result(is_adult=True, total=40.674418604651166)\n",
+ "INFO:root:Result(is_adult=False, total=11.714285714285714\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "All the previous pipelines were linear - output of one transform was an input to the next transform. In YAML we can also create nonlinear pipelines. To do this, we need to specify `type: composite` (or omit this line at all, this is the default mode) and specify the `name` and `input` in each of the following transforms.\n",
Review Comment:
good suggestion, done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1864328559
Hi @Polber, thanks for the review. All the changes are applied now.
@VeronicaWasson, could you perhaps have another look at this notebook?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1329917400
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,582 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK. You can use any text editor.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and downloads some text files from Cloud Storage to your local file system. We'll use these text files as input to the pipelines."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('wget -O data/kinglear.txt https://storage.googleapis.com/dataflow-samples/shakespeare/kinglear.txt')\n",
+ "run('wget -O data/SMSSpamCollection.csv https://storage.googleapis.com/apache-beam-samples/SMSSpamCollection/SMSSpamCollection')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with two datasets. We'll use `kinglear.txt` for the first example, and `SMSSpamCollection.csv` for the second and third examples.\n",
+ "Let's first take a look at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file that contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file that contains 5,574 rows of SMS messages labeled either spam or not-spam (\"ham\"). Each row contains two columns separated by a tab character:\n",
+ "1. `Column 1`: The label, either `ham` or `spam`\n",
+ "2. `Column 2`: The SMS message as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: Word count\n",
+ "This example is a version of the [WordCount](https://beam.apache.org/get-started/wordcount-example/)). It reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using CombinePerKey transform with the 'sum' function as a combine function,\n",
+ " # we'll calculate the occurrence of each word.\n",
+ " - type: CombinePerKey\n",
+ " config:\n",
+ " combine_fn: sum\n",
+ " name: GroupAndSum\n",
+ " input: PairWordsWith1\n",
+ "\n",
+ " # Format results - each record should be represented as 'word: count'.\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyMap\n",
+ " name: FormatResults\n",
+ " input: GroupAndSum\n",
+ " config:\n",
+ " fn: \"lambda word_count_tuple: f'{word_count_tuple[0]}: {word_count_tuple[1]}'\"\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: FormatResults\n",
+ " config:\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Each pipeline specification must start with a `pipeline` key on the first line.\n",
+ "The `pipeline` keyh is followed by a list of transforms. For example, the first transform reads the input file:\n",
+ "```\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "```\n",
+ "Note: The indentation is important, because it specifies object hierarchy.\n",
+ "YAML supports comments. Everything after the `#` is always treated as a comment. Use them to improve readability.\n",
+ "\n",
+ "Each operation must specify the `type` descriptor and other fields, such as `name` and other transform-specific parameters.\n",
+ "For a list of available transforms and their parameters, see the YAML API documentation. # todo(yaml) add link\n",
Review Comment:
Hi @amotley, this comment is also still open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1280861083
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
Review Comment:
I think we should optimize for whatever raw ipython jupyter dues. Which of these options (if any) get highlighted in a sensible way?
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
Review Comment:
Not yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1279533122
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
Review Comment:
I was thinking about the same but opted for standardization across other getting-started notebooks, where the 'run()' function is used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1247811220
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,646 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": 134,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> pip install --quiet apache-beam\n",
+ "\n",
+ ">> mkdir -p data\n",
+ "\n",
+ ">> gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt\n",
+ "Copying gs://dataflow-samples/shakespeare/kinglear.txt...\r\n",
+ "/ [1 files][153.6 KiB/153.6 KiB] \r\n",
+ "Operation completed over 1 objects/153.6 KiB. \r\n",
+ "\n",
+ ">> gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv\n",
+ "Copying gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection...\r\n",
+ "/ [1 files][466.7 KiB/466.7 KiB] \r\n",
+ "Operation completed over 1 objects/466.7 KiB. \r\n",
+ "\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 135,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/kinglear.txt\n",
+ "\tKING LEAR\r\n",
+ "\r\n",
+ "\r\n",
+ "\tDRAMATIS PERSONAE\r\n",
+ "\r\n",
+ "\r\n",
+ "LEAR\tking of Britain (KING LEAR:)\r\n",
+ "\r\n",
+ "KING OF FRANCE:\r\n",
+ "\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 136,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ ">> head data/SMSSpamCollection.csv\n",
+ "ham\tGo until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...\r\n",
+ "ham\tOk lar... Joking wif u oni...\r\n",
+ "spam\tFree entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's\r\n",
+ "ham\tU dun say so early hor... U c already then say...\r\n",
+ "ham\tNah I don't think he goes to usf, he lives around here though\r\n",
+ "spam\tFreeMsg Hey there darling it's been 3 week's now and no word back! I'd like some fun you up for it still? Tb ok! XxX std chgs to send, £1.50 to rcv\r\n",
+ "ham\tEven my brother is not like to speak with me. They treat me like aids patent.\r\n",
+ "ham\tAs per your request 'Melle Melle (Oru Minnaminunginte Nurungu Vettam)' has been set as your callertune for all Callers. Press *9 to copy your friends Callertune\r\n",
+ "spam\tWINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.\r\n",
+ "spam\tHad your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030\r\n",
+ "\n",
+ ">> wc -l data/SMSSpamCollection.csv\n",
+ "5574 data/SMSSpamCollection.csv\r\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1848398670
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1284594146
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -415,6 +415,10 @@ def _parse_window_spec(spec):
'Flatten': Flatten,
'WindowInto': WindowInto,
'GroupByKey': beam.GroupByKey,
+ 'SumPerKey': lambda: beam.CombinePerKey(sum),
Review Comment:
What would you suggest for this PR? Implement a more generic transform, such as CombineGlobally, instead of defining `TopNLargest`? What arguments would the CombineGlobally take in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1350279662
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,582 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK. You can use any text editor.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and downloads some text files from Cloud Storage to your local file system. We'll use these text files as input to the pipelines."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('wget -O data/kinglear.txt https://storage.googleapis.com/dataflow-samples/shakespeare/kinglear.txt')\n",
+ "run('wget -O data/SMSSpamCollection.csv https://storage.googleapis.com/apache-beam-samples/SMSSpamCollection/SMSSpamCollection')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with two datasets. We'll use `kinglear.txt` for the first example, and `SMSSpamCollection.csv` for the second and third examples.\n",
+ "Let's first take a look at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file that contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file that contains 5,574 rows of SMS messages labeled either spam or not-spam (\"ham\"). Each row contains two columns separated by a tab character:\n",
+ "1. `Column 1`: The label, either `ham` or `spam`\n",
+ "2. `Column 2`: The SMS message as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: Word count\n",
+ "This example is a version of the [WordCount](https://beam.apache.org/get-started/wordcount-example/)). It reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using CombinePerKey transform with the 'sum' function as a combine function,\n",
+ " # we'll calculate the occurrence of each word.\n",
+ " - type: CombinePerKey\n",
+ " config:\n",
+ " combine_fn: sum\n",
+ " name: GroupAndSum\n",
+ " input: PairWordsWith1\n",
+ "\n",
+ " # Format results - each record should be represented as 'word: count'.\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyMap\n",
+ " name: FormatResults\n",
+ " input: GroupAndSum\n",
+ " config:\n",
+ " fn: \"lambda word_count_tuple: f'{word_count_tuple[0]}: {word_count_tuple[1]}'\"\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: FormatResults\n",
+ " config:\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Each pipeline specification must start with a `pipeline` key on the first line.\n",
+ "The `pipeline` keyh is followed by a list of transforms. For example, the first transform reads the input file:\n",
+ "```\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "```\n",
+ "Note: The indentation is important, because it specifies object hierarchy.\n",
+ "YAML supports comments. Everything after the `#` is always treated as a comment. Use them to improve readability.\n",
+ "\n",
+ "Each operation must specify the `type` descriptor and other fields, such as `name` and other transform-specific parameters.\n",
+ "For a list of available transforms and their parameters, see the YAML API documentation. # todo(yaml) add link\n",
Review Comment:
Created: https://buganizer.corp.google.com/issues/304234991
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Polber commented on pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1737655145
fixes #28695
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1276608035
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
Review Comment:
Does this require any authentication? It certainly requires gcloud to be installed (which we can't just assume). As these are public, is there an http url we could use here instead?
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
Review Comment:
Maybe do
%%writefile pipeline-01.yaml
instead?
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -415,6 +415,10 @@ def _parse_window_spec(spec):
'Flatten': Flatten,
'WindowInto': WindowInto,
'GroupByKey': beam.GroupByKey,
+ 'SumPerKey': lambda: beam.CombinePerKey(sum),
Review Comment:
Oh, I see you're adding them here.
Would it be better to add a CombinePerKey (that takes a combine_fn "callable")? I'm a bit wary of adding something like TopNLargest just for this example. But I suppose right now for aggregation all we offer is SQL.
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
Review Comment:
Should we try to finalize the migration of Py* to Map with a language specification before the proliferation of tutorials?
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
Review Comment:
I'm wondering if this is preferable to `!cmd` in cells directly.
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
Review Comment:
Similarly elsewhere.
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using SumPerKey transform, we'll calculate the occurrence of each word.\n",
+ " - type: SumPerKey\n",
Review Comment:
This is just a test transform, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1301490434
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,582 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK. You can use any text editor.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and downloads some text files from Cloud Storage to your local file system. We'll use these text files as input to the pipelines."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('wget -O data/kinglear.txt https://storage.googleapis.com/dataflow-samples/shakespeare/kinglear.txt')\n",
+ "run('wget -O data/SMSSpamCollection.csv https://storage.googleapis.com/apache-beam-samples/SMSSpamCollection/SMSSpamCollection')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with two datasets. We'll use `kinglear.txt` for the first example, and `SMSSpamCollection.csv` for the second and third examples.\n",
+ "Let's first take a look at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file that contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file that contains 5,574 rows of SMS messages labeled either spam or not-spam (\"ham\"). Each row contains two columns separated by a tab character:\n",
+ "1. `Column 1`: The label, either `ham` or `spam`\n",
+ "2. `Column 2`: The SMS message as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: Word count\n",
+ "This example is a version of the [WordCount](https://beam.apache.org/get-started/wordcount-example/)). It reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using CombinePerKey transform with the 'sum' function as a combine function,\n",
+ " # we'll calculate the occurrence of each word.\n",
+ " - type: CombinePerKey\n",
+ " config:\n",
+ " combine_fn: sum\n",
+ " name: GroupAndSum\n",
+ " input: PairWordsWith1\n",
+ "\n",
+ " # Format results - each record should be represented as 'word: count'.\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyMap\n",
+ " name: FormatResults\n",
+ " input: GroupAndSum\n",
+ " config:\n",
+ " fn: \"lambda word_count_tuple: f'{word_count_tuple[0]}: {word_count_tuple[1]}'\"\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: FormatResults\n",
+ " config:\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Each pipeline specification must start with a `pipeline` key on the first line.\n",
+ "The `pipeline` keyh is followed by a list of transforms. For example, the first transform reads the input file:\n",
+ "```\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "```\n",
+ "Note: The indentation is important, because it specifies object hierarchy.\n",
+ "YAML supports comments. Everything after the `#` is always treated as a comment. Use them to improve readability.\n",
+ "\n",
+ "Each operation must specify the `type` descriptor and other fields, such as `name` and other transform-specific parameters.\n",
+ "For a list of available transforms and their parameters, see the YAML API documentation. # todo(yaml) add link\n",
Review Comment:
Hi @amotley. What should I do with this and 2 others `todo(yaml)`? Should I leave them and create tickets to not forget about replacing them once the documentation is in place? Or should rewrite these sentences and remove the placeholders for the urls?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amotley commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "amotley (via GitHub)" <gi...@apache.org>.
amotley commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1247123109
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
Review Comment:
I could be wrong, but I imagine it's just a matter of making a new page similar to that one and adding to the TOC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1247804927
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
Review Comment:
Maybe this one: https://beam.apache.org/get-started/quickstart/python/#set-up-your-development-environment
But that's not entirely fitting here since we don't want users to clone a beam-starter-python repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amotley commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "amotley (via GitHub)" <gi...@apache.org>.
amotley commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1248097967
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
Review Comment:
hmm okay, let's skip this. Maybe it's not typical to have issues installing Beam.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1611163844
R: @robertwb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1969712110
@bzablocki The docs have been pushed to the beam site, if you want to link them, I think we can get this PR merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb merged PR #27284:
URL: https://github.com/apache/beam/pull/27284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1513143678
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
Review Comment:
I'm not overly concerned with how it displays in IntelliJ (especially considering the target audience is not using an IDE anyway), and I think the lack of highlighting in raw Jupyter does not make it unclear what is going on, and makes editing the yaml easier.
For reference, this is how it looks in raw Jupyter
I would opt for the inline function - WDYT @robertwb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-2000235874
As discussed in person, it feels a bit awkward to have every command be a combination of save file + execute command in Python. Could we set the interpreter to shell instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "amotley (via GitHub)" <gi...@apache.org>.
amotley commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1344503603
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
Review Comment:
SGTM, can you use the new YAML tags that Jeff created? https://bugdashboard.corp.google.com/app/tree;dashboardId=551658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1849672015
I'll pick it up this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1430155626
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,581 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('wget -O data/kinglear.txt https://storage.googleapis.com/dataflow-samples/shakespeare/kinglear.txt')\n",
+ "run('wget -O data/SMSSpamCollection.csv https://storage.googleapis.com/apache-beam-samples/SMSSpamCollection/SMSSpamCollection')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using CombinePerKey transform with the 'sum' function as a combine function,\n",
+ " # we'll calculate the occurrence of each word.\n",
+ " - type: CombinePerKey\n",
+ " config:\n",
+ " combine_fn: sum\n",
+ " name: GroupAndSum\n",
+ " input: PairWordsWith1\n",
+ "\n",
+ " # Format results - each record should be represented as 'word: count'.\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyMap\n",
+ " name: FormatResults\n",
+ " input: GroupAndSum\n",
+ " config:\n",
+ " fn: \"lambda word_count_tuple: f'{word_count_tuple[0]}: {word_count_tuple[1]}'\"\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: FormatResults\n",
+ " config:\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Each pipeline specification has to start with a `pipeline` key on the first line.\n",
+ "Following, there is a list of transforms, such as the first one:\n",
+ "```\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "```\n",
+ "This one reads an input file for futher processing.\n",
+ "Remember that the indentation is important here - it specifies object hierarchy.\n",
+ "YAML also supports comments - everything after the `#` is always treated as a comment. Use them to improve readability.\n",
+ "\n",
+ "Each operation has to specify the `type` descriptor and other fields, such as `name` and other transform-specific parameters.\n",
+ "Take a look at the documentation for a list of available transforms and their parameters. # todo(yaml) add link\n",
+ "\n",
+ "To link two operations, use the `input` field, just like in the third operation:\n",
+ "```\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "```\n",
+ "The `input` refers to the name of another transform, the previous one in this case.\n",
+ "This particular operation takes `fn` (stands for function) as an argument. Currently only Python functions are supported.\n",
+ "For more complicated functions, you can take advantage of YAML's multiline feature, as you can see in the second operation:\n",
+ "```\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "```\n",
+ "Here we had to import Python's regex package - `re`. To do that, we indicated a multiline string with `|` character and wrote the function in 2 lines.\n",
+ "\n",
+ "Let's run the pipeline executing the Python entry-point script (`apache_beam.yaml.main`) with our pipeline file as an argument:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-01.yaml')"
Review Comment:
Hi @yardenbm,
there were some changes in the yaml api, which make this notebook outdated. I'm updating it as we speak. It should be functional by the end of this week, and merged soon after.
If you're looking for some examples on how to construct a pipeline, take a look at the Readme.md (https://github.com/apache/beam/tree/master/sdks/python/apache_beam/yaml#readme) and other .md files and unit tests in https://github.com/apache/beam/tree/master/sdks/python/apache_beam/yaml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1432602839
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
Review Comment:
The new version of the notebook doesn't use PyMap. Closing this comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1434043197
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,680 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs the elements for debugging purposes.\n",
+ "\n",
+ "The `LogForTesting` transform lets us log the data when developing a pipeline. Remember, it is not advised to use this transform in production.\n",
+ "\n",
+ "Let's define the pipeline and save it to a file:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We can verify the contents of this file by running:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "! cat pipelines/pipeline-01.yaml"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Now, we can execute the yaml pipeline by passing this file as an argument to the following command:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here we use Python and `apache_beam` package to execute the pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for that in the future.\n",
+ "\n",
+ "If you scroll through the output logs, you'll find entries such as:\n",
+ "```\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=1, firstname='Reeba', age=58, country='Belgium', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=2, firstname='Maud', age=45, country='Spain', profession='firefighter')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=3, firstname='Meg', age=11, country='France', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=4, firstname='Rani', age=53, country='Spain', profession='doctor')\n",
+ "```\n",
+ "This is a representation of records from the input dataset."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's add a transform - `Filter`. To use this transform you need to specify the 'keep' condition and a language your condition is written in. Below you'll find an example with a condition written in Python.\n",
+ "This pipeline will filter out records containing people that are younger than 18 years old. The only records left to the next transform will be records representing adults. Verify this by scrolling to the bottom of the output logs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Similarly, we can create a condition in other languages, for example SQL. In this example we filter out people that are younger than 18 and have a profession other than being 'unemployed'."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: sql\n",
+ " keep: \"age >= 18 or (age < 18 and profession = 'unemployed')\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Another useful transform is `MapToFields`. This transform lets us manipulate fields of a record. For example, we can add a field to our records, which is a boolean field specifying if the person is adult or not."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam will try to infer the types involved in the mappings, but sometimes this is not possible. In these cases we can explicitly denote the expected output type, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult:\n",
+ " expression: \"age >= 18\"\n",
+ " output_type: boolean\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "When the `append` field is specified, one can `drop` fields as well, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " drop:\n",
+ " - age\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-03.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-03.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can also create simple UDFs (User Defined Functions) using Python or other languages. In the example below we add a field `random_number` which value is a random number not bigger than the age of the person."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " random_number:\n",
+ " callable: |\n",
+ " import random\n",
+ " def my_mapping(row):\n",
+ " return random.randrange(row.age)\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-04.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-04.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam YAML has EXPERIMENTAL ability to do aggregations to group and combine values across records. The is accomplished via the `Combine` transform type. Currently `Combine` needs to be in the `yaml_experimental_features` option (see the bottom of the pipeline) to use this transform.\n",
+ "\n",
+ "In this example we'll aggregate our records based on the `is_adult` classification. We'll calculate an average age for each of the groups."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: Combine\n",
+ " config:\n",
+ " group_by: is_adult\n",
+ " combine:\n",
+ " total:\n",
+ " value: age\n",
+ " fn: mean\n",
+ " - type: LogForTesting\n",
+ "options:\n",
+ " yaml_experimental_features: Combine\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-combine-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-combine-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If all was executed correctly, you should see the following lines at the bottom of the output log:\n",
+ "```\n",
+ "INFO:root:Result(is_adult=True, total=40.674418604651166)\n",
+ "INFO:root:Result(is_adult=False, total=11.714285714285714\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "All the previous pipelines were linear - output of one transform was an input to the next transform. This is also known as a `chain` pipeline. This is designated in the top-level pipeline configuration, for example:\n",
+ "```\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " ...\n",
+ "```\n",
+ "In YAML we can also create nonlinear pipelines. To do this, we need to specify `type: composite` (or omit this line at all, this is the default mode) and specify the `name` and `input` in each of the following transforms.\n",
+ "The specification below will create the following pipeline:\n",
+ "```\n",
+ " +----> Doctors -----------> SaveDoctors\n",
+ "InputData ---+\n",
+ " +----> OtherProfessions --> SaveOtherProfessions\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: composite\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " name: InputData\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " name: Doctors\n",
+ " input: InputData\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"profession == 'doctor'\"\n",
+ " - type: Filter\n",
+ " name: OtherProfessions\n",
+ " input: InputData\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"profession != 'doctor'\"\n",
+ " - type: WriteToCsv\n",
+ " name: SaveDoctors\n",
+ " input: Doctors\n",
+ " config:\n",
+ " path: results/doctors\n",
+ " - type: WriteToCsv\n",
+ " name: SaveOtherProfessions\n",
+ " input: OtherProfessions\n",
+ " config:\n",
+ " path: results/other-professions\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-nonlinear-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-nonlinear-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The output are 2 files: `results/doctors-00000-of-00001` and `results/other-professions-00000-of-00001`. Let's see their contents:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head results/doctors-00000-of-00001"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head results/other-professions-00000-of-00001"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "Congratulations! You've just run Apache Beam pipelines using YAML.\n",
+ "\n",
+ "For all the available operations visit the documentation: # todo(yaml) add url\n",
+ "\n",
+ "For a list of available transforms, visit # todo(yaml) add url\n",
+ "\n",
+ "To run your pipeline in Dataflow, you'll need to set up your Google Cloud and run the pipeline with the `DataflowRunner`. For more information, follow https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#run-on-dataflow"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [],
+ "metadata": {
+ "collapsed": false
+ }
+ }
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] VeronicaWasson commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "VeronicaWasson (via GitHub)" <gi...@apache.org>.
VeronicaWasson commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1297751839
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,581 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('wget -O data/kinglear.txt https://storage.googleapis.com/dataflow-samples/shakespeare/kinglear.txt')\n",
+ "run('wget -O data/SMSSpamCollection.csv https://storage.googleapis.com/apache-beam-samples/SMSSpamCollection/SMSSpamCollection')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using CombinePerKey transform with the 'sum' function as a combine function,\n",
+ " # we'll calculate the occurrence of each word.\n",
+ " - type: CombinePerKey\n",
+ " config:\n",
+ " combine_fn: sum\n",
+ " name: GroupAndSum\n",
+ " input: PairWordsWith1\n",
+ "\n",
+ " # Format results - each record should be represented as 'word: count'.\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyMap\n",
+ " name: FormatResults\n",
+ " input: GroupAndSum\n",
+ " config:\n",
+ " fn: \"lambda word_count_tuple: f'{word_count_tuple[0]}: {word_count_tuple[1]}'\"\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: FormatResults\n",
+ " config:\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Each pipeline specification has to start with a `pipeline` key on the first line.\n",
+ "Following, there is a list of transforms, such as the first one:\n",
+ "```\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "```\n",
+ "This one reads an input file for futher processing.\n",
+ "Remember that the indentation is important here - it specifies object hierarchy.\n",
+ "YAML also supports comments - everything after the `#` is always treated as a comment. Use them to improve readability.\n",
+ "\n",
+ "Each operation has to specify the `type` descriptor and other fields, such as `name` and other transform-specific parameters.\n",
+ "Take a look at the documentation for a list of available transforms and their parameters. # todo(yaml) add link\n",
+ "\n",
+ "To link two operations, use the `input` field, just like in the third operation:\n",
+ "```\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "```\n",
+ "The `input` refers to the name of another transform, the previous one in this case.\n",
+ "This particular operation takes `fn` (stands for function) as an argument. Currently only Python functions are supported.\n",
+ "For more complicated functions, you can take advantage of YAML's multiline feature, as you can see in the second operation:\n",
+ "```\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "```\n",
+ "Here we had to import Python's regex package - `re`. To do that, we indicated a multiline string with `|` character and wrote the function in 2 lines.\n",
+ "\n",
+ "Let's run the pipeline executing the Python entry-point script (`apache_beam.yaml.main`) with our pipeline file as an argument:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-01.yaml')"
Review Comment:
I get an exception here:
ValueError: Invalid transform specification at "InputText" at line 4: PTransform.__init__() got an unexpected argument 'config'
(Previous steps seemed to work.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on PR #27284:
URL: https://github.com/apache/beam/pull/27284#issuecomment-1927195264
Thanks for taking another look at this PR @Polber!
> Hey @bzablocki We are currently trying to get some docs onto the Beam website which I think would be a good idea to get submitted first so that users have docs to reference after trying out the day zero notebook.
That makes a lot of sense, I agree it's better to release some docs first. Thanks for the update. I'll wait for the documentation and I'll update this PR with the relevant references once that is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1475289458
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,671 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs the elements for debugging purposes.\n",
+ "\n",
+ "The `LogForTesting` transform lets us log the data when developing a pipeline. Remember, it is not advised to use this transform in production.\n",
+ "\n",
+ "Let's define the pipeline and save it to a file:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We can verify the contents of this file by running:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "! cat pipelines/pipeline-01.yaml"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Now, we can execute the yaml pipeline by passing this file as an argument to the following command:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here we use Python and `apache_beam` package to execute the pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for that in the future.\n",
+ "\n",
+ "If you scroll through the output logs, you'll find entries such as:\n",
+ "```\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=1, firstname='Reeba', age=58, country='Belgium', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=2, firstname='Maud', age=45, country='Spain', profession='firefighter')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=3, firstname='Meg', age=11, country='France', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=4, firstname='Rani', age=53, country='Spain', profession='doctor')\n",
+ "```\n",
+ "This is a representation of records from the input dataset."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's add a transform - `Filter`. To use this transform you need to specify the 'keep' condition and a language your condition is written in. Below you'll find an example with a condition written in Python.\n",
+ "This pipeline will filter out records containing people that are younger than 18 years old. The only records left to the next transform will be records representing adults. Verify this by scrolling to the bottom of the output logs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-01.yaml"
Review Comment:
In which case I would move the run command to its own cell.
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,671 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs the elements for debugging purposes.\n",
+ "\n",
+ "The `LogForTesting` transform lets us log the data when developing a pipeline. Remember, it is not advised to use this transform in production.\n",
+ "\n",
+ "Let's define the pipeline and save it to a file:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')"
Review Comment:
Colab has a built-in writefile method that I think is less messy and allows makes specifying string literals in YAML easier
```suggestion
"%%writefile pipelines/pipeline-01.yaml\n",
"\n",
"pipeline:\n",
" type: chain\n",
" transforms:\n",
" - type: ReadFromCsv\n",
" config:\n",
" path: data/people.csv\n",
" - type: LogForTesting\n"
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,671 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
Review Comment:
I believe you can get rid of this with my other comments
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,671 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs the elements for debugging purposes.\n",
+ "\n",
+ "The `LogForTesting` transform lets us log the data when developing a pipeline. Remember, it is not advised to use this transform in production.\n",
+ "\n",
+ "Let's define the pipeline and save it to a file:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We can verify the contents of this file by running:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "! cat pipelines/pipeline-01.yaml"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Now, we can execute the yaml pipeline by passing this file as an argument to the following command:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here we use Python and `apache_beam` package to execute the pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for that in the future.\n",
+ "\n",
+ "If you scroll through the output logs, you'll find entries such as:\n",
+ "```\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=1, firstname='Reeba', age=58, country='Belgium', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=2, firstname='Maud', age=45, country='Spain', profession='firefighter')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=3, firstname='Meg', age=11, country='France', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=4, firstname='Rani', age=53, country='Spain', profession='doctor')\n",
+ "```\n",
+ "This is a representation of records from the input dataset."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's add a transform - `Filter`. To use this transform you need to specify the 'keep' condition and a language your condition is written in. Below you'll find an example with a condition written in Python.\n",
+ "This pipeline will filter out records containing people that are younger than 18 years old. The only records left to the next transform will be records representing adults. Verify this by scrolling to the bottom of the output logs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-01.yaml"
Review Comment:
Same here.
```suggestion
"%%writefile pipelines/pipeline-filter-01.yaml\n",
"pipeline:\n",
" type: chain\n",
" transforms:\n",
" - type: ReadFromCsv\n",
" config:\n",
" path: data/people.csv\n",
" - type: Filter\n",
" config:\n",
" language: python\n",
" keep: \"age >= 18\"\n",
" - type: LogForTesting\n"
```
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,671 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs the elements for debugging purposes.\n",
+ "\n",
+ "The `LogForTesting` transform lets us log the data when developing a pipeline. Remember, it is not advised to use this transform in production.\n",
+ "\n",
+ "Let's define the pipeline and save it to a file:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We can verify the contents of this file by running:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "! cat pipelines/pipeline-01.yaml"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Now, we can execute the yaml pipeline by passing this file as an argument to the following command:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here we use Python and `apache_beam` package to execute the pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for that in the future.\n",
+ "\n",
+ "If you scroll through the output logs, you'll find entries such as:\n",
+ "```\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=1, firstname='Reeba', age=58, country='Belgium', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=2, firstname='Maud', age=45, country='Spain', profession='firefighter')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=3, firstname='Meg', age=11, country='France', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=4, firstname='Rani', age=53, country='Spain', profession='doctor')\n",
+ "```\n",
+ "This is a representation of records from the input dataset."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's add a transform - `Filter`. To use this transform you need to specify the 'keep' condition and a language your condition is written in. Below you'll find an example with a condition written in Python.\n",
+ "This pipeline will filter out records containing people that are younger than 18 years old. The only records left to the next transform will be records representing adults. Verify this by scrolling to the bottom of the output logs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Similarly, we can create a condition in other languages, for example SQL. In this example we filter out people that are younger than 18 and have a profession other than being 'unemployed'."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: sql\n",
+ " keep: \"age >= 18 or (age < 18 and profession = 'unemployed')\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-02.yaml"
Review Comment:
Same here (and subsequent Yaml pipeline blocks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] VeronicaWasson commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "VeronicaWasson (via GitHub)" <gi...@apache.org>.
VeronicaWasson commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1297754706
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,581 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('wget -O data/kinglear.txt https://storage.googleapis.com/dataflow-samples/shakespeare/kinglear.txt')\n",
+ "run('wget -O data/SMSSpamCollection.csv https://storage.googleapis.com/apache-beam-samples/SMSSpamCollection/SMSSpamCollection')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using CombinePerKey transform with the 'sum' function as a combine function,\n",
+ " # we'll calculate the occurrence of each word.\n",
+ " - type: CombinePerKey\n",
+ " config:\n",
+ " combine_fn: sum\n",
+ " name: GroupAndSum\n",
+ " input: PairWordsWith1\n",
+ "\n",
+ " # Format results - each record should be represented as 'word: count'.\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyMap\n",
+ " name: FormatResults\n",
+ " input: GroupAndSum\n",
+ " config:\n",
+ " fn: \"lambda word_count_tuple: f'{word_count_tuple[0]}: {word_count_tuple[1]}'\"\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: FormatResults\n",
+ " config:\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Each pipeline specification has to start with a `pipeline` key on the first line.\n",
+ "Following, there is a list of transforms, such as the first one:\n",
+ "```\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "```\n",
+ "This one reads an input file for futher processing.\n",
+ "Remember that the indentation is important here - it specifies object hierarchy.\n",
+ "YAML also supports comments - everything after the `#` is always treated as a comment. Use them to improve readability.\n",
+ "\n",
+ "Each operation has to specify the `type` descriptor and other fields, such as `name` and other transform-specific parameters.\n",
+ "Take a look at the documentation for a list of available transforms and their parameters. # todo(yaml) add link\n",
+ "\n",
+ "To link two operations, use the `input` field, just like in the third operation:\n",
+ "```\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "```\n",
+ "The `input` refers to the name of another transform, the previous one in this case.\n",
+ "This particular operation takes `fn` (stands for function) as an argument. Currently only Python functions are supported.\n",
+ "For more complicated functions, you can take advantage of YAML's multiline feature, as you can see in the second operation:\n",
+ "```\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "```\n",
+ "Here we had to import Python's regex package - `re`. To do that, we indicated a multiline string with `|` character and wrote the function in 2 lines.\n",
+ "\n",
+ "Let's run the pipeline executing the Python entry-point script (`apache_beam.yaml.main`) with our pipeline file as an argument:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's inspect the results. Each line contains a word and an associated count."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/result-pipeline-01-00000-of-00001.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 2: load data, filter unwanted lines and save results to a text file.\n",
+ "In this example, we'll create a pipeline which loads a `.csv` file containing SMS messages, filters out valid messages, and saves only valid sms messages to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " config:\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " # Split each line into an array, where the first element is message label (ham or spam) and the second is the content of the message.\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " input: SmsData\n",
+ " config:\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " # Keep only the rows that contain spam messages, based on the first element in the array - the label.\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " input: SplitLine\n",
+ " config:\n",
+ " keep: 'lambda row: row[0] == \"spam\"' # this is a function in Python, similar to the 'fn' in the previous example.\n",
+ "\n",
+ " # Save only the rows from the input file that are classified as spam.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: KeepSpam\n",
+ " config:\n",
+ " file_path_prefix: \"data/result-pipeline-02\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-02.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's run the pipeline with our `.yaml` file as an input:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-02.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's verify the results and see the content of the output file."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/result-pipeline-02-00000-of-00001.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "If everything went well, you should see only spam messages from our input dataset. Congratulations, onto the next example!\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 3: count words in spam messages, select top 10 popular words and write results to a file.\n",
+ "\n",
+ "We'd like to write a pipeline which counts words occurring in spam messages, selects the most popular ones and writes the result to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " config:\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " # Split each line into an array, where the first element is message label (ham or spam) and the second is the content of the message.\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " input: SmsData\n",
+ " config:\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " # Keep only the rows that contain spam messages, based on the first element in the array - the label.\n",
+ " - type: PyFilter\n",
+ " name: SpamMessages\n",
+ " input: SplitLine\n",
+ " config:\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings)\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: SpamMessages\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line[1])\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairLoweredWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word.lower(), 1)'\n",
+ "\n",
+ " # Using CombinePerKey transform with the 'sum' function as a combine function,\n",
+ " # we'll calculate the occurrence of each word.\n",
+ " - type: CombinePerKey\n",
+ " config:\n",
+ " combine_fn: sum\n",
+ " name: GroupAndSum\n",
+ " input: PairLoweredWordsWith1\n",
+ "\n",
+ " # Select 10 most popular words. Input format to this step is a tuple (word: count),\n",
+ " # so we provide the count (row[1]) as the key to compare the numbers\n",
+ " - type: TopNLargest\n",
+ " name: MostPopular\n",
+ " input: GroupAndSum\n",
+ " config:\n",
+ " n: 10\n",
+ " key: 'lambda row: row[1]'\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: MostPopular\n",
+ " config:\n",
+ " file_path_prefix: \"data/result-pipeline-03\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-03.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's run the pipeline:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-03.yaml')"
Review Comment:
Here I get an exception:
ValueError: Invalid transform specification at "SmsData" at line 4: PTransform.__init__() got an unexpected argument 'config'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1279506869
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
Review Comment:
Good point that the gcloud might not be installed, I copy-pasted this from the try-apache-beam-py.ipynb. The data is public, so I'll change it to a wget command. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amotley commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "amotley (via GitHub)" <gi...@apache.org>.
amotley commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1300290541
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,582 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK. You can use any text editor.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and downloads some text files from Cloud Storage to your local file system. We'll use these text files as input to the pipelines."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('wget -O data/kinglear.txt https://storage.googleapis.com/dataflow-samples/shakespeare/kinglear.txt')\n",
+ "run('wget -O data/SMSSpamCollection.csv https://storage.googleapis.com/apache-beam-samples/SMSSpamCollection/SMSSpamCollection')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with two datasets. We'll use `kinglear.txt` for the first example, and `SMSSpamCollection.csv` for the second and third examples.\n",
+ "Let's first take a look at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file that contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file that contains 5,574 rows of SMS messages labeled either spam or not-spam (\"ham\"). Each row contains two columns separated by a tab character:\n",
+ "1. `Column 1`: The label, either `ham` or `spam`\n",
+ "2. `Column 2`: The SMS message as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: Word count\n",
+ "This example is a version of the [WordCount](https://beam.apache.org/get-started/wordcount-example/)). It reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " config:\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " config:\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " config:\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using CombinePerKey transform with the 'sum' function as a combine function,\n",
+ " # we'll calculate the occurrence of each word.\n",
+ " - type: CombinePerKey\n",
+ " config:\n",
+ " combine_fn: sum\n",
+ " name: GroupAndSum\n",
+ " input: PairWordsWith1\n",
+ "\n",
+ " # Format results - each record should be represented as 'word: count'.\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyMap\n",
+ " name: FormatResults\n",
+ " input: GroupAndSum\n",
+ " config:\n",
+ " fn: \"lambda word_count_tuple: f'{word_count_tuple[0]}: {word_count_tuple[1]}'\"\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: FormatResults\n",
+ " config:\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Each pipeline specification must start with a `pipeline` key on the first line.\n",
+ "The `pipeline` keyh is followed by a list of transforms. For example, the first transform reads the input file:\n",
Review Comment:
```suggestion
"The `pipeline` key is followed by a list of transforms. For example, the first transform reads the input file:\n",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1432581031
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,629 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some of the available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs to elements for debugging purposes. The `LogForTesting` transform lets us see the data when developing a pipeline. Remember, it is not advised to use this transform in production."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If you scroll through the logs, you'll find entries such as:\n",
+ "```\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=1, firstname='Reeba', age=58, country='Belgium', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=2, firstname='Maud', age=45, country='Spain', profession='firefighter')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=3, firstname='Meg', age=11, country='France', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=4, firstname='Rani', age=53, country='Spain', profession='doctor')\n",
+ "```\n",
+ "This is a representation of records from the input dataset."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's add a transform - `Filter`. To use this transform you need to specify the 'keep' condition and a language your condition is written in. Below you'll find an example with a condition written in Python.\n",
+ "This pipeline will filter out records containing people that are younger than 18 years old. The only records left to the next transform will be records representing adults. Verify this by scrolling to the bottom of the output logs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Similarly, we can create a condition in other languages, for example SQL. In this example we filter out people that are younger than 18 and have a profession other than being 'unemployed'."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: sql\n",
+ " keep: \"age >= 18 or (age < 18 and profession = 'unemployed')\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Another useful transform is `MapToFields`. This transform lets us manipulate fields of a record. For example, we can add a field to our records, which is a boolean field specifying if the person is adult or not."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam will try to infer the types involved in the mappings, but sometimes this is not possible. In these cases we can explicitly denote the expected output type, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult:\n",
+ " expression: \"age >= 18\"\n",
+ " output_type: boolean\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "When the `append` field is specified, one can `drop` fields as well, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " drop:\n",
+ " - age\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-03.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-03.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can also create simple UDFs (User Defined Functions) using Python or other languages. In the example below we add a field `random_number` which value is a random number not bigger than the age of the person."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " random_number:\n",
+ " callable: |\n",
+ " import random\n",
+ " def my_mapping(row):\n",
+ " return random.randrange(row.age)\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-04.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-04.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam YAML has EXPERIMENTAL ability to do aggregations to group and combine values across records. The is accomplished via the `Combine` transform type. Currently `Combine` needs to be in the `yaml_experimental_features` option (see the bottom of the pipeline) to use this transform.\n",
+ "\n",
+ "In this example we'll aggregate our records based on the `is_adult` classification. We'll calculate an average age for each of the groups."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:K\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: Combine\n",
+ " config:\n",
+ " group_by: is_adult\n",
+ " combine:\n",
+ " total:\n",
+ " value: age\n",
+ " fn:\n",
+ " type: mean\n",
Review Comment:
Thanks, adjusted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Yaml API: Day Zero tutorial notebook [beam]
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1432580553
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,629 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form"
+ },
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ },
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be a challenge.\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "It should be noted that everything here is still under development, but any features already included are considered stable. Feedback is welcome at dev@apache.beam.org.\n",
+ "\n",
+ "In this notebook, you set up your development environment and write a simple pipeline using YAML. Then you run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, click the **Run cell** button at the top left of the cell, or select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment. The following code installs `apache-beam` and creates directories for your data, pipelines and results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ },
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c"
+ },
+ "outputs": [],
+ "source": [
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "! pip install --quiet apache-beam\n",
+ "\n",
+ "# Create a directory for storing the data, pipelines and results\n",
+ "! mkdir -p data\n",
+ "! mkdir -p pipelines\n",
+ "! mkdir -p results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll also create an artificial dataset that represents a simple database. The csv file contains information about different people. Each line represents a single person and their details are separated by commas. The file contains 5 columns: id, firstname, age, country and a profession."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "csv_data= '''\n",
+ "id,firstname,age,country,profession\n",
+ "1,Reeba,58,Belgium,unemployed\n",
+ "2,Maud,45,Spain,firefighter\n",
+ "3,Meg,11,France,unemployed\n",
+ "4,Rani,53,Spain,doctor\n",
+ "5,Natka,26,France,doctor\n",
+ "6,Aurore,32,Italy,police officer\n",
+ "7,Elvira,39,Italy,doctor\n",
+ "8,Asia,10,Belgium,doctor\n",
+ "9,Lesly,35,Spain,firefighter\n",
+ "10,Orelia,31,Germany,police officer\n",
+ "11,Theodora,16,Italy,unemployed\n",
+ "12,Viviene,44,Germany,police officer\n",
+ "13,Teriann,50,Belgium,police officer\n",
+ "14,Carol-Jean,23,Germany,unemployed\n",
+ "15,Drucie,15,Spain,police officer\n",
+ "16,Elie,10,Italy,doctor\n",
+ "17,Shaylyn,34,Belgium,worker\n",
+ "18,Fayre,33,Spain,police officer\n",
+ "19,Sabina,52,Germany,police officer\n",
+ "20,Aryn,20,Belgium,police officer\n",
+ "21,Darlleen,49,Spain,worker\n",
+ "22,Jere,18,Italy,worker\n",
+ "23,Candi,60,Germany,police officer\n",
+ "24,Sindee,40,Germany,firefighter\n",
+ "25,Selma,20,Spain,worker\n",
+ "26,Vonny,35,Germany,doctor\n",
+ "27,Kate,53,Spain,worker\n",
+ "28,Annabela,48,Belgium,worker\n",
+ "29,Jenilee,55,Germany,police officer\n",
+ "30,Gusella,44,France,police officer\n",
+ "31,Fawne,35,Spain,worker\n",
+ "32,Karolina,39,Spain,police officer\n",
+ "33,Sadie,58,Germany,firefighter\n",
+ "34,Clo,10,Italy,police officer\n",
+ "35,Beth,46,Spain,firefighter\n",
+ "36,Adore,18,Italy,firefighter\n",
+ "37,Tarra,29,Spain,doctor\n",
+ "38,Jessamyn,36,France,police officer\n",
+ "39,Deedee,24,Germany,unemployed\n",
+ "40,Patricia,45,Italy,doctor\n",
+ "41,Wileen,39,Spain,police officer\n",
+ "42,Paola,55,Italy,worker\n",
+ "43,Gwyneth,37,Italy,worker\n",
+ "44,Stacey,36,Spain,worker\n",
+ "45,Camile,60,Germany,unemployed\n",
+ "46,Sheree,10,Spain,unemployed\n",
+ "47,Albertina,53,France,police officer\n",
+ "48,Jinny,30,Spain,firefighter\n",
+ "49,Kayla,57,Italy,firefighter\n",
+ "50,Jaime,55,France,doctor\n",
+ "'''\n",
+ "\n",
+ "save_to_file(csv_data, 'data/people.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's validate if the file was created correctly. You should see the first few lines from the generated file. Validate if the beginning of the file matches with the declared content above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/people.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Your first YAML pipelines\n",
+ "\n",
+ "In this section we'll present you the basic structure of a YAML pipeline and present you some of the available transforms.\n",
+ "Below is a simple pipeline that reads data from the csv file we've just created and logs to elements for debugging purposes. The `LogForTesting` transform lets us see the data when developing a pipeline. Remember, it is not advised to use this transform in production."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If you scroll through the logs, you'll find entries such as:\n",
+ "```\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=1, firstname='Reeba', age=58, country='Belgium', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=2, firstname='Maud', age=45, country='Spain', profession='firefighter')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=3, firstname='Meg', age=11, country='France', profession='unemployed')\n",
+ "INFO:root:BeamSchema_edf39b51_91da_418a_b28e_af04c9bae811(id=4, firstname='Rani', age=53, country='Spain', profession='doctor')\n",
+ "```\n",
+ "This is a representation of records from the input dataset."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's add a transform - `Filter`. To use this transform you need to specify the 'keep' condition and a language your condition is written in. Below you'll find an example with a condition written in Python.\n",
+ "This pipeline will filter out records containing people that are younger than 18 years old. The only records left to the next transform will be records representing adults. Verify this by scrolling to the bottom of the output logs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: python\n",
+ " keep: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Similarly, we can create a condition in other languages, for example SQL. In this example we filter out people that are younger than 18 and have a profession other than being 'unemployed'."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: Filter\n",
+ " config:\n",
+ " language: sql\n",
+ " keep: \"age >= 18 or (age < 18 and profession = 'unemployed')\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-filter-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Another useful transform is `MapToFields`. This transform lets us manipulate fields of a record. For example, we can add a field to our records, which is a boolean field specifying if the person is adult or not."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-01.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-01.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam will try to infer the types involved in the mappings, but sometimes this is not possible. In these cases we can explicitly denote the expected output type, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult:\n",
+ " expression: \"age >= 18\"\n",
+ " output_type: boolean\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-02.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-02.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "When the `append` field is specified, one can `drop` fields as well, e.g."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " is_adult: \"age >= 18\"\n",
+ " drop:\n",
+ " - age\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-03.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-03.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can also create simple UDFs (User Defined Functions) using Python or other languages. In the example below we add a field `random_number` which value is a random number not bigger than the age of the person."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:\n",
+ " random_number:\n",
+ " callable: |\n",
+ " import random\n",
+ " def my_mapping(row):\n",
+ " return random.randrange(row.age)\n",
+ " - type: LogForTesting\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipelines/pipeline-map-04.yaml')\n",
+ "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-map-04.yaml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Beam YAML has EXPERIMENTAL ability to do aggregations to group and combine values across records. The is accomplished via the `Combine` transform type. Currently `Combine` needs to be in the `yaml_experimental_features` option (see the bottom of the pipeline) to use this transform.\n",
+ "\n",
+ "In this example we'll aggregate our records based on the `is_adult` classification. We'll calculate an average age for each of the groups."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromCsv\n",
+ " config:\n",
+ " path: data/people.csv\n",
+ " - type: MapToFields\n",
+ " config:\n",
+ " language: python\n",
+ " append: true\n",
+ " fields:K\n",
Review Comment:
Good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1280862019
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
Review Comment:
OK. (By "other" getting-started notebooks, do you mean across Beam, or the wider GCP (or beyond) ecosystem?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1280863071
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -415,6 +415,10 @@ def _parse_window_spec(spec):
'Flatten': Flatten,
'WindowInto': WindowInto,
'GroupByKey': beam.GroupByKey,
+ 'SumPerKey': lambda: beam.CombinePerKey(sum),
Review Comment:
Given everything is put into the global namespace, I think we should be judicious. Note that TopNLargest is just CombineGlobally(TopN).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1284569306
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input files into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/kinglear.txt')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "We'll be working with 2 datasets. We'll use `kinglear.txt` for the first example - word count, and `SMSSpamCollection.csv` for the second and third.\n",
+ "Let's first take a loot at the `kinglear.txt` dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/kinglear.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This is just a `txt` file - it contains lines of text.\n",
+ "Let's take a look at the other dataset."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Example 1: word count\n",
+ "In this popular introductory exercise, we will build a pipeline that reads lines of text from the input dataset `kinglear.txt` and counts the number of times each word appears in the text.\n",
+ "To start, we'll create a `.yaml` file specifying our pipeline."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " # Read input data. Each line from the txt file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: InputText\n",
+ " file_pattern: data/kinglear.txt\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings).\n",
+ " # The 'fn' parameter accepts functions written in Python\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " input: InputText\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line)\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairWordsWith1\n",
+ " input: FindWords\n",
+ " fn: 'lambda word: (word, 1)'\n",
+ "\n",
+ " # Using SumPerKey transform, we'll calculate the occurrence of each word.\n",
+ " - type: SumPerKey\n",
Review Comment:
Removed, changed to CombinePerKey with a callable argument
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1284590242
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,556 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - YAML\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it still has a high barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate.\n",
+ "\n",
+ "Here we provide a simple YAML syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do.\n",
+ "\n",
+ "Please note: YAML API is still EXPERIMENTAL and subject to change.\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using YAML. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading files from Cloud Storage to your local file system. We'll use these files as an input to the pipelines in this guide."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
Review Comment:
Other getting started notebooks in the same directory ( [try-apache-beam-java.ipynb](https://github.com/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-java.ipynb), [try-apache-beam-py.ipynb](https://github.com/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-py.ipynb), [try-apache-beam-go.ipynb](https://github.com/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-go.ipynb) )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bzablocki commented on a diff in pull request #27284: Yaml API: Day Zero tutorial notebook
Posted by "bzablocki (via GitHub)" <gi...@apache.org>.
bzablocki commented on code in PR #27284:
URL: https://github.com/apache/beam/pull/27284#discussion_r1246486332
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
Review Comment:
That would be great! Is this something @robertwb can arrange?
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input file into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "Let’s see how our data looks like."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## First pipeline\n",
+ "We’ll start with creating a pipeline which loads the data, filters out valid messages leaving spam, and saves only valid lines to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " input: SmsData\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " input: SplitLine\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: KeepSpam\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "In this example, each transformation contains the 'input' key, but if the pipeline is linear, such as ours, we can let the inputs be implicit by designating the pipeline as a `chain` type.\n"
Review Comment:
In the current implementation we need to specify `type: chain` if we don't want to pass the `input` parameter to every transform. So we have 2 options:
1) pipeline with 'input' specification in every transform:
```
pipeline:
- type: ReadFromText
name: InputText
file_pattern: data/kinglear.txt
- type: PyFlatMap
name: FindWords
input: InputText
fn: |
import re
lambda line: re.findall(r"[a-zA-Z]+", line)
- type: PyMap
name: PairWordsWith1
input: FindWords
fn: 'lambda word: (word, 1)'
```
2) Specify 'type: chain' and specify transforms under the 'transforms' key
```
pipeline:
type: chain
transforms:
- type: ReadFromText
name: InputText
file_pattern: data/kinglear.txt
- type: PyFlatMap
name: FindWords
fn: |
import re
lambda line: re.findall(r"[a-zA-Z]+", line)
- type: PyMap
name: PairWordsWith1
fn: 'lambda word: (word, 1)'
```
3) The third option is to change the implementation to default to 'type: chain' when there is no type specified, but that is for another PR. I imagine it would look like this:
```
pipeline:
- type: ReadFromText
name: InputText
file_pattern: data/kinglear.txt
- type: PyFlatMap
name: FindWords
fn: |
import re
lambda line: re.findall(r"[a-zA-Z]+", line)
- type: PyMap
name: PairWordsWith1
fn: 'lambda word: (word, 1)'
```
What do you think @amotley and @svetakvsundhar ?
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input file into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "Let’s see how our data looks like."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## First pipeline\n",
+ "We’ll start with creating a pipeline which loads the data, filters out valid messages leaving spam, and saves only valid lines to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " input: SmsData\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " input: SplitLine\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: KeepSpam\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "In this example, each transformation contains the 'input' key, but if the pipeline is linear, such as ours, we can let the inputs be implicit by designating the pipeline as a `chain` type.\n"
Review Comment:
I rewrote the text around the `type:chain`, please take another look.
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "def save_to_file(content, file_name):\n",
+ " with open(file_name, 'w') as f:\n",
+ " f.write(content)\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input file into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://apache-beam-samples/SMSSpamCollection/SMSSpamCollection data/SMSSpamCollection.csv')"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Inspect the data\n",
+ "Let’s see how our data looks like."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/SMSSpamCollection.csv')\n",
+ "run('wc -l data/SMSSpamCollection.csv')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This dataset is a `csv` file with 5,574 rows and 2 columns recording the following attributes separated by a tab sign:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## First pipeline\n",
+ "We’ll start with creating a pipeline which loads the data, filters out valid messages leaving spam, and saves only valid lines to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " input: SmsData\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " input: SplitLine\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " input: KeepSpam\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "In this example, each transformation contains the 'input' key, but if the pipeline is linear, such as ours, we can let the inputs be implicit by designating the pipeline as a `chain` type.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ " transforms:\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " - type: PyFilter\n",
+ " name: KeepSpam\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " file_path_prefix: \"data/result-pipeline-01\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-01-chain.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To run the pipeline locally, using a DirectRunner, you need to run the yaml's main python script, passing the `pipeline-01-chain.yaml` (or `pipeline-01.yaml`) file as an input:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-01-chain.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's verify the results and see the content of the output file."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/result-pipeline-01-00000-of-00001.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "If everything went well, you should see only spam messages from our input dataset. Congratulations, onto the next example!\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Count words in spam messages, select top 10 popular words and write results to a file\n",
+ "\n",
+ "We'd like to write a pipeline which counts words occurring in spam messages, selects the most popular ones and writes the result to a file.\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "pipeline = '''\n",
+ "pipeline:\n",
+ " type: chain\n",
+ "\n",
+ " transforms:\n",
+ " # Read input data. Each line from the csv file is a String.\n",
+ " - type: ReadFromText\n",
+ " name: SmsData\n",
+ " file_pattern: data/SMSSpamCollection.csv\n",
+ "\n",
+ " # Split each line into an array, where the first element is message label (ham or spam) and the second is the content of the message.\n",
+ " - type: PyMap\n",
+ " name: SplitLine\n",
+ " fn: 'lambda line: line.split(\"\\\\t\")'\n",
+ "\n",
+ " # Select only the rows that contain spam messages, based on the label.\n",
+ " - type: PyFilter\n",
+ " name: SpamMessages\n",
+ " keep: 'lambda row: row[0] == \"spam\"'\n",
+ "\n",
+ " # Using a regex, we'll split the content of the message (one long string) into words (list of strings)\n",
+ " - type: PyFlatMap\n",
+ " name: FindWords\n",
+ " fn: |\n",
+ " import re\n",
+ " lambda line: re.findall(r\"[a-zA-Z]+\", line[1])\n",
+ "\n",
+ " # Transforming each word to lower case and combining it with a '1'. Result of this step are pairs (word: 1).\n",
+ " - type: PyMap\n",
+ " name: PairLoweredWordsWith1\n",
+ " fn: 'lambda word: (word.lower(), 1)'\n",
+ "\n",
+ " # Using SumPerKey transform, we'll calculate the occurence of each word.\n",
+ " - type: SumPerKey\n",
+ " name: GroupAndSum\n",
+ "\n",
+ " # Select 10 most popular words. Input format to this step is a tuple (word: count),\n",
+ " # so we provide the count (row[1]) as the key to compare the numbers\n",
+ " - type: TopNLargest\n",
+ " name: Largest\n",
+ " n: 10\n",
+ " key: 'lambda row: row[1]'\n",
+ "\n",
+ " # Save results to a text file.\n",
+ " - type: WriteToText\n",
+ " name: SaveToText\n",
+ " file_path_prefix: \"data/result-pipeline-02\"\n",
+ " file_name_suffix: \".txt\"\n",
+ "'''\n",
+ "save_to_file(pipeline, 'pipeline-02.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's run the pipeline:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('python -m apache_beam.yaml.main --pipeline_spec_file=pipeline-02.yaml')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To view the output:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "run('head data/result-pipeline-02-00000-of-00001.txt')"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Summary\n",
Review Comment:
Sure, that's a good idea. It should be easy easy to do
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "outputs": [],
+ "metadata": {
+ "cellView": "form"
+ }
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Yaml\n",
+ "\n",
+ "While Beam provides powerful APIs for authoring sophisticated data processing pipelines, it often still has too high a barrier for getting started and authoring simple pipelines. Even setting up the environment, installing the dependencies, and setting up the project can be an overwhelming amount of boilerplate for some (though https://beam.apache.org/blog/beam-starter-projects/ has gone a long way in making this easier).\n",
+ "\n",
+ "Here we provide a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually *execute* a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).\n",
+ "\n",
+ "In this notebook, we set up your development environment and write a simple pipeline using Yaml API. We'll run it locally, using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
Review Comment:
Do we have any already? Or should I leave a todo note here?
##########
examples/notebooks/get-started/try-apache-beam-yaml.ipynb:
##########
@@ -0,0 +1,424 @@
+{
Review Comment:
Good idea. I'll reach out to rszper@.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org