You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by me...@apache.org on 2019/04/22 23:00:57 UTC
[beam] branch master updated: Add interactive "Try Apache Beam" page
This is an automated email from the ASF dual-hosted git repository.
melap pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new fc6a7de Add interactive "Try Apache Beam" page
new bdb1a71 Merge pull request #7679: [BEAM-6557] Adds an interactive "Try Apache Beam" page
fc6a7de is described below
commit fc6a7de31111339d9e61040a16f64a89872c2dc0
Author: David Cavazos <da...@gmail.com>
AuthorDate: Mon Feb 25 13:00:33 2019 -0800
Add interactive "Try Apache Beam" page
---
examples/notebooks/README.md | 66 ++
.../notebooks/get-started/try-apache-beam-go.ipynb | 585 +++++++++++
.../get-started/try-apache-beam-java.ipynb | 1099 ++++++++++++++++++++
.../notebooks/get-started/try-apache-beam-py.ipynb | 410 ++++++++
examples/notebooks/patch.py | 96 ++
website/_config.yml | 6 +
website/build.gradle | 91 +-
.../src/_includes/section-menu/get-started.html | 1 +
website/src/get-started/beam-overview.md | 8 +-
website/src/get-started/try-apache-beam.md | 192 ++++
website/src/index.md | 3 +-
11 files changed, 2542 insertions(+), 15 deletions(-)
diff --git a/examples/notebooks/README.md b/examples/notebooks/README.md
new file mode 100644
index 0000000..34505d2
--- /dev/null
+++ b/examples/notebooks/README.md
@@ -0,0 +1,66 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Interactive Python Notebooks
+
+## Adding a notebooks
+1. Import an .ipynb file or create a new notebook on [Google Colab](https://colab.research.google.com).
+1. In the upper-left corner menu, go to *File* -> *Save a copy in Github...*
+ > You'll be asked to authenticate on GitHub.
+1. Select the *Repository* as `your-username/beam` and the *Branch* you're working on, not master!
+1. Set the file path to somewhere inside `examples/notebooks/`, e.g. `examples/notebooks/get-started/try-apache-beam-py.ipynb`.
+ > You can leave the commit message as the default value, we'll squash all the commits into a single one at the end anyways.
+1. Make sure the *Include a link to Colaboratory* is checked.
+1. Pull the latest changes from the remote branch.
+ ```sh
+ git pull
+ ```
+ > If you have made local changes to the files, you can checkout them to discard the changes and pull the latest ones.
+ ```sh
+ git checkout -- file1 file2 .. fileN
+ git pull
+ ```
+1. Repeat for all the notebooks you want to add.
+1. From the project root directory, patch the Notebooks to point to the `master` branch instead of the local branch.
+ ```sh
+ python examples/notebooks/patch.py
+ ```
+1. Squash all the commits into a single commit.
+ ```sh
+ git commit --all --amend
+ ```
+ After this check the commit hash of the first (bottom) commit you want to include.
+ ```sh
+ git log
+ ```
+ Knowing the commit hash, we'll now rebase to that commit.
+ ```sh
+ git rebase -i <commit-hash>
+ ```
+ Your editor will open. Leave the first commit as `pick` and the rest replace them as `s` or `squash`. Don't worry about the commit messages for now.
+ ```
+ pick 55ed426e22 commit message 1
+ s 0cbabbf704 commit message 2
+ s d1513977fc commit message 3
+ s 17c6db7950 commit message 4
+ s 81634761e9 commit message 5
+ ```
+ > NOTE: in vim you can do this with `:2,$s/^pick/s/g`
+
+ Finally, your editor will open again. All the commit messages will be visible, delete and reword as necessary to leave only one uncommented commit message. After closing your editor all your commits should be squashed :)
\ No newline at end of file
diff --git a/examples/notebooks/get-started/try-apache-beam-go.ipynb b/examples/notebooks/get-started/try-apache-beam-go.ipynb
new file mode 100644
index 0000000..a092291
--- /dev/null
+++ b/examples/notebooks/get-started/try-apache-beam-go.ipynb
@@ -0,0 +1,585 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "cells": [
+ {
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-go.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ],
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ }
+ },
+ {
+ "source": [
+ "# Try Apache Beam - Go\n",
+ "\n",
+ "In this notebook, we set up your development environment and work through a simple example using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ],
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "lNKIMlEDZ_Vw"
+ }
+ },
+ {
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment."
+ ],
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fz6KSQ13_3Rr"
+ }
+ },
+ {
+ "source": [
+ "import os\n",
+ "\n",
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "# Change directory to $HOME.\n",
+ "print(f\"Changing directory to $HOME: {os.environ['HOME']}\\n\")\n",
+ "os.chdir(os.environ['HOME'])\n",
+ "\n",
+ "# Copy the input file into the local filesystem.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/')"
+ ],
+ "cell_type": "code",
+ "execution_count": 1,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Changing directory to $HOME: /root\n",
+ "\n",
+ ">> mkdir -p data\n",
+ "\n",
+ ">> gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/\n",
+ "Copying gs://dataflow-samples/shakespeare/kinglear.txt...\n",
+ "/ [1 files][153.6 KiB/153.6 KiB] \n",
+ "Operation completed over 1 objects/153.6 KiB. \n",
+ "\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "outputId": "54640caa-f322-4b1e-c124-bfb274a70b56",
+ "colab_type": "code",
+ "id": "GOOk81Jj_yUy",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ }
+ },
+ {
+ "source": [
+ "## Installing development tools\n",
+ "\n",
+ "Let's start by installing Go. This will take a while, so feel free to go for a walk or do some stretching."
+ ],
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "bgSegMTHlSqb"
+ }
+ },
+ {
+ "source": [
+ "# Update and upgrade the system before installing anything else.\n",
+ "run('apt-get update > /dev/null')\n",
+ "run('apt-get upgrade > /dev/null')\n",
+ "\n",
+ "# Install the Go package.\n",
+ "run('apt-get install golang-go > /dev/null')\n",
+ "\n",
+ "# Check the Go version to see if everything is working well.\n",
+ "run('go version')\n",
+ "\n",
+ "# Finally, let's install the Apache Beam SDK for Go.\n",
+ "run('go get -u github.com/apache/beam/sdks/go/...')"
+ ],
+ "cell_type": "code",
+ "execution_count": 2,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ ">> apt-get update > /dev/null\n",
+ "\n",
+ ">> apt-get upgrade > /dev/null\n",
+ "\n",
+ ">> apt-get install golang-go > /dev/null\n",
+ "\n",
+ ">> go version\n",
+ "go version go1.10.4 linux/amd64\n",
+ "\n",
+ ">> go get -u github.com/apache/beam/sdks/go/...\n",
+ "\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "outputId": "60d90f8a-a949-4584-c824-f671cd241d5d",
+ "colab_type": "code",
+ "id": "ibHfVnpolP3b",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 204
+ }
+ }
+ },
+ {
+ "source": [
+ "## Creating the directory structure\n",
+ "\n",
+ "Go requires all packages to be contained within the `GOPATH`. By default it is located in `$HOME/go`, you can check yours using the `go env GOPATH` command.\n",
+ "\n",
+ "Inside the `GOPATH` there should be a `src` directory that holds up all the packages, and a `bin` directory will be created containing all the compiled binaries.\n",
+ "\n",
+ "To learn more about Go's directory structure, see [How to Write Go Code](https://golang.org/doc/code.html)."
+ ],
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "ungfz1aFqYnn"
+ }
+ },
+ {
+ "source": [
+ "# Get the GOPATH.\n",
+ "cmd_stdout = !go env GOPATH\n",
+ "GOPATH = cmd_stdout[0]\n",
+ "print(f\"GOPATH={GOPATH}\\n\")\n",
+ "\n",
+ "# Create our source code wordcount package.\n",
+ "run(f\"mkdir -p {GOPATH}/src/wordcount\")"
+ ],
+ "cell_type": "code",
+ "execution_count": 3,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "GOPATH=/root/go\n",
+ "\n",
+ ">> mkdir -p /root/go/src/wordcount\n",
+ "\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "outputId": "8171ef7c-9498-4cb7-9b55-6a61fca3ed14",
+ "colab_type": "code",
+ "id": "yU3F1Snrt6Nv",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 85
+ }
+ }
+ },
+ {
+ "source": [
+ "# Minimal word count\n",
+ "\n",
+ "The following example is the \"Hello, World!\" of data processing, a basic implementation of word count. We're creating a simple data processing pipeline that reads a text file and counts the number of occurrences of every word.\n",
+ "\n",
+ "There are many scenarios where all the data does not fit in memory. Notice that the outputs of the pipeline go to the file system, which allows for large processing jobs in distributed environments."
+ ],
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "cPvvFB19uXNw"
+ }
+ },
+ {
+ "source": [
+ "## wordcount.go"
+ ],
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "3MUaWD4Dm5NB"
+ }
+ },
+ {
+ "source": [
+ "%%writefile go/src/wordcount/wordcount.go\n",
+ "\n",
+ "package main\n",
+ "\n",
+ "import (\n",
+ "\t\"context\"\n",
+ " \t\"flag\"\n",
+ "\t\"fmt\"\n",
+ "\t\"regexp\"\n",
+ "\n",
+ "\t\"github.com/apache/beam/sdks/go/pkg/beam\"\n",
+ "\t\"github.com/apache/beam/sdks/go/pkg/beam/io/textio\"\n",
+ "\t\"github.com/apache/beam/sdks/go/pkg/beam/runners/direct\"\n",
+ "\t\"github.com/apache/beam/sdks/go/pkg/beam/transforms/stats\"\n",
+ "\n",
+ "\t_ \"github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local\"\n",
+ ")\n",
+ "\n",
+ "var (\n",
+ "\tinput = flag.String(\"input\", \"data/*\", \"File(s) to read.\")\n",
+ "\toutput = flag.String(\"output\", \"outputs/wordcounts.txt\", \"Output filename.\")\n",
+ ")\n",
+ "\n",
+ "var wordRE = regexp.MustCompile(`[a-zA-Z]+('[a-z])?`)\n",
+ "\n",
+ "func main() {\n",
+ " flag.Parse()\n",
+ "\n",
+ "\tbeam.Init()\n",
+ "\n",
+ "\tpipeline := beam.NewPipeline()\n",
+ "\troot := pipeline.Root()\n",
+ "\n",
+ "\tlines := textio.Read(root, *input)\n",
+ "\twords := beam.ParDo(root, func(line string, emit func(string)) {\n",
+ "\t\tfor _, word := range wordRE.FindAllString(line, -1) {\n",
+ "\t\t\temit(word)\n",
+ "\t\t}\n",
+ "\t}, lines)\n",
+ "\tcounted := stats.Count(root, words)\n",
+ "\tformatted := beam.ParDo(root, func(word string, count int) string {\n",
+ "\t\treturn fmt.Sprintf(\"%s: %v\", word, count)\n",
+ "\t}, counted)\n",
+ "\ttextio.Write(root, *output, formatted)\n",
+ "\n",
+ "\tdirect.Execute(context.Background(), pipeline)\n",
+ "}"
+ ],
+ "cell_type": "code",
+ "execution_count": 4,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Writing go/src/wordcount/wordcount.go\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "outputId": "cdb8f492-e653-4daa-dd1d-e2c3c257dd0e",
+ "colab_type": "code",
+ "id": "oUqfqWyMuIfR",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ }
+ },
+ {
+ "source": [
+ "## Building and running\n",
+ "\n",
+ "Go allows us to run a program without having to explicitly compile it. Internally it will compile the source code into a binary and then run it."
+ ],
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "to3rfuOhq0i3"
+ }
+ },
+ {
+ "source": [
+ "# Build and run the program.\n",
+ "run('rm -rf outputs/')\n",
+ "run(f\"go run {GOPATH}/src/wordcount/*.go\")\n",
+ "\n",
+ "# Sample the first 20 results, remember there are no ordering guarantees.\n",
+ "run('head -n 20 outputs/*')"
+ ],
+ "cell_type": "code",
+ "execution_count": 5,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ ">> rm -rf outputs/\n",
+ "\n",
+ ">> go run /root/go/src/wordcount/*.go\n",
+ "2019/03/04 23:05:37 Executing pipeline with the direct runner.\n",
+ "2019/03/04 23:05:37 Pipeline:\n",
+ "2019/03/04 23:05:37 Nodes: {1: []uint8/bytes GLO}\n",
+ "{2: string/string[string] GLO}\n",
+ "{3: string/string[string] GLO}\n",
+ "{4: string/string[string] GLO}\n",
+ "{5: string/string[string] GLO}\n",
+ "{6: KV<string,int>/KV<string[string],int[varintz]> GLO}\n",
+ "{7: CoGBK<string,int>/CoGBK<string[string],int[varintz]> GLO}\n",
+ "{8: KV<string,int>/KV<string[string],int[varintz]> GLO}\n",
+ "{9: string/string[string] GLO}\n",
+ "{10: KV<int,string>/KV<int[varintz],string[string]> GLO}\n",
+ "{11: CoGBK<int,string>/CoGBK<int[varintz],string[string]> GLO}\n",
+ "Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]\n",
+ "2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: string/string[string] GLO}]\n",
+ "3: ParDo [In(Main): string <- {2: string/string[string] GLO}] -> [Out: string -> {3: string/string[string] GLO}]\n",
+ "4: ParDo [In(Main): string <- {3: string/string[string] GLO}] -> [Out: string -> {4: string/string[string] GLO}]\n",
+ "5: ParDo [In(Main): string <- {4: string/string[string] GLO}] -> [Out: string -> {5: string/string[string] GLO}]\n",
+ "6: ParDo [In(Main): T <- {5: string/string[string] GLO}] -> [Out: KV<T,int> -> {6: KV<string,int>/KV<string[string],int[varintz]> GLO}]\n",
+ "7: CoGBK [In(Main): KV<string,int> <- {6: KV<string,int>/KV<string[string],int[varintz]> GLO}] -> [Out: CoGBK<string,int> -> {7: CoGBK<string,int>/CoGBK<string[string],int[varintz]> GLO}]\n",
+ "8: Combine [In(Main): int <- {7: CoGBK<string,int>/CoGBK<string[string],int[varintz]> GLO}] -> [Out: KV<string,int> -> {8: KV<string,int>/KV<string[string],int[varintz]> GLO}]\n",
+ "9: ParDo [In(Main): KV<string,int> <- {8: KV<string,int>/KV<string[string],int[varintz]> GLO}] -> [Out: string -> {9: string/string[string] GLO}]\n",
+ "10: ParDo [In(Main): T <- {9: string/string[string] GLO}] -> [Out: KV<int,T> -> {10: KV<int,string>/KV<int[varintz],string[string]> GLO}]\n",
+ "11: CoGBK [In(Main): KV<int,string> <- {10: KV<int,string>/KV<int[varintz],string[string]> GLO}] -> [Out: CoGBK<int,string> -> {11: CoGBK<int,string>/CoGBK<int[varintz],string[string]> GLO}]\n",
+ "12: ParDo [In(Main): CoGBK<int,string> <- {11: CoGBK<int,string>/CoGBK<int[varintz],string[string]> GLO}] -> []\n",
+ "2019/03/04 23:05:37 Plan[plan]:\n",
+ "14: Impulse[0]\n",
+ "1: ParDo[textio.writeFileFn] Out:[]\n",
+ "2: CoGBK. Out:1\n",
+ "3: Inject[0]. Out:2\n",
+ "4: ParDo[beam.addFixedKeyFn] Out:[3]\n",
+ "5: ParDo[main.main.func2] Out:[4]\n",
+ "6: Combine[stats.sumIntFn] Keyed:false Out:5\n",
+ "7: CoGBK. Out:6\n",
+ "8: Inject[0]. Out:7\n",
+ "9: ParDo[stats.mapFn] Out:[8]\n",
+ "10: ParDo[main.main.func1] Out:[9]\n",
+ "11: ParDo[textio.readFn] Out:[10]\n",
+ "12: ParDo[textio.expandFn] Out:[11]\n",
+ "13: ParDo[beam.createFn] Out:[12]\n",
+ "2019/03/04 23:05:37 Reading from data/kinglear.txt\n",
+ "2019/03/04 23:05:37 Writing to outputs/wordcounts.txt\n",
+ "\n",
+ ">> head -n 20 outputs/*\n",
+ "breeding: 3\n",
+ "alas: 1\n",
+ "condition: 2\n",
+ "whole: 1\n",
+ "rarity: 1\n",
+ "hoping: 1\n",
+ "oath: 4\n",
+ "pretence: 2\n",
+ "beastly: 1\n",
+ "chide: 1\n",
+ "mile: 1\n",
+ "Villain: 1\n",
+ "preach: 1\n",
+ "rescue: 1\n",
+ "Alarum: 2\n",
+ "loath: 1\n",
+ "clotpoll: 1\n",
+ "shortly: 2\n",
+ "alack: 3\n",
+ "What: 75\n",
+ "\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "outputId": "bb7e1169-e080-4d80-deb7-11af178a5230",
+ "colab_type": "code",
+ "id": "0FbGD-rpocgx",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1193
+ }
+ }
+ },
+ {
+ "source": [
+ "# Word count with comments\n",
+ "\n",
+ "Below is mostly the same code as above, but with comments explaining every line in more detail."
+ ],
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "k-HubCrk-h_G"
+ }
+ },
+ {
+ "source": [
+ "%%writefile go/src/wordcount/wordcount.go\n",
+ "\n",
+ "package main\n",
+ "\n",
+ "import (\n",
+ "\t\"context\"\n",
+ " \"flag\"\n",
+ "\t\"fmt\"\n",
+ "\t\"regexp\"\n",
+ "\n",
+ "\t\"github.com/apache/beam/sdks/go/pkg/beam\"\n",
+ "\t\"github.com/apache/beam/sdks/go/pkg/beam/io/textio\"\n",
+ "\t\"github.com/apache/beam/sdks/go/pkg/beam/runners/direct\"\n",
+ "\t\"github.com/apache/beam/sdks/go/pkg/beam/transforms/stats\"\n",
+ "\n",
+ "\t_ \"github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local\"\n",
+ ")\n",
+ "\n",
+ "var (\n",
+ "\tinput = flag.String(\"input\", \"data/*\", \"File(s) to read.\")\n",
+ "\toutput = flag.String(\"output\", \"outputs/wordcounts.txt\", \"Output filename.\")\n",
+ ")\n",
+ "\n",
+ "var wordRE = regexp.MustCompile(`[a-zA-Z]+('[a-z])?`)\n",
+ "\n",
+ "func main() {\n",
+ " flag.Parse()\n",
+ "\n",
+ "\tbeam.Init()\n",
+ "\n",
+ "\tpipeline := beam.NewPipeline()\n",
+ "\troot := pipeline.Root()\n",
+ "\n",
+ " // Read lines from a text file.\n",
+ "\tlines := textio.Read(root, *input)\n",
+ "\n",
+ " // Use a regular expression to iterate over all words in the line.\n",
+ "\twords := beam.ParDo(root, func(line string, emit func(string)) {\n",
+ "\t\tfor _, word := range wordRE.FindAllString(line, -1) {\n",
+ "\t\t\temit(word)\n",
+ "\t\t}\n",
+ "\t}, lines)\n",
+ "\n",
+ " // Count each unique word.\n",
+ "\tcounted := stats.Count(root, words)\n",
+ "\n",
+ " // Format the results into a string so we can write them to a file.\n",
+ "\tformatted := beam.ParDo(root, func(word string, count int) string {\n",
+ "\t\treturn fmt.Sprintf(\"%s: %v\", word, count)\n",
+ "\t}, counted)\n",
+ "\n",
+ " // Finally, write the results to a file.\n",
+ "\ttextio.Write(root, *output, formatted)\n",
+ "\n",
+ " // We have to explicitly run the pipeline, otherwise it's only a definition.\n",
+ "\tdirect.Execute(context.Background(), pipeline)\n",
+ "}"
+ ],
+ "cell_type": "code",
+ "execution_count": 6,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Overwriting go/src/wordcount/wordcount.go\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "outputId": "6242a213-effd-4cf9-aaba-caf3ebf72ef2",
+ "colab_type": "code",
+ "id": "x_D7sxUHFzUp",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ }
+ },
+ {
+ "source": [
+ "# Build and run the program.\n",
+ "run('rm -rf outputs/')\n",
+ "run('go run go/src/wordcount/*.go 2>/dev/null')\n",
+ "\n",
+ "# Sample the first 20 results, remember there are no ordering guarantees.\n",
+ "run('head -n 20 outputs/*')"
+ ],
+ "cell_type": "code",
+ "execution_count": 7,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ ">> rm -rf outputs/\n",
+ "\n",
+ ">> go run go/src/wordcount/*.go 2>/dev/null\n",
+ "\n",
+ ">> head -n 20 outputs/*\n",
+ "hawthorn: 2\n",
+ "With: 31\n",
+ "vain: 3\n",
+ "football: 1\n",
+ "showest: 1\n",
+ "rarest: 1\n",
+ "Acquaint: 1\n",
+ "Bids: 1\n",
+ "another: 9\n",
+ "tadpole: 1\n",
+ "Oppressed: 1\n",
+ "Revoke: 1\n",
+ "images: 1\n",
+ "lameness: 1\n",
+ "Instantly: 1\n",
+ "rages: 1\n",
+ "Neither: 1\n",
+ "quest: 1\n",
+ "mills: 1\n",
+ "weapons: 1\n",
+ "\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "outputId": "78f66139-06df-4991-f19c-1d13803980bc",
+ "colab_type": "code",
+ "id": "H620PSl46wDK",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 459
+ }
+ }
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "colab": {
+ "include_colab_link": true,
+ "name": "Try Apache Beam - Go",
+ "toc_visible": true,
+ "provenance": [],
+ "collapsed_sections": [],
+ "version": "0.3.2"
+ }
+ },
+ "nbformat_minor": 0
+}
\ No newline at end of file
diff --git a/examples/notebooks/get-started/try-apache-beam-java.ipynb b/examples/notebooks/get-started/try-apache-beam-java.ipynb
new file mode 100644
index 0000000..c19762c
--- /dev/null
+++ b/examples/notebooks/get-started/try-apache-beam-java.ipynb
@@ -0,0 +1,1099 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Java",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-java.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Java\n",
+ "\n",
+ "In this notebook, we set up a Java development environment and work through a simple example using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "68240031-2990-41fa-a327-38e15dc9fdf9",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 136
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd} # This is magic to run 'cmd' in the shell.\n",
+ " print('')\n",
+ "\n",
+ "# Copy the input file into the local filesystem.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/')"
+ ],
+ "execution_count": 1,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ ">> mkdir -p data\n",
+ "\n",
+ ">> gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/\n",
+ "Copying gs://dataflow-samples/shakespeare/kinglear.txt...\n",
+ "/ [1 files][153.6 KiB/153.6 KiB] \n",
+ "Operation completed over 1 objects/153.6 KiB. \n",
+ "\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Hmto8JTSWwUK",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## Installing development tools\n",
+ "\n",
+ "Let's start by installing Java. We'll use the `default-jdk`, which uses [OpenJDK](https://openjdk.java.net/). This will take a while, so feel free to go for a walk or do some stretching.\n",
+ "\n",
+ "**Note:** Alternatively, you could install the propietary [Oracle JDK](https://www.oracle.com/technetwork/java/javase/downloads/index.html) instead."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "ONYtX0doWpFz",
+ "colab_type": "code",
+ "outputId": "04bfa861-0bf8-4352-e878-0f24c6c7b61e",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 187
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Update and upgrade the system before installing anything else.\n",
+ "run('apt-get update > /dev/null')\n",
+ "run('apt-get upgrade > /dev/null')\n",
+ "\n",
+ "# Install the Java JDK.\n",
+ "run('apt-get install default-jdk > /dev/null')\n",
+ "\n",
+ "# Check the Java version to see if everything is working well.\n",
+ "run('javac -version')"
+ ],
+ "execution_count": 2,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ ">> apt-get update > /dev/null\n",
+ "\n",
+ ">> apt-get upgrade > /dev/null\n",
+ "Extracting templates from packages: 100%\n",
+ "\n",
+ ">> apt-get install default-jdk > /dev/null\n",
+ "\n",
+ ">> javac -version\n",
+ "javac 10.0.2\n",
+ "\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Wab7H4IZW9xZ",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Now, let's install [Gradle](https://gradle.org/), which we'll need to automate the build and running processes for our application. \n",
+ "\n",
+ "**Note:** Alternatively, you could install and configure [Maven](https://maven.apache.org/) instead."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "xS3Oeu3DW7vy",
+ "colab_type": "code",
+ "outputId": "1b2c1f11-5e35-4d22-8002-814ea61224c9",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 595
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "\n",
+ "# Download the gradle source.\n",
+ "gradle_version = 'gradle-5.0'\n",
+ "gradle_path = f\"/opt/{gradle_version}\"\n",
+ "if not os.path.exists(gradle_path):\n",
+ " run(f\"wget -q -nc -O gradle.zip https://services.gradle.org/distributions/{gradle_version}-bin.zip\")\n",
+ " run('unzip -q -d /opt gradle.zip')\n",
+ " run('rm -f gradle.zip')\n",
+ "\n",
+ "# We're choosing to use the absolute path instead of adding it to the $PATH environment variable.\n",
+ "def gradle(args):\n",
+ " run(f\"{gradle_path}/bin/gradle --console=plain {args}\")\n",
+ "\n",
+ "gradle('-v')"
+ ],
+ "execution_count": 3,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ ">> wget -q -nc -O gradle.zip https://services.gradle.org/distributions/gradle-5.0-bin.zip\n",
+ "\n",
+ ">> unzip -q -d /opt gradle.zip\n",
+ "\n",
+ ">> rm -f gradle.zip\n",
+ "\n",
+ ">> /opt/gradle-5.0/bin/gradle --console=plain -v\n",
+ "\u001b[m\n",
+ "Welcome to Gradle 5.0!\n",
+ "\n",
+ "Here are the highlights of this release:\n",
+ " - Kotlin DSL 1.0\n",
+ " - Task timeouts\n",
+ " - Dependency alignment aka BOM support\n",
+ " - Interactive `gradle init`\n",
+ "\n",
+ "For more details see https://docs.gradle.org/5.0/release-notes.html\n",
+ "\n",
+ "\n",
+ "------------------------------------------------------------\n",
+ "Gradle 5.0\n",
+ "------------------------------------------------------------\n",
+ "\n",
+ "Build time: 2018-11-26 11:48:43 UTC\n",
+ "Revision: 7fc6e5abf2fc5fe0824aec8a0f5462664dbcd987\n",
+ "\n",
+ "Kotlin DSL: 1.0.4\n",
+ "Kotlin: 1.3.10\n",
+ "Groovy: 2.5.4\n",
+ "Ant: Apache Ant(TM) version 1.9.13 compiled on July 10 2018\n",
+ "JVM: 10.0.2 (Oracle Corporation 10.0.2+13-Ubuntu-1ubuntu0.18.04.4)\n",
+ "OS: Linux 4.14.79+ amd64\n",
+ "\n",
+ "\u001b[m\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "YTkkapX9KVhA",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## build.gradle\n",
+ "\n",
+ "We'll also need a [`build.gradle`](https://guides.gradle.org/creating-new-gradle-builds/) file which will allow us to invoke some useful commands."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "oUqfqWyMuIfR",
+ "colab_type": "code",
+ "outputId": "292a06b2-ce06-46b6-8598-480d83974bbb",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "%%writefile build.gradle\n",
+ "\n",
+ "plugins {\n",
+ " // id 'idea' // Uncomment for IntelliJ IDE\n",
+ " // id 'eclipse' // Uncomment for Eclipse IDE\n",
+ "\n",
+ " // Apply java plugin and make it a runnable application.\n",
+ " id 'java'\n",
+ " id 'application'\n",
+ "\n",
+ " // 'shadow' allows us to embed all the dependencies into a fat jar.\n",
+ " id 'com.github.johnrengelman.shadow' version '4.0.3'\n",
+ "}\n",
+ "\n",
+ "// This is the path of the main class, stored within ./src/main/java/\n",
+ "mainClassName = 'samples.quickstart.WordCount'\n",
+ "\n",
+ "// Declare the sources from which to fetch dependencies.\n",
+ "repositories {\n",
+ " mavenCentral()\n",
+ "}\n",
+ "\n",
+ "// Java version compatibility.\n",
+ "sourceCompatibility = 1.8\n",
+ "targetCompatibility = 1.8\n",
+ "\n",
+ "// Use the latest Apache Beam major version 2.\n",
+ "// You can also lock into a minor version like '2.9.+'.\n",
+ "ext.apacheBeamVersion = '2.+'\n",
+ "\n",
+ "// Declare the dependencies of the project.\n",
+ "dependencies {\n",
+ " shadow \"org.apache.beam:beam-sdks-java-core:$apacheBeamVersion\"\n",
+ "\n",
+ " runtime \"org.apache.beam:beam-runners-direct-java:$apacheBeamVersion\"\n",
+ " runtime \"org.slf4j:slf4j-api:1.+\"\n",
+ " runtime \"org.slf4j:slf4j-jdk14:1.+\"\n",
+ "\n",
+ " testCompile \"junit:junit:4.+\"\n",
+ "}\n",
+ "\n",
+ "// Configure 'shadowJar' instead of 'jar' to set up the fat jar.\n",
+ "shadowJar {\n",
+ " baseName = 'WordCount' // Name of the fat jar file.\n",
+ " classifier = null // Set to null, otherwise 'shadow' appends a '-all' to the jar file name.\n",
+ " manifest {\n",
+ " attributes('Main-Class': mainClassName) // Specify where the main class resides.\n",
+ " }\n",
+ "}"
+ ],
+ "execution_count": 4,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Writing build.gradle\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "cwZcqmFgoLJ9",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## Creating the directory structure\n",
+ "\n",
+ "Java and Gradle expect a specific [directory structure](https://docs.gradle.org/current/userguide/organizing_gradle_projects.html). This helps organize large projects into a standard structure.\n",
+ "\n",
+ "For now, we only need a place where our quickstart code will reside. That has to go within `./src/main/java/`."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Mr1KTQznbd9F",
+ "colab_type": "code",
+ "outputId": "2e4635b9-0577-4399-b8d6-078183ff9da2",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 51
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "run('mkdir -p src/main/java/samples/quickstart')"
+ ],
+ "execution_count": 5,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ ">> mkdir -p src/main/java/samples/quickstart\n",
+ "\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "cPvvFB19uXNw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Minimal word count\n",
+ "\n",
+ "The following example is the \"Hello, World!\" of data processing, a basic implementation of word count. We're creating a simple data processing pipeline that reads a text file and counts the number of occurrences of every word.\n",
+ "\n",
+ "There are many scenarios where all the data does not fit in memory. Notice that the outputs of the pipeline go to the file system, which allows for large processing jobs in distributed environments."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fl3iUat7KYIE",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## WordCount.java"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "5l3S2mjMBKhT",
+ "colab_type": "code",
+ "outputId": "6e55ec70-e727-44c9-a425-4afed97188fe",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "%%writefile src/main/java/samples/quickstart/WordCount.java\n",
+ "\n",
+ "package samples.quickstart;\n",
+ "\n",
+ "import org.apache.beam.sdk.Pipeline;\n",
+ "import org.apache.beam.sdk.io.TextIO;\n",
+ "import org.apache.beam.sdk.options.PipelineOptions;\n",
+ "import org.apache.beam.sdk.options.PipelineOptionsFactory;\n",
+ "import org.apache.beam.sdk.transforms.Count;\n",
+ "import org.apache.beam.sdk.transforms.Filter;\n",
+ "import org.apache.beam.sdk.transforms.FlatMapElements;\n",
+ "import org.apache.beam.sdk.transforms.MapElements;\n",
+ "import org.apache.beam.sdk.values.KV;\n",
+ "import org.apache.beam.sdk.values.TypeDescriptors;\n",
+ "\n",
+ "import java.util.Arrays;\n",
+ "\n",
+ "public class WordCount {\n",
+ " public static void main(String[] args) {\n",
+ " String inputsDir = \"data/*\";\n",
+ " String outputsPrefix = \"outputs/part\";\n",
+ "\n",
+ " PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();\n",
+ " Pipeline pipeline = Pipeline.create(options);\n",
+ " pipeline\n",
+ " .apply(\"Read lines\", TextIO.read().from(inputsDir))\n",
+ " .apply(\"Find words\", FlatMapElements.into(TypeDescriptors.strings())\n",
+ " .via((String line) -> Arrays.asList(line.split(\"[^\\\\p{L}]+\"))))\n",
+ " .apply(\"Filter empty words\", Filter.by((String word) -> !word.isEmpty()))\n",
+ " .apply(\"Count words\", Count.perElement())\n",
+ " .apply(\"Write results\", MapElements.into(TypeDescriptors.strings())\n",
+ " .via((KV<String, Long> wordCount) ->\n",
+ " wordCount.getKey() + \": \" + wordCount.getValue()))\n",
+ " .apply(TextIO.write().to(outputsPrefix));\n",
+ " pipeline.run();\n",
+ " }\n",
+ "}"
+ ],
+ "execution_count": 6,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Writing src/main/java/samples/quickstart/WordCount.java\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "yoO4xHnaKiz9",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## Build and run"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "giJMbbcq2OPu",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Let's first check how the final file system structure looks like. These are all the files required to build and run our application.\n",
+ "\n",
+ "* `build.gradle` - build configuration for Gradle\n",
+ "* `src/main/java/samples/quickstart/WordCount.java` - application source code\n",
+ "* `data/kinglear.txt` - input data, this could be any file or files\n",
+ "\n",
+ "We are now ready to build the application using `gradle build`."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "urmCmtG08F-0",
+ "colab_type": "code",
+ "outputId": "a2b65437-4244-4844-82d2-1789d5cfd7ca",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 510
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Build the project.\n",
+ "gradle('build')\n",
+ "\n",
+ "# Check the generated build files.\n",
+ "run('ls -lh build/libs/')"
+ ],
+ "execution_count": 11,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ ">> /opt/gradle-5.0/bin/gradle --console=plain build\n",
+ "\u001b[mStarting a Gradle Daemon (subsequent builds will be faster)\n",
+ "> Task :compileJava\n",
+ "> Task :processResources NO-SOURCE\n",
+ "> Task :classes\n",
+ "> Task :jar\n",
+ "> Task :startScripts\n",
+ "> Task :distTar\n",
+ "> Task :distZip\n",
+ "> Task :shadowJar\n",
+ "> Task :startShadowScripts\n",
+ "> Task :shadowDistTar\n",
+ "> Task :shadowDistZip\n",
+ "> Task :assemble\n",
+ "> Task :compileTestJava NO-SOURCE\n",
+ "> Task :processTestResources NO-SOURCE\n",
+ "> Task :testClasses UP-TO-DATE\n",
+ "> Task :test NO-SOURCE\n",
+ "> Task :check UP-TO-DATE\n",
+ "> Task :build\n",
+ "\n",
+ "BUILD SUCCESSFUL in 56s\n",
+ "9 actionable tasks: 9 executed\n",
+ "\u001b[m\n",
+ ">> ls -lh build/libs/\n",
+ "total 40M\n",
+ "-rw-r--r-- 1 root root 2.9K Mar 4 22:59 content.jar\n",
+ "-rw-r--r-- 1 root root 40M Mar 4 23:00 WordCount.jar\n",
+ "\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "LrRFNZHD8dtu",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "There are two files generated:\n",
+ "* The `content.jar` file, the application generated from the regular `build` command. It's only a few kilobytes in size.\n",
+ "* The `WordCount.jar` file, with the `baseName` we specified in the `shadowJar` section of the `gradle.build` file. It's a several megabytes in size, with all the required libraries it needs to run embedded in it.\n",
+ "\n",
+ "The file we're actually interested in is the fat JAR file `WordCount.jar`. To run the fat JAR, we'll use the `gradle runShadow` command."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "CgTXBdTsBn1F",
+ "colab_type": "code",
+ "outputId": "5e447cf9-a01a-4a82-9237-676f0091d4bd",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1822
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run the shadow (fat jar) build.\n",
+ "gradle('runShadow')\n",
+ "\n",
+ "# Sample the first 20 results, remember there are no ordering guarantees.\n",
+ "run('head -n 20 outputs/part-00000-of-*')"
+ ],
+ "execution_count": 12,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ ">> /opt/gradle-5.0/bin/gradle --console=plain runShadow\n",
+ "\u001b[m> Task :compileJava UP-TO-DATE\n",
+ "> Task :processResources NO-SOURCE\n",
+ "> Task :classes UP-TO-DATE\n",
+ "> Task :shadowJar UP-TO-DATE\n",
+ "> Task :startShadowScripts UP-TO-DATE\n",
+ "> Task :installShadowDist\n",
+ "\n",
+ "> Task :runShadow\n",
+ "WARNING: An illegal reflective access operation has occurred\n",
+ "WARNING: Illegal reflective access by org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.UnsafeUtil (file:/content/build/install/content-shadow/lib/WordCount.jar) to field java.nio.Buffer.address\n",
+ "WARNING: Please consider reporting this to the maintainers of org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.UnsafeUtil\n",
+ "WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations\n",
+ "WARNING: All illegal access operations will be denied in a future release\n",
+ "Mar 04, 2019 11:00:24 PM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes\n",
+ "INFO: Filepattern data/* matched 1 files with total size 157283\n",
+ "Mar 04, 2019 11:00:24 PM org.apache.beam.sdk.io.FileBasedSource split\n",
+ "INFO: Splitting filepattern data/* into bundles of size 52427 took 1 ms and produced 1 files and 3 bundles\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement\n",
+ "INFO: Opening writer 7d12bbc4-9165-4493-8563-fb710b827daa for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@3ad2e17 pane PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} destination null\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement\n",
+ "INFO: Opening writer 5b1bdb18-9f9a-47cd-80d2-a65baa31aa60 for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@3ad2e17 pane PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} destination null\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement\n",
+ "INFO: Opening writer 6302e6b5-5428-48e6-b571-76a9282d7f45 for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@3ad2e17 pane PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} destination null\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.FileBasedSink$Writer close\n",
+ "INFO: Successfully wrote temporary file /content/outputs/.temp-beam-2019-03-04_23-00-23-1/7d12bbc4-9165-4493-8563-fb710b827daa\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.FileBasedSink$Writer close\n",
+ "INFO: Successfully wrote temporary file /content/outputs/.temp-beam-2019-03-04_23-00-23-1/5b1bdb18-9f9a-47cd-80d2-a65baa31aa60\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement\n",
+ "INFO: Opening writer 493e3ec4-f0f7-4d10-8209-079f7ac4db16 for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@3ad2e17 pane PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} destination null\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.FileBasedSink$Writer close\n",
+ "INFO: Successfully wrote temporary file /content/outputs/.temp-beam-2019-03-04_23-00-23-1/6302e6b5-5428-48e6-b571-76a9282d7f45\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.FileBasedSink$Writer close\n",
+ "INFO: Successfully wrote temporary file /content/outputs/.temp-beam-2019-03-04_23-00-23-1/493e3ec4-f0f7-4d10-8209-079f7ac4db16\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn process\n",
+ "INFO: Finalizing 4 file results\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation createMissingEmptyShards\n",
+ "INFO: Finalizing for destination null num shards 4.\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation moveToOutputFiles\n",
+ "INFO: Will copy temporary file FileResult{tempFilename=/content/outputs/.temp-beam-2019-03-04_23-00-23-1/7d12bbc4-9165-4493-8563-fb710b827daa, shard=3, window=org.apache.beam.sdk.transforms.windowing.GlobalWindow@3ad2e17, paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}} to final location /content/outputs/part-00003-of-00004\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation moveToOutputFiles\n",
+ "INFO: Will copy temporary file FileResult{tempFilename=/content/outputs/.temp-beam-2019-03-04_23-00-23-1/5b1bdb18-9f9a-47cd-80d2-a65baa31aa60, shard=2, window=org.apache.beam.sdk.transforms.windowing.GlobalWindow@3ad2e17, paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}} to final location /content/outputs/part-00002-of-00004\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation moveToOutputFiles\n",
+ "INFO: Will copy temporary file FileResult{tempFilename=/content/outputs/.temp-beam-2019-03-04_23-00-23-1/6302e6b5-5428-48e6-b571-76a9282d7f45, shard=1, window=org.apache.beam.sdk.transforms.windowing.GlobalWindow@3ad2e17, paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}} to final location /content/outputs/part-00001-of-00004\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation moveToOutputFiles\n",
+ "INFO: Will copy temporary file FileResult{tempFilename=/content/outputs/.temp-beam-2019-03-04_23-00-23-1/493e3ec4-f0f7-4d10-8209-079f7ac4db16, shard=0, window=org.apache.beam.sdk.transforms.windowing.GlobalWindow@3ad2e17, paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}} to final location /content/outputs/part-00000-of-00004\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation removeTemporaryFiles\n",
+ "INFO: Will remove known temporary file /content/outputs/.temp-beam-2019-03-04_23-00-23-1/6302e6b5-5428-48e6-b571-76a9282d7f45\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation removeTemporaryFiles\n",
+ "INFO: Will remove known temporary file /content/outputs/.temp-beam-2019-03-04_23-00-23-1/5b1bdb18-9f9a-47cd-80d2-a65baa31aa60\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation removeTemporaryFiles\n",
+ "INFO: Will remove known temporary file /content/outputs/.temp-beam-2019-03-04_23-00-23-1/7d12bbc4-9165-4493-8563-fb710b827daa\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation removeTemporaryFiles\n",
+ "INFO: Will remove known temporary file /content/outputs/.temp-beam-2019-03-04_23-00-23-1/493e3ec4-f0f7-4d10-8209-079f7ac4db16\n",
+ "Mar 04, 2019 11:00:43 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation removeTemporaryFiles\n",
+ "WARNING: Failed to match temporary files under: [/content/outputs/.temp-beam-2019-03-04_23-00-23-1/].\n",
+ "\n",
+ "BUILD SUCCESSFUL in 24s\n",
+ "5 actionable tasks: 2 executed, 3 up-to-date\n",
+ "\u001b[m\n",
+ ">> head -n 20 outputs/part-00000-of-*\n",
+ "==> outputs/part-00000-of-00001 <==\n",
+ "(u'canker', 1)\n",
+ "(u'bounty', 2)\n",
+ "(u'provision', 3)\n",
+ "(u'to', 438)\n",
+ "(u'terms', 2)\n",
+ "(u'unnecessary', 2)\n",
+ "(u'tongue', 5)\n",
+ "(u'knives', 1)\n",
+ "(u'Commend', 1)\n",
+ "(u'Hum', 2)\n",
+ "(u'Set', 2)\n",
+ "(u'smell', 6)\n",
+ "(u'dreadful', 3)\n",
+ "(u'frowning', 1)\n",
+ "(u'World', 1)\n",
+ "(u'tike', 1)\n",
+ "(u'yes', 3)\n",
+ "(u'oldness', 1)\n",
+ "(u'boat', 1)\n",
+ "(u\"in's\", 1)\n",
+ "\n",
+ "==> outputs/part-00000-of-00004 <==\n",
+ "retinue: 1\n",
+ "stink: 1\n",
+ "beaks: 1\n",
+ "Ten: 1\n",
+ "riots: 2\n",
+ "Service: 1\n",
+ "dealing: 1\n",
+ "stop: 2\n",
+ "detain: 1\n",
+ "beware: 1\n",
+ "pilferings: 1\n",
+ "swimming: 1\n",
+ "The: 124\n",
+ "Been: 1\n",
+ "behavior: 1\n",
+ "impetuous: 1\n",
+ "Thy: 20\n",
+ "Tis: 24\n",
+ "Soldiers: 7\n",
+ "Juno: 1\n",
+ "\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "T_oqlIM55MzM",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## Distributing your application\n",
+ "\n",
+ "We can run our fat JAR file as long as we have a Java Runtime Environment installed.\n",
+ "\n",
+ "To distribute, we copy the fat JAR file and run it with `java -jar`."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "b3YSRjYnavpd",
+ "colab_type": "code",
+ "outputId": "ef88153a-f75f-4e80-8434-ac452a77a199",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1907
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# You can now distribute and run your Java application as a standalone jar file.\n",
+ "run('cp build/libs/WordCount.jar .')\n",
+ "run('java -jar WordCount.jar')\n",
+ "\n",
+ "# Sample the first 20 results, remember there are no ordering guarantees.\n",
+ "run('head -n 20 outputs/part-00000-of-*')"
+ ],
+ "execution_count": 13,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ ">> cp build/libs/WordCount.jar .\n",
+ "\n",
+ ">> java -jar WordCount.jar\n",
+ "WARNING: An illegal reflective access operation has occurred\n",
+ "WARNING: Illegal reflective access by org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.UnsafeUtil (file:/content/WordCount.jar) to field java.nio.Buffer.address\n",
+ "WARNING: Please consider reporting this to the maintainers of org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.UnsafeUtil\n",
+ "WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations\n",
+ "WARNING: All illegal access operations will be denied in a future release\n",
+ "Mar 04, 2019 11:00:49 PM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes\n",
+ "INFO: Filepattern data/* matched 1 files with total size 157283\n",
+ "Mar 04, 2019 11:00:49 PM org.apache.beam.sdk.io.FileBasedSource split\n",
+ "INFO: Splitting filepattern data/* into bundles of size 52427 took 1 ms and produced 1 files and 3 bundles\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement\n",
+ "INFO: Opening writer 273fc5ad-09b8-4e87-95c9-5d9ec72ed294 for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@7a362b6b pane PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} destination null\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement\n",
+ "INFO: Opening writer c224db69-5869-4259-bd43-ca0431ec77fe for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@7a362b6b pane PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} destination null\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement\n",
+ "INFO: Opening writer f45d6e07-d37a-4af3-ad8b-bc316cef7d99 for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@7a362b6b pane PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} destination null\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.FileBasedSink$Writer close\n",
+ "INFO: Successfully wrote temporary file /content/outputs/.temp-beam-2019-03-04_23-00-49-1/f45d6e07-d37a-4af3-ad8b-bc316cef7d99\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.FileBasedSink$Writer close\n",
+ "INFO: Successfully wrote temporary file /content/outputs/.temp-beam-2019-03-04_23-00-49-1/c224db69-5869-4259-bd43-ca0431ec77fe\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.FileBasedSink$Writer close\n",
+ "INFO: Successfully wrote temporary file /content/outputs/.temp-beam-2019-03-04_23-00-49-1/273fc5ad-09b8-4e87-95c9-5d9ec72ed294\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn process\n",
+ "INFO: Finalizing 3 file results\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation createMissingEmptyShards\n",
+ "INFO: Finalizing for destination null num shards 3.\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation moveToOutputFiles\n",
+ "INFO: Will copy temporary file FileResult{tempFilename=/content/outputs/.temp-beam-2019-03-04_23-00-49-1/273fc5ad-09b8-4e87-95c9-5d9ec72ed294, shard=2, window=org.apache.beam.sdk.transforms.windowing.GlobalWindow@7a362b6b, paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}} to final location /content/outputs/part-00002-of-00003\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation moveToOutputFiles\n",
+ "INFO: Will copy temporary file FileResult{tempFilename=/content/outputs/.temp-beam-2019-03-04_23-00-49-1/c224db69-5869-4259-bd43-ca0431ec77fe, shard=0, window=org.apache.beam.sdk.transforms.windowing.GlobalWindow@7a362b6b, paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}} to final location /content/outputs/part-00000-of-00003\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation moveToOutputFiles\n",
+ "INFO: Will copy temporary file FileResult{tempFilename=/content/outputs/.temp-beam-2019-03-04_23-00-49-1/f45d6e07-d37a-4af3-ad8b-bc316cef7d99, shard=1, window=org.apache.beam.sdk.transforms.windowing.GlobalWindow@7a362b6b, paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}} to final location /content/outputs/part-00001-of-00003\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation removeTemporaryFiles\n",
+ "INFO: Will remove known temporary file /content/outputs/.temp-beam-2019-03-04_23-00-49-1/273fc5ad-09b8-4e87-95c9-5d9ec72ed294\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation removeTemporaryFiles\n",
+ "INFO: Will remove known temporary file /content/outputs/.temp-beam-2019-03-04_23-00-49-1/f45d6e07-d37a-4af3-ad8b-bc316cef7d99\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation removeTemporaryFiles\n",
+ "INFO: Will remove known temporary file /content/outputs/.temp-beam-2019-03-04_23-00-49-1/c224db69-5869-4259-bd43-ca0431ec77fe\n",
+ "Mar 04, 2019 11:01:10 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation removeTemporaryFiles\n",
+ "WARNING: Failed to match temporary files under: [/content/outputs/.temp-beam-2019-03-04_23-00-49-1/].\n",
+ "\n",
+ ">> head -n 20 outputs/part-00000-of-*\n",
+ "==> outputs/part-00000-of-00001 <==\n",
+ "(u'canker', 1)\n",
+ "(u'bounty', 2)\n",
+ "(u'provision', 3)\n",
+ "(u'to', 438)\n",
+ "(u'terms', 2)\n",
+ "(u'unnecessary', 2)\n",
+ "(u'tongue', 5)\n",
+ "(u'knives', 1)\n",
+ "(u'Commend', 1)\n",
+ "(u'Hum', 2)\n",
+ "(u'Set', 2)\n",
+ "(u'smell', 6)\n",
+ "(u'dreadful', 3)\n",
+ "(u'frowning', 1)\n",
+ "(u'World', 1)\n",
+ "(u'tike', 1)\n",
+ "(u'yes', 3)\n",
+ "(u'oldness', 1)\n",
+ "(u'boat', 1)\n",
+ "(u\"in's\", 1)\n",
+ "\n",
+ "==> outputs/part-00000-of-00003 <==\n",
+ "With: 31\n",
+ "justification: 1\n",
+ "hither: 15\n",
+ "make: 46\n",
+ "opposed: 2\n",
+ "prince: 5\n",
+ "Burn: 1\n",
+ "waking: 1\n",
+ "waked: 3\n",
+ "inform: 6\n",
+ "mercy: 5\n",
+ "about: 11\n",
+ "danger: 6\n",
+ "Croak: 1\n",
+ "happier: 1\n",
+ "stick: 2\n",
+ "oppressed: 1\n",
+ "erlook: 1\n",
+ "untented: 1\n",
+ "myself: 10\n",
+ "\n",
+ "==> outputs/part-00000-of-00004 <==\n",
+ "retinue: 1\n",
+ "stink: 1\n",
+ "beaks: 1\n",
+ "Ten: 1\n",
+ "riots: 2\n",
+ "Service: 1\n",
+ "dealing: 1\n",
+ "stop: 2\n",
+ "detain: 1\n",
+ "beware: 1\n",
+ "pilferings: 1\n",
+ "swimming: 1\n",
+ "The: 124\n",
+ "Been: 1\n",
+ "behavior: 1\n",
+ "impetuous: 1\n",
+ "Thy: 20\n",
+ "Tis: 24\n",
+ "Soldiers: 7\n",
+ "Juno: 1\n",
+ "\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "k-HubCrk-h_G",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Word count with comments\n",
+ "\n",
+ "Below is mostly the same code as above, but with comments explaining every line in more detail."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "wvnWyYklCXer",
+ "colab_type": "code",
+ "outputId": "275507a3-05e9-44ca-8625-d745154d5720",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "%%writefile src/main/java/samples/quickstart/WordCount.java\n",
+ "\n",
+ "package samples.quickstart;\n",
+ "\n",
+ "import org.apache.beam.sdk.Pipeline;\n",
+ "import org.apache.beam.sdk.io.TextIO;\n",
+ "import org.apache.beam.sdk.options.PipelineOptions;\n",
+ "import org.apache.beam.sdk.options.PipelineOptionsFactory;\n",
+ "import org.apache.beam.sdk.transforms.Count;\n",
+ "import org.apache.beam.sdk.transforms.Filter;\n",
+ "import org.apache.beam.sdk.transforms.FlatMapElements;\n",
+ "import org.apache.beam.sdk.transforms.MapElements;\n",
+ "import org.apache.beam.sdk.values.KV;\n",
+ "import org.apache.beam.sdk.values.PCollection;\n",
+ "import org.apache.beam.sdk.values.TypeDescriptors;\n",
+ "\n",
+ "import java.util.Arrays;\n",
+ "\n",
+ "public class WordCount {\n",
+ " public static void main(String[] args) {\n",
+ " String inputsDir = \"data/*\";\n",
+ " String outputsPrefix = \"outputs/part\";\n",
+ "\n",
+ " PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();\n",
+ " Pipeline pipeline = Pipeline.create(options);\n",
+ "\n",
+ " // Store the word counts in a PCollection.\n",
+ " // Each element is a KeyValue of (word, count) of types KV<String, Long>.\n",
+ " PCollection<KV<String, Long>> wordCounts =\n",
+ " // The input PCollection is an empty pipeline.\n",
+ " pipeline\n",
+ "\n",
+ " // Read lines from a text file.\n",
+ " .apply(\"Read lines\", TextIO.read().from(inputsDir))\n",
+ " // Element type: String - text line\n",
+ "\n",
+ " // Use a regular expression to iterate over all words in the line.\n",
+ " // FlatMapElements will yield an element for every element in an iterable.\n",
+ " .apply(\"Find words\", FlatMapElements.into(TypeDescriptors.strings())\n",
+ " .via((String line) -> Arrays.asList(line.split(\"[^\\\\p{L}]+\"))))\n",
+ " // Element type: String - word\n",
+ "\n",
+ " // Keep only non-empty words.\n",
+ " .apply(\"Filter empty words\", Filter.by((String word) -> !word.isEmpty()))\n",
+ " // Element type: String - word\n",
+ "\n",
+ " // Count each unique word.\n",
+ " .apply(\"Count words\", Count.perElement());\n",
+ " // Element type: KV<String, Long> - key: word, value: counts\n",
+ "\n",
+ " // We can process a PCollection through other pipelines, too.\n",
+ " // The input PCollection are the wordCounts from the previous step.\n",
+ " wordCounts\n",
+ " // Format the results into a string so we can write them to a file.\n",
+ " .apply(\"Write results\", MapElements.into(TypeDescriptors.strings())\n",
+ " .via((KV<String, Long> wordCount) ->\n",
+ " wordCount.getKey() + \": \" + wordCount.getValue()))\n",
+ " // Element type: str - text line\n",
+ "\n",
+ " // Finally, write the results to a file.\n",
+ " .apply(TextIO.write().to(outputsPrefix));\n",
+ "\n",
+ " // We have to explicitly run the pipeline, otherwise it's only a definition.\n",
+ " pipeline.run();\n",
+ " }\n",
+ "}"
+ ],
+ "execution_count": 14,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Overwriting src/main/java/samples/quickstart/WordCount.java\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "wKAJp7ON4Vpp",
+ "colab_type": "code",
+ "outputId": "9a4c7a72-70a1-4d31-89c1-cf7fb8fcdf53",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 2060
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Build and run the project. The 'runShadow' task implicitly does a 'build'.\n",
+ "gradle('runShadow')\n",
+ "\n",
+ "# Sample the first 20 results, remember there are no ordering guarantees.\n",
+ "run('head -n 20 outputs/part-00000-of-*')"
+ ],
+ "execution_count": 15,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ ">> /opt/gradle-5.0/bin/gradle --console=plain runShadow\n",
+ "\u001b[m> Task :compileJava\n",
+ "> Task :processResources NO-SOURCE\n",
+ "> Task :classes\n",
+ "> Task :shadowJar\n",
+ "> Task :startShadowScripts\n",
+ "> Task :installShadowDist\n",
+ "\n",
+ "> Task :runShadow\n",
+ "WARNING: An illegal reflective access operation has occurred\n",
+ "WARNING: Illegal reflective access by org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.UnsafeUtil (file:/content/build/install/content-shadow/lib/WordCount.jar) to field java.nio.Buffer.address\n",
+ "WARNING: Please consider reporting this to the maintainers of org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.UnsafeUtil\n",
+ "WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations\n",
+ "WARNING: All illegal access operations will be denied in a future release\n",
+ "Mar 04, 2019 11:01:26 PM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes\n",
+ "INFO: Filepattern data/* matched 1 files with total size 157283\n",
+ "Mar 04, 2019 11:01:26 PM org.apache.beam.sdk.io.FileBasedSource split\n",
+ "INFO: Splitting filepattern data/* into bundles of size 52427 took 1 ms and produced 1 files and 3 bundles\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement\n",
+ "INFO: Opening writer e2eeada2-5a8b-4493-acc5-c706204d9669 for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@8c3619e pane PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} destination null\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement\n",
+ "INFO: Opening writer 7acdc85e-ff7d-42d0-9b2f-9ce385956c0e for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@8c3619e pane PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} destination null\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement\n",
+ "INFO: Opening writer d1a6a591-77f0-4994-affc-d83378e7b7c0 for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@8c3619e pane PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} destination null\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.FileBasedSink$Writer close\n",
+ "INFO: Successfully wrote temporary file /content/outputs/.temp-beam-2019-03-04_23-01-25-1/e2eeada2-5a8b-4493-acc5-c706204d9669\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.FileBasedSink$Writer close\n",
+ "INFO: Successfully wrote temporary file /content/outputs/.temp-beam-2019-03-04_23-01-25-1/7acdc85e-ff7d-42d0-9b2f-9ce385956c0e\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.FileBasedSink$Writer close\n",
+ "INFO: Successfully wrote temporary file /content/outputs/.temp-beam-2019-03-04_23-01-25-1/d1a6a591-77f0-4994-affc-d83378e7b7c0\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn process\n",
+ "INFO: Finalizing 3 file results\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation createMissingEmptyShards\n",
+ "INFO: Finalizing for destination null num shards 3.\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation moveToOutputFiles\n",
+ "INFO: Will copy temporary file FileResult{tempFilename=/content/outputs/.temp-beam-2019-03-04_23-01-25-1/7acdc85e-ff7d-42d0-9b2f-9ce385956c0e, shard=2, window=org.apache.beam.sdk.transforms.windowing.GlobalWindow@8c3619e, paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}} to final location /content/outputs/part-00002-of-00003\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation moveToOutputFiles\n",
+ "INFO: Will copy temporary file FileResult{tempFilename=/content/outputs/.temp-beam-2019-03-04_23-01-25-1/e2eeada2-5a8b-4493-acc5-c706204d9669, shard=0, window=org.apache.beam.sdk.transforms.windowing.GlobalWindow@8c3619e, paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}} to final location /content/outputs/part-00000-of-00003\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation moveToOutputFiles\n",
+ "INFO: Will copy temporary file FileResult{tempFilename=/content/outputs/.temp-beam-2019-03-04_23-01-25-1/d1a6a591-77f0-4994-affc-d83378e7b7c0, shard=1, window=org.apache.beam.sdk.transforms.windowing.GlobalWindow@8c3619e, paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}} to final location /content/outputs/part-00001-of-00003\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation removeTemporaryFiles\n",
+ "INFO: Will remove known temporary file /content/outputs/.temp-beam-2019-03-04_23-01-25-1/e2eeada2-5a8b-4493-acc5-c706204d9669\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation removeTemporaryFiles\n",
+ "INFO: Will remove known temporary file /content/outputs/.temp-beam-2019-03-04_23-01-25-1/7acdc85e-ff7d-42d0-9b2f-9ce385956c0e\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation removeTemporaryFiles\n",
+ "INFO: Will remove known temporary file /content/outputs/.temp-beam-2019-03-04_23-01-25-1/d1a6a591-77f0-4994-affc-d83378e7b7c0\n",
+ "Mar 04, 2019 11:01:46 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation removeTemporaryFiles\n",
+ "WARNING: Failed to match temporary files under: [/content/outputs/.temp-beam-2019-03-04_23-01-25-1/].\n",
+ "\n",
+ "BUILD SUCCESSFUL in 33s\n",
+ "5 actionable tasks: 5 executed\n",
+ "\u001b[m\n",
+ ">> head -n 20 outputs/part-00000-of-*\n",
+ "==> outputs/part-00000-of-00001 <==\n",
+ "(u'canker', 1)\n",
+ "(u'bounty', 2)\n",
+ "(u'provision', 3)\n",
+ "(u'to', 438)\n",
+ "(u'terms', 2)\n",
+ "(u'unnecessary', 2)\n",
+ "(u'tongue', 5)\n",
+ "(u'knives', 1)\n",
+ "(u'Commend', 1)\n",
+ "(u'Hum', 2)\n",
+ "(u'Set', 2)\n",
+ "(u'smell', 6)\n",
+ "(u'dreadful', 3)\n",
+ "(u'frowning', 1)\n",
+ "(u'World', 1)\n",
+ "(u'tike', 1)\n",
+ "(u'yes', 3)\n",
+ "(u'oldness', 1)\n",
+ "(u'boat', 1)\n",
+ "(u\"in's\", 1)\n",
+ "\n",
+ "==> outputs/part-00000-of-00003 <==\n",
+ "wrath: 3\n",
+ "nicely: 2\n",
+ "hall: 1\n",
+ "Sure: 2\n",
+ "legs: 4\n",
+ "ten: 1\n",
+ "yourselves: 1\n",
+ "embossed: 1\n",
+ "poorly: 1\n",
+ "temper: 2\n",
+ "Dismissing: 1\n",
+ "Legitimate: 1\n",
+ "tyrannous: 1\n",
+ "turn: 13\n",
+ "gold: 2\n",
+ "minds: 1\n",
+ "dowers: 2\n",
+ "policy: 1\n",
+ "I: 708\n",
+ "V: 6\n",
+ "\n",
+ "==> outputs/part-00000-of-00004 <==\n",
+ "retinue: 1\n",
+ "stink: 1\n",
+ "beaks: 1\n",
+ "Ten: 1\n",
+ "riots: 2\n",
+ "Service: 1\n",
+ "dealing: 1\n",
+ "stop: 2\n",
+ "detain: 1\n",
+ "beware: 1\n",
+ "pilferings: 1\n",
+ "swimming: 1\n",
+ "The: 124\n",
+ "Been: 1\n",
+ "behavior: 1\n",
+ "impetuous: 1\n",
+ "Thy: 20\n",
+ "Tis: 24\n",
+ "Soldiers: 7\n",
+ "Juno: 1\n",
+ "\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ }
+ ]
+}
\ No newline at end of file
diff --git a/examples/notebooks/get-started/try-apache-beam-py.ipynb b/examples/notebooks/get-started/try-apache-beam-py.ipynb
new file mode 100644
index 0000000..50dda3a
--- /dev/null
+++ b/examples/notebooks/get-started/try-apache-beam-py.ipynb
@@ -0,0 +1,410 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Try Apache Beam - Python",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python2",
+ "display_name": "Python 2"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "lNKIMlEDZ_Vw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Try Apache Beam - Python\n",
+ "\n",
+ "In this notebook, we set up your development environment and work through a simple example using the [DirectRunner](https://beam.apache.org/documentation/runners/direct/). You can explore other runners with the [Beam Capatibility Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From **View** drop-down list, select **Table of contents**.\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell, or by select it and press **`Shift+Enter`**. Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "To learn more about Colab, see [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Fz6KSQ13_3Rr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Setup\n",
+ "\n",
+ "First, you need to set up your environment, which includes installing `apache-beam` and downloading a text file from Cloud Storage to your local file system. We are using this file to test your pipeline."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GOOk81Jj_yUy",
+ "colab_type": "code",
+ "outputId": "d283dfb2-4f51-4fec-816b-f57b0cb9b71c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 170
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Run and print a shell command.\n",
+ "def run(cmd):\n",
+ " print('>> {}'.format(cmd))\n",
+ " !{cmd}\n",
+ " print('')\n",
+ "\n",
+ "# Install apache-beam.\n",
+ "run('pip install --quiet apache-beam')\n",
+ "\n",
+ "# Copy the input file into the local file system.\n",
+ "run('mkdir -p data')\n",
+ "run('gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/')"
+ ],
+ "execution_count": 1,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ ">> pip install --quiet apache-beam\n",
+ "\n",
+ ">> mkdir -p data\n",
+ "\n",
+ ">> gsutil cp gs://dataflow-samples/shakespeare/kinglear.txt data/\n",
+ "Copying gs://dataflow-samples/shakespeare/kinglear.txt...\n",
+ "/ [1 files][153.6 KiB/153.6 KiB] \n",
+ "Operation completed over 1 objects/153.6 KiB. \n",
+ "\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "cPvvFB19uXNw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Minimal word count\n",
+ "\n",
+ "The following example is the \"Hello, World!\" of data processing, a basic implementation of word count. We're creating a simple data processing pipeline that reads a text file and counts the number of occurrences of every word.\n",
+ "\n",
+ "There are many scenarios where all the data does not fit in memory. Notice that the outputs of the pipeline go to the file system, which allows for large processing jobs in distributed environments."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "oUqfqWyMuIfR",
+ "colab_type": "code",
+ "outputId": "52de90fb-abcc-46c7-bccc-5b0416beb50c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1173
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "import apache_beam as beam\n",
+ "import re\n",
+ "\n",
+ "inputs_pattern = 'data/*'\n",
+ "outputs_prefix = 'outputs/part'\n",
+ "\n",
+ "# Running locally in the DirectRunner.\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " (\n",
+ " pipeline\n",
+ " | 'Read lines' >> beam.io.ReadFromText(inputs_pattern)\n",
+ " | 'Find words' >> beam.FlatMap(lambda line: re.findall(r\"[a-zA-Z']+\", line))\n",
+ " | 'Pair words with 1' >> beam.Map(lambda word: (word, 1))\n",
+ " | 'Group and sum' >> beam.CombinePerKey(sum)\n",
+ " | 'Format results' >> beam.Map(lambda word_count: str(word_count))\n",
+ " | 'Write results' >> beam.io.WriteToText(outputs_prefix)\n",
+ " )\n",
+ "\n",
+ "# Sample the first 20 results, remember there are no ordering guarantees.\n",
+ "run('head -n 20 {}-00000-of-*'.format(outputs_prefix))"
+ ],
+ "execution_count": 2,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "WARNING:root:Deleting 1 existing files in target path matching: -*-of-%(num_shards)05d\n"
+ ],
+ "name": "stderr"
+ },
+ {
+ "output_type": "stream",
+ "text": [
+ ">> head -n 20 outputs/part-00000-of-*\n",
+ "==> outputs/part-00000-of-00001 <==\n",
+ "(u'canker', 1)\n",
+ "(u'bounty', 2)\n",
+ "(u'provision', 3)\n",
+ "(u'to', 438)\n",
+ "(u'terms', 2)\n",
+ "(u'unnecessary', 2)\n",
+ "(u'tongue', 5)\n",
+ "(u'knives', 1)\n",
+ "(u'Commend', 1)\n",
+ "(u'Hum', 2)\n",
+ "(u'Set', 2)\n",
+ "(u'smell', 6)\n",
+ "(u'dreadful', 3)\n",
+ "(u'frowning', 1)\n",
+ "(u'World', 1)\n",
+ "(u'tike', 1)\n",
+ "(u'yes', 3)\n",
+ "(u'oldness', 1)\n",
+ "(u'boat', 1)\n",
+ "(u\"in's\", 1)\n",
+ "\n",
+ "==> outputs/part-00000-of-00003 <==\n",
+ "wrath: 3\n",
+ "nicely: 2\n",
+ "hall: 1\n",
+ "Sure: 2\n",
+ "legs: 4\n",
+ "ten: 1\n",
+ "yourselves: 1\n",
+ "embossed: 1\n",
+ "poorly: 1\n",
+ "temper: 2\n",
+ "Dismissing: 1\n",
+ "Legitimate: 1\n",
+ "tyrannous: 1\n",
+ "turn: 13\n",
+ "gold: 2\n",
+ "minds: 1\n",
+ "dowers: 2\n",
+ "policy: 1\n",
+ "I: 708\n",
+ "V: 6\n",
+ "\n",
+ "==> outputs/part-00000-of-00004 <==\n",
+ "retinue: 1\n",
+ "stink: 1\n",
+ "beaks: 1\n",
+ "Ten: 1\n",
+ "riots: 2\n",
+ "Service: 1\n",
+ "dealing: 1\n",
+ "stop: 2\n",
+ "detain: 1\n",
+ "beware: 1\n",
+ "pilferings: 1\n",
+ "swimming: 1\n",
+ "The: 124\n",
+ "Been: 1\n",
+ "behavior: 1\n",
+ "impetuous: 1\n",
+ "Thy: 20\n",
+ "Tis: 24\n",
+ "Soldiers: 7\n",
+ "Juno: 1\n",
+ "\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "k-HubCrk-h_G",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Word count with comments\n",
+ "\n",
+ "Below is mostly the same code as above, but with comments explaining every line in more detail."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "x_D7sxUHFzUp",
+ "colab_type": "code",
+ "outputId": "44c926df-aa4a-4bea-9247-27c7cb537717",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1173
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "import apache_beam as beam\n",
+ "import re\n",
+ "\n",
+ "inputs_pattern = 'data/*'\n",
+ "outputs_prefix = 'outputs/part'\n",
+ "\n",
+ "# Running locally in the DirectRunner.\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " # Store the word counts in a PCollection.\n",
+ " # Each element is a tuple of (word, count) of types (str, int).\n",
+ " word_counts = (\n",
+ " # The input PCollection is an empty pipeline.\n",
+ " pipeline\n",
+ "\n",
+ " # Read lines from a text file.\n",
+ " | 'Read lines' >> beam.io.ReadFromText(inputs_pattern)\n",
+ " # Element type: str - text line\n",
+ "\n",
+ " # Use a regular expression to iterate over all words in the line.\n",
+ " # FlatMap will yield an element for every element in an iterable.\n",
+ " | 'Find words' >> beam.FlatMap(lambda line: re.findall(r\"[a-zA-Z']+\", line))\n",
+ " # Element type: str - word\n",
+ "\n",
+ " # Create key-value pairs where the value is 1, this way we can group by\n",
+ " # the same word while adding those 1s and get the counts for every word.\n",
+ " | 'Pair words with 1' >> beam.Map(lambda word: (word, 1))\n",
+ " # Element type: (str, int) - key: word, value: 1\n",
+ "\n",
+ " # Group by key while combining the value using the sum() function.\n",
+ " | 'Group and sum' >> beam.CombinePerKey(sum)\n",
+ " # Element type: (str, int) - key: word, value: counts\n",
+ " )\n",
+ "\n",
+ " # We can process a PCollection through other pipelines too.\n",
+ " (\n",
+ " # The input PCollection is the word_counts created from the previous step.\n",
+ " word_counts\n",
+ "\n",
+ " # Format the results into a string so we can write them to a file.\n",
+ " | 'Format results' >> beam.Map(lambda word_count: str(word_count))\n",
+ " # Element type: str - text line\n",
+ "\n",
+ " # Finally, write the results to a file.\n",
+ " | 'Write results' >> beam.io.WriteToText(outputs_prefix)\n",
+ " )\n",
+ "\n",
+ "# Sample the first 20 results, remember there are no ordering guarantees.\n",
+ "run('head -n 20 {}-00000-of-*'.format(outputs_prefix))"
+ ],
+ "execution_count": 3,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "WARNING:root:Deleting 1 existing files in target path matching: -*-of-%(num_shards)05d\n"
+ ],
+ "name": "stderr"
+ },
+ {
+ "output_type": "stream",
+ "text": [
+ ">> head -n 20 outputs/part-00000-of-*\n",
+ "==> outputs/part-00000-of-00001 <==\n",
+ "(u'canker', 1)\n",
+ "(u'bounty', 2)\n",
+ "(u'provision', 3)\n",
+ "(u'to', 438)\n",
+ "(u'terms', 2)\n",
+ "(u'unnecessary', 2)\n",
+ "(u'tongue', 5)\n",
+ "(u'knives', 1)\n",
+ "(u'Commend', 1)\n",
+ "(u'Hum', 2)\n",
+ "(u'Set', 2)\n",
+ "(u'smell', 6)\n",
+ "(u'dreadful', 3)\n",
+ "(u'frowning', 1)\n",
+ "(u'World', 1)\n",
+ "(u'tike', 1)\n",
+ "(u'yes', 3)\n",
+ "(u'oldness', 1)\n",
+ "(u'boat', 1)\n",
+ "(u\"in's\", 1)\n",
+ "\n",
+ "==> outputs/part-00000-of-00003 <==\n",
+ "wrath: 3\n",
+ "nicely: 2\n",
+ "hall: 1\n",
+ "Sure: 2\n",
+ "legs: 4\n",
+ "ten: 1\n",
+ "yourselves: 1\n",
+ "embossed: 1\n",
+ "poorly: 1\n",
+ "temper: 2\n",
+ "Dismissing: 1\n",
+ "Legitimate: 1\n",
+ "tyrannous: 1\n",
+ "turn: 13\n",
+ "gold: 2\n",
+ "minds: 1\n",
+ "dowers: 2\n",
+ "policy: 1\n",
+ "I: 708\n",
+ "V: 6\n",
+ "\n",
+ "==> outputs/part-00000-of-00004 <==\n",
+ "retinue: 1\n",
+ "stink: 1\n",
+ "beaks: 1\n",
+ "Ten: 1\n",
+ "riots: 2\n",
+ "Service: 1\n",
+ "dealing: 1\n",
+ "stop: 2\n",
+ "detain: 1\n",
+ "beware: 1\n",
+ "pilferings: 1\n",
+ "swimming: 1\n",
+ "The: 124\n",
+ "Been: 1\n",
+ "behavior: 1\n",
+ "impetuous: 1\n",
+ "Thy: 20\n",
+ "Tis: 24\n",
+ "Soldiers: 7\n",
+ "Juno: 1\n",
+ "\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ }
+ ]
+}
\ No newline at end of file
diff --git a/examples/notebooks/patch.py b/examples/notebooks/patch.py
new file mode 100644
index 0000000..1979823
--- /dev/null
+++ b/examples/notebooks/patch.py
@@ -0,0 +1,96 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Script to recursively patch Colab notebooks.
+Replaces links to point to the master branch and adds a license.
+
+Usage:
+ From root directory, execute
+ 'python examples/notebooks/patch.py'.
+"""
+import argparse
+import collections
+import json
+import os
+import re
+
+license_text = """\
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+""".splitlines()
+
+colab_url = 'https://colab.research.google.com/github'
+github_url = 'https://github.com'
+branch_repo_re = re.compile(r'[\w-]+/[\w-]+/blob/[\w-]+')
+target_branch_repo = 'apache/beam/blob/master'
+
+
+def run(root_dir):
+ for path, dirs, files in os.walk(root_dir):
+ for filename in files:
+ if filename.endswith('.ipynb'):
+ patch_notebook(os.path.join(path, filename))
+
+
+def patch_notebook(full_path):
+ # Replace the branch repo URL.
+ lines = []
+ with open(full_path) as f:
+ for line in f:
+ if colab_url in line or github_url in line:
+ line = branch_repo_re.sub(target_branch_repo, line)
+ lines += [line]
+ content = ''.join(lines)
+
+ # Add the license block.
+ decoder = json.JSONDecoder(object_pairs_hook=collections.OrderedDict)
+ elements = decoder.decode(content).items()
+ first_key, _ = elements[0]
+ if first_key != 'license':
+ print('Patching {}'.format(full_path))
+ elements.insert(0, ('license', license_text))
+
+ # Overwrite the file with the new contents.
+ with open(full_path, 'w') as f:
+ json.dump(collections.OrderedDict(elements), f, indent=2)
+
+
+if __name__ == '__main__':
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ '--root-dir',
+ type=str,
+ default='examples/notebooks',
+ help='Root directory to recursively patch *.ipynb notebooks.',
+ )
+ args = parser.parse_args()
+
+ run(args.root_dir)
diff --git a/website/_config.yml b/website/_config.yml
index 9ad9731..879ffa8 100644
--- a/website/_config.yml
+++ b/website/_config.yml
@@ -34,6 +34,12 @@ source: src
# the base hostname & protocol for your site
url: "https://beam.apache.org"
+# The repository and branch where the files live in Github or Colab. This is used
+# to serve and stage from your local branch, but publish to the master branch.
+# e.g. https://github.com/{{ site.branch_repo }}/path/to/notebook.ipynb
+# e.g. https://colab.sandbox.google.com/github/{{ site.branch_repo }}/path/to/notebook.ipynb
+branch_repo: "apache/beam/blob/master"
+
twitter_username: apachebeam
# Build settings
diff --git a/website/build.gradle b/website/build.gradle
index 175cff2..761ac3e 100644
--- a/website/build.gradle
+++ b/website/build.gradle
@@ -83,6 +83,18 @@ task startDockerContainer(type: Exec) {
"${->createDockerContainer.containerId()}" // Lazily evaluate containerId.
}
+task setupDockerContainer(type: Exec) {
+ dependsOn startDockerContainer
+ ext.containerId = {
+ return startDockerContainer.containerId()
+ }
+
+ // Create the config to point to a GitHub or Colab blob in the repo, e.g. apache/beam/blob/master
+ commandLine 'docker', 'exec', '-u', 'root',
+ "${->startDockerContainer.containerId()}", '/bin/bash', '-c',
+ """echo 'branch_repo: "${getBranchRepo()}"' > /tmp/_config_branch_repo.yml"""
+}
+
task stopAndRemoveDockerContainer(type: Exec) {
commandLine 'docker', 'rm', '-f', "${->createDockerContainer.containerId()}"
}
@@ -103,13 +115,14 @@ clean.dependsOn cleanWebsite
class BuildTaskConfiguration {
String name
boolean useTestConfig = false
+ boolean useBranchRepoConfig = false
String baseUrl = ''
String dockerWorkDir = ''
}
def createBuildTask = {
BuildTaskConfiguration config = it as BuildTaskConfiguration
task "build${config.name}Website" (type:Exec) {
- dependsOn startDockerContainer, setupBuildDir
+ dependsOn setupDockerContainer, setupBuildDir
finalizedBy stopAndRemoveDockerContainer
inputs.files 'Gemfile.lock', '_config.yml'
inputs.dir 'src'
@@ -119,23 +132,46 @@ def createBuildTask = {
if (config.useTestConfig) {
configs += ",${config.dockerWorkDir}/website/_config_test.yml"
}
+ if (config.useBranchRepoConfig) {
+ configs += ",/tmp/_config_branch_repo.yml"
+ }
def baseUrlFlag = config.baseUrl ? "--baseurl=/${config.baseUrl}" : ""
commandLine 'docker', 'exec',
- "${->startDockerContainer.containerId()}", '/bin/bash', '-c',
+ "${->setupDockerContainer.containerId()}", '/bin/bash', '-c',
"""cd ${config.dockerWorkDir}/build/website && \
bundle exec jekyll build \
- --destination generated-${config.name.toLowerCase()}-content \
+ --destination generated-${config.name.toLowerCase()}-content \
--config ${configs} \
--incremental ${baseUrlFlag} \
--source ${config.dockerWorkDir}/website/src
"""
}
}
-createBuildTask(name:'Local', useTestConfig: true, dockerWorkDir: dockerWorkDir)
+
+// task buildLocalWebsite
+createBuildTask(
+ name:'Local',
+ useTestConfig: true,
+ useBranchRepoConfig: true,
+ dockerWorkDir: dockerWorkDir,
+)
task buildWebsite(dependsOn:buildLocalWebsite)
build.dependsOn buildWebsite
-createBuildTask( name:'Gcs', useTestConfig: true, baseUrl: getBaseUrl(), dockerWorkDir: dockerWorkDir)
-createBuildTask( name:'Apache', dockerWorkDir: dockerWorkDir)
+
+// task buildGcsWebsite
+createBuildTask(
+ name:'Gcs',
+ useTestConfig: true,
+ useBranchRepoConfig: true,
+ baseUrl: getBaseUrl(),
+ dockerWorkDir: dockerWorkDir,
+)
+
+// task buildApacheWebsite
+createBuildTask(
+ name:'Apache',
+ dockerWorkDir: dockerWorkDir,
+)
/**
* Use the pull request ID in the URL path, or the git branch when building locally.
@@ -152,22 +188,55 @@ def getBaseUrl() {
return "${System.getProperty('user.name')}-${buildContext}"
}
+/**
+ * Gets the branch repository where the new files are located. This is used to point
+ * all the GitHub and Colab files to the pull request's branch when testing, while
+ * pointing them to the apache/beam master branch on production. This is done so tests
+ * and staged versions work even when the files don't exist in master yet.
+ */
+def getBranchRepo() {
+ // Jenkins stores the GitHub author in $ghprbPullAuthorLogin
+ def author = System.env.ghprbPullAuthorLogin
+ if (author == null) {
+ // If the author is not defined, it's most probably running locally.
+ // Try to infer the author from the remote URL
+ for (remote in grgit.remote.list()) {
+ if (remote.getName() == 'origin') {
+ // remote.url = 'git@github.com:author/beam.git'
+ author = remote.url.split(':')[1].split('/')[0]
+ break
+ }
+ }
+ }
+ // Jenkins stores the branch in botho of the following environment variables.
+ def branch = System.env.ghprbSourceBranch ?: System.env.GIT_BRANCH
+ if (branch == null && grgit) {
+ branch = grgit.branch.current().getName()
+ }
+
+ // Return the author's branch repo, otherwise default to the master repo.
+ if (author && branch) {
+ return "${author}/beam/blob/${branch}"
+ }
+ return "apache/beam/blob/master"
+}
+
def buildContentDir(name) {
"${project.rootDir}/build/website/generated-${name.toLowerCase()}-content"
}
task serveWebsite(type: Exec) {
- dependsOn startDockerContainer, setupBuildDir
+ dependsOn setupDockerContainer, setupBuildDir
finalizedBy stopAndRemoveDockerContainer
inputs.files 'Gemfile.lock', '_config.yml'
inputs.dir 'src'
outputs.dir "$buildDir/.sass-cache"
outputs.dir buildContentDir('local')
commandLine 'docker', 'exec',
- "${->startDockerContainer.containerId()}", '/bin/bash', '-c',
+ "${->setupDockerContainer.containerId()}", '/bin/bash', '-c',
"""cd $dockerWorkDir/build/website && \
bundle exec jekyll serve \
- --config $dockerWorkDir/website/_config.yml \
+ --config $dockerWorkDir/website/_config.yml,/tmp/_config_branch_repo.yml \
--incremental \
--source $dockerWorkDir/website/src \
--host 0.0.0.0
@@ -175,13 +244,13 @@ task serveWebsite(type: Exec) {
}
task testWebsite(type: Exec) {
- // dependsOn startDockerContainer, 'buildWebsite'
+ // dependsOn setupDockerContainer, 'buildWebsite'
finalizedBy stopAndRemoveDockerContainer
inputs.files "$buildDir/Rakefile"
inputs.dir buildContentDir('local')
commandLine 'docker', 'exec',
- "${->startDockerContainer.containerId()}", '/bin/bash', '-c',
+ "${->setupDockerContainer.containerId()}", '/bin/bash', '-c',
"""cd $dockerWorkDir/build/website && \
bundle exec rake test"""
}
diff --git a/website/src/_includes/section-menu/get-started.html b/website/src/_includes/section-menu/get-started.html
index 860b483..61270a4 100644
--- a/website/src/_includes/section-menu/get-started.html
+++ b/website/src/_includes/section-menu/get-started.html
@@ -16,6 +16,7 @@
<span class="section-nav-list-title">Quickstarts</span>
<ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/get-started/try-apache-beam/">Try Apache Beam</a></li>
<li><a href="{{ site.baseurl }}/get-started/quickstart-java/">Quickstart - Java</a></li>
<li><a href="{{ site.baseurl }}/get-started/quickstart-py/">Quickstart - Python</a></li>
<li><a href="{{ site.baseurl }}/get-started/quickstart-go/">Quickstart - Go</a></li>
diff --git a/website/src/get-started/beam-overview.md b/website/src/get-started/beam-overview.md
index 5dddf94..6d70ab84 100644
--- a/website/src/get-started/beam-overview.md
+++ b/website/src/get-started/beam-overview.md
@@ -58,13 +58,15 @@ Beam currently supports Runners that work with the following distributed process
Get started using Beam for your data processing tasks.
+1. [Try Apache Beam]({{ stie.baseurl }}/get-started/try-apache-beam) in an online interactive environment.
+
1. Follow the Quickstart for the [Java SDK]({{ site.baseurl }}/get-started/quickstart-java), the [Python SDK]({{ site.baseurl }}/get-started/quickstart-py) or the [Go SDK]({{ site.baseurl }}/get-started/quickstart-go).
-2. See the [WordCount Examples Walkthrough]({{ site.baseurl }}/get-started/wordcount-example) for examples that introduce various features of the SDKs.
+1. See the [WordCount Examples Walkthrough]({{ site.baseurl }}/get-started/wordcount-example) for examples that introduce various features of the SDKs.
-3. Take a self-paced tour through our [Learning Resources]({{ site.baseurl }}/documentation/resources/learning-resources).
+1. Take a self-paced tour through our [Learning Resources]({{ site.baseurl }}/documentation/resources/learning-resources).
-4. Dive into the [Documentation]({{ site.baseurl }}/documentation/) section for in-depth concepts and reference materials for the Beam model, SDKs, and runners.
+1. Dive into the [Documentation]({{ site.baseurl }}/documentation/) section for in-depth concepts and reference materials for the Beam model, SDKs, and runners.
## Contribute
diff --git a/website/src/get-started/try-apache-beam.md b/website/src/get-started/try-apache-beam.md
new file mode 100644
index 0000000..6ec16b2
--- /dev/null
+++ b/website/src/get-started/try-apache-beam.md
@@ -0,0 +1,192 @@
+---
+layout: section
+title: "Try Apache Beam"
+permalink: /get-started/try-apache-beam/
+section_menu: section-menu/get-started.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Try Apache Beam
+
+You can try Apache Beam using our interactive notebooks, which are hosted in [Colab](https://colab.research.google.com). The notebooks allow you to interactively play with the code and see how your changes affect the pipeline. You don't need to install anything or modify your computer in any way to use these notebooks.
+
+<nav class="language-switcher">
+ <strong>Adapt for:</strong>
+ <ul>
+ <li data-type="language-java" class="active">Java SDK</li>
+ <li data-type="language-py">Python SDK</li>
+ <li data-type="language-go">Go SDK</li>
+ </ul>
+</nav>
+
+## Interactive WordCount in Colab
+
+This interactive notebook shows you what a simple, minimal version of WordCount looks like.
+
+```java
+package samples.quickstart;
+
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Filter;
+import org.apache.beam.sdk.transforms.FlatMapElements;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.TypeDescriptors;
+
+import java.util.Arrays;
+
+public class WordCount {
+ public static void main(String[] args) {
+ String inputsDir = "data/*";
+ String outputsPrefix = "outputs/part";
+
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+ pipeline
+ .apply("Read lines", TextIO.read().from(inputsDir))
+ .apply("Find words", FlatMapElements.into(TypeDescriptors.strings())
+ .via((String line) -> Arrays.asList(line.split("[^\\p{L}]+"))))
+ .apply("Filter empty words", Filter.by((String word) -> !word.isEmpty()))
+ .apply("Count words", Count.perElement())
+ .apply("Write results", MapElements.into(TypeDescriptors.strings())
+ .via((KV<String, Long> wordCount) ->
+ wordCount.getKey() + ": " + wordCount.getValue()))
+ .apply(TextIO.write().to(outputsPrefix));
+ pipeline.run();
+ }
+}
+```
+
+{:.language-java}
+<a class="button button--primary" target="_blank"
+ href="https://colab.sandbox.google.com/github/{{ site.branch_repo }}/examples/notebooks/get-started/try-apache-beam-java.ipynb">
+ Run in Colab
+</a>
+<a class="button button--primary" target="_blank"
+ href="https://github.com/{{ site.branch_repo }}/examples/notebooks/get-started/try-apache-beam-java.ipynb">
+ View on GitHub
+</a>
+
+{:.language-java}
+To learn how to install and run the Apache Beam Java SDK on your own computer, follow the instructions in the [Java Quickstart]({{ site.baseurl }}/get-started/quickstart-java).
+
+```py
+import apache_beam as beam
+import re
+
+inputs_pattern = 'data/*'
+outputs_prefix = 'outputs/part'
+
+with beam.Pipeline() as pipeline:
+ (
+ pipeline
+ | 'Read lines' >> beam.io.ReadFromText(inputs_pattern)
+ | 'Find words' >> beam.FlatMap(lambda line: re.findall(r"[a-zA-Z']+", line))
+ | 'Pair words with 1' >> beam.Map(lambda word: (word, 1))
+ | 'Group and sum' >> beam.CombinePerKey(sum)
+ | 'Format results' >> beam.Map(lambda word_count: str(word_count))
+ | 'Write results' >> beam.io.WriteToText(outputs_prefix)
+ )
+```
+
+{:.language-py}
+<a class="button button--primary" target="_blank"
+ href="https://colab.sandbox.google.com/github/{{ site.branch_repo }}/examples/notebooks/get-started/try-apache-beam-py.ipynb">
+ Run in Colab
+</a>
+<a class="button button--primary" target="_blank"
+ href="https://github.com/{{ site.branch_repo }}/examples/notebooks/get-started/try-apache-beam-py.ipynb">
+ View on GitHub
+</a>
+
+{:.language-py}
+To learn how to install and run the Apache Beam Python SDK on your own computer, follow the instructions in the [Python Quickstart]({{ site.baseurl }}/get-started/quickstart-py).
+
+```go
+package main
+
+import (
+ "context"
+ "flag"
+ "fmt"
+ "regexp"
+
+ "github.com/apache/beam/sdks/go/pkg/beam"
+ "github.com/apache/beam/sdks/go/pkg/beam/io/textio"
+ "github.com/apache/beam/sdks/go/pkg/beam/runners/direct"
+ "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats"
+
+ _ "github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local"
+)
+
+var (
+ input = flag.String("input", "data/*", "File(s) to read.")
+ output = flag.String("output", "outputs/wordcounts.txt", "Output filename.")
+)
+
+var wordRE = regexp.MustCompile(`[a-zA-Z]+('[a-z])?`)
+
+func main() {
+ flag.Parse()
+
+ beam.Init()
+
+ pipeline := beam.NewPipeline()
+ root := pipeline.Root()
+
+ lines := textio.Read(root, *input)
+ words := beam.ParDo(root, func(line string, emit func(string)) {
+ for _, word := range wordRE.FindAllString(line, -1) {
+ emit(word)
+ }
+ }, lines)
+ counted := stats.Count(root, words)
+ formatted := beam.ParDo(root, func(word string, count int) string {
+ return fmt.Sprintf("%s: %v", word, count)
+ }, counted)
+ textio.Write(root, *output, formatted)
+
+ direct.Execute(context.Background(), pipeline)
+}
+```
+
+{:.language-go}
+<a class="button button--primary" target="_blank"
+ href="https://colab.sandbox.google.com/github/{{ site.branch_repo }}/examples/notebooks/get-started/try-apache-beam-go.ipynb">
+ Run in Colab
+</a>
+<a class="button button--primary" target="_blank"
+ href="https://github.com/{{ site.branch_repo }}/examples/notebooks/get-started/try-apache-beam-go.ipynb">
+ View on GitHub
+</a>
+
+{:.language-go}
+To learn how to install and run the Apache Beam Go SDK on your own computer, follow the instructions in the [Go Quickstart]({{ site.baseurl }}/get-started/quickstart-go).
+
+For a more detailed explanation about how WordCount works, see the [WordCount Example Walkthrough]({{ site.baseurl }}/get-started/wordcount-example).
+
+## Next Steps
+
+* Walk through additional WordCount examples in the [WordCount Example Walkthrough]({{ site.baseurl }}/get-started/wordcount-example).
+* Take a self-paced tour through our [Learning Resources]({{ site.baseurl }}/documentation/resources/learning-resources).
+* Dive in to some of our favorite [Videos and Podcasts]({{ site.baseurl }}/documentation/resources/videos-and-podcasts).
+* Join the Beam [users@]({{ site.baseurl }}/community/contact-us) mailing list.
+* If you're interested in contributing to the Apache Beam codebase, see the [Contribution Guide]({{ site.baseurl }}/contribute).
+
+Please don't hesitate to [reach out]({{ site.baseurl }}/community/contact-us) if you encounter any issues!
diff --git a/website/src/index.md b/website/src/index.md
index 26df873..7559a3e 100644
--- a/website/src/index.md
+++ b/website/src/index.md
@@ -64,12 +64,13 @@ limitations under the License.
</div>
<div class="hero__ctas hero__ctas--first">
<a class="button button--primary" href="{{'/get-started/beam-overview/'|prepend:site.baseurl}}">Learn more</a>
+ <a class="button button--primary" href="{{'/get-started/try-apache-beam/'|prepend:site.baseurl}}">Try Beam</a>
<a class="button button--primary" href="{{'/get-started/downloads/'|prepend:site.baseurl}}">Download Beam SDK {{ site.release_latest }}</a>
</div>
<div class="hero__ctas">
<a class="button" href="{{'/get-started/quickstart-java/'|prepend:site.baseurl}}">Java Quickstart</a>
<a class="button" href="{{'/get-started/quickstart-py/'|prepend:site.baseurl}}">Python Quickstart</a>
- <a class="button" href="{{'/get-started/quickstart-go/'|prepend:site.baseurl}}">Go Quickstart</a>
+ <a class="button" href="{{'/get-started/quickstart-go/'|prepend:site.baseurl}}">Go Quickstart</a>
</div>
</div>
</div>