You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by da...@apache.org on 2016/11/16 17:36:11 UTC
[1/3] incubator-beam-site git commit: [BEAM-899] Add Flink
Instructions to quickstart.md
Repository: incubator-beam-site
Updated Branches:
refs/heads/asf-site 3b93d1b56 -> 4e6947c63
[BEAM-899] Add Flink Instructions to quickstart.md
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/a53e4d88
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/a53e4d88
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/a53e4d88
Branch: refs/heads/asf-site
Commit: a53e4d88cebdcaab45d5d7090f40fd4d986c1151
Parents: 3b93d1b
Author: Aljoscha Krettek <al...@gmail.com>
Authored: Sat Nov 12 11:36:49 2016 +0100
Committer: Aljoscha Krettek <al...@gmail.com>
Committed: Wed Nov 16 18:08:41 2016 +0100
----------------------------------------------------------------------
src/get-started/quickstart.md | 73 +++++++++++++++++++++++++++++---------
1 file changed, 56 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/a53e4d88/src/get-started/quickstart.md
----------------------------------------------------------------------
diff --git a/src/get-started/quickstart.md b/src/get-started/quickstart.md
index 0a956d2..07ace51 100644
--- a/src/get-started/quickstart.md
+++ b/src/get-started/quickstart.md
@@ -7,7 +7,7 @@ redirect_from:
- /getting-started/
---
-# Apache Beam Java SDK Quickstart
+# Apache Beam Java SDK Quickstart
This Quickstart will walk you through executing your first Beam pipeline to run [WordCount]({{ site.baseurl }}/get-started/wordcount-example), written using Beam's [Java SDK]({{ site.baseurl }}/documentation/sdks/java), on a [runner]({{ site.baseurl }}/documentation#runners) of your choice.
@@ -16,7 +16,7 @@ This Quickstart will walk you through executing your first Beam pipeline to run
## Set up your Development Environment
-
+
1. Download and install the [Java Development Kit (JDK)](http://www.oracle.com/technetwork/java/javase/downloads/index.html) version 1.7 or later. Verify that the [JAVA_HOME](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html) environment variable is set and points to your JDK installation.
1. Download and install [Apache Maven](http://maven.apache.org/download.cgi) by following Maven's [installation guide](http://maven.apache.org/install.html) for your specific operating system.
@@ -24,7 +24,7 @@ This Quickstart will walk you through executing your first Beam pipeline to run
## Get the WordCount Code
-The easiest way to get a copy of the WordCount pipeline is to use the following command to generate a simple Maven project that contains Beam's WordCount examples and builds against the most recent Beam release:
+The easiest way to get a copy of the WordCount pipeline is to use the following command to generate a simple Maven project that contains Beam's WordCount examples and builds against the most recent Beam release:
```
$ mvn archetype:generate \
@@ -38,7 +38,7 @@ $ mvn archetype:generate \
-Dpackage=org.apache.beam.examples
```
-This will create a directory `word-count-beam` that contains a simple `pom.xml` and a series of example pipelines that count words in text files.
+This will create a directory `word-count-beam` that contains a simple `pom.xml` and a series of example pipelines that count words in text files.
```
$ cd beam-word-count/
@@ -63,7 +63,7 @@ After you've chosen which runner you'd like to use:
1. Ensure you've done any runner-specific setup.
1. Build your commandline by:
1. Specifying a specific runner with `--runner=<runner>` (defaults to the [DirectRunner]({{ site.baseurl }}/documentation/runners/direct))
- 1. Adding any runner-specific required options
+ 1. Adding any runner-specific required options
1. Choosing input files and an output location are accessible on the chosen runner. (For example, you can't access a local file if you are running the pipeline on an external cluster.)
1. Run your first WordCount pipeline.
@@ -74,14 +74,27 @@ $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
```
{:.runner-apex}
-```
+```
$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
-Dexec.args="--inputFile=pom.xml --output=counts --runner=ApexRunner" -Papex-runner
```
-{:.runner-flink}
-```
-TODO BEAM-899
+{:.runner-flink-local}
+```
+$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--runner=FlinkRunner --inputFile=pom.xml --output=counts" -Pflink-runner
+```
+
+{:.runner-flink-cluster}
+```
+$ mvn package -Pflink-runner
+$ cp target/word-count-beam-bundled-0.1.jar /path/to/flink/lib/
+$ bin/flink run -c org.apache.beam.examples.WordCount lib/word-count-beam-0.1.jar \
+ --inputFile=/path/to/quickstart/pom.xml \
+ --output=/tmp/counts \
+ --runner=org.apache.beam.runners.flink.FlinkRunner
+
+You can monitor the running job by visiting the Flink dashboard at http://<flink master>:8081
```
{:.runner-spark}
@@ -111,9 +124,14 @@ $ ls counts*
$ ls counts*
```
-{:.runner-flink}
-```
-TODO BEAM-899
+{:.runner-flink-local}
+```
+$ ls counts*
+```
+
+{:.runner-flink-cluster}
+```
+$ ls /tmp/counts*
```
{:.runner-spark}
@@ -126,7 +144,7 @@ TODO BEAM-900
```
$ gsutil ls gs://<your-gcs-bucket>/counts*
```
-
+
When you look into the contents of the file, you'll see that they contain unique words and the number of occurrences of each word. The order of elements within the file may differ because the Beam model does not generally guarantee ordering, again to allow runners to optimize for efficiency.
{:.runner-direct}
@@ -153,9 +171,30 @@ PAssert: 1
...
```
-{:.runner-flink}
-```
-TODO BEAM-899
+{:.runner-flink-local}
+```
+$ more counts*
+The: 1
+api: 9
+old: 4
+Apache: 2
+limitations: 1
+bundled: 1
+Foundation: 1
+...
+```
+
+{:.runner-flink-cluster}
+```
+$ more /tmp/counts*
+The: 1
+api: 9
+old: 4
+Apache: 2
+limitations: 1
+bundled: 1
+Foundation: 1
+...
```
{:.runner-spark}
@@ -184,4 +223,4 @@ barrenly: 1
* Join the Beam [users@]({{ site.baseurl }}/get-started/support#mailing-lists) mailing list.
Please don't hesitate to [reach out]({{ site.baseurl }}/get-started/support) if you encounter any issues!
-
+
[3/3] incubator-beam-site git commit: This closes #79
Posted by da...@apache.org.
This closes #79
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/4e6947c6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/4e6947c6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/4e6947c6
Branch: refs/heads/asf-site
Commit: 4e6947c63bcc54e9ec7685da54b52af3beb601b0
Parents: 3b93d1b bfbd64c
Author: Davor Bonaci <da...@google.com>
Authored: Wed Nov 16 09:35:39 2016 -0800
Committer: Davor Bonaci <da...@google.com>
Committed: Wed Nov 16 09:35:39 2016 -0800
----------------------------------------------------------------------
content/get-started/quickstart/index.html | 42 +++++++++++++--
src/get-started/quickstart.md | 73 ++++++++++++++++++++------
2 files changed, 95 insertions(+), 20 deletions(-)
----------------------------------------------------------------------
[2/3] incubator-beam-site git commit: Regenerate website
Posted by da...@apache.org.
Regenerate website
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/bfbd64c3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/bfbd64c3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/bfbd64c3
Branch: refs/heads/asf-site
Commit: bfbd64c327b3aea2bc02f5c1f829f6fc1229e703
Parents: a53e4d8
Author: Davor Bonaci <da...@google.com>
Authored: Wed Nov 16 09:35:38 2016 -0800
Committer: Davor Bonaci <da...@google.com>
Committed: Wed Nov 16 09:35:38 2016 -0800
----------------------------------------------------------------------
content/get-started/quickstart/index.html | 42 ++++++++++++++++++++++++--
1 file changed, 39 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/bfbd64c3/content/get-started/quickstart/index.html
----------------------------------------------------------------------
diff --git a/content/get-started/quickstart/index.html b/content/get-started/quickstart/index.html
index 1ef229a..92176b5 100644
--- a/content/get-started/quickstart/index.html
+++ b/content/get-started/quickstart/index.html
@@ -222,7 +222,19 @@ MinimalWordCount.java WordCount.java
</code></pre>
</div>
-<div class="runner-flink highlighter-rouge"><pre class="highlight"><code>TODO BEAM-899
+<div class="runner-flink-local highlighter-rouge"><pre class="highlight"><code>$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--runner=FlinkRunner --inputFile=pom.xml --output=counts" -Pflink-runner
+</code></pre>
+</div>
+
+<div class="runner-flink-cluster highlighter-rouge"><pre class="highlight"><code>$ mvn package -Pflink-runner
+$ cp target/word-count-beam-bundled-0.1.jar /path/to/flink/lib/
+$ bin/flink run -c org.apache.beam.examples.WordCount lib/word-count-beam-0.1.jar \
+ --inputFile=/path/to/quickstart/pom.xml \
+ --output=/tmp/counts \
+ --runner=org.apache.beam.runners.flink.FlinkRunner
+
+You can monitor the running job by visiting the Flink dashboard at http://<flink master>:8081
</code></pre>
</div>
@@ -248,7 +260,11 @@ MinimalWordCount.java WordCount.java
</code></pre>
</div>
-<div class="runner-flink highlighter-rouge"><pre class="highlight"><code>TODO BEAM-899
+<div class="runner-flink-local highlighter-rouge"><pre class="highlight"><code>$ ls counts*
+</code></pre>
+</div>
+
+<div class="runner-flink-cluster highlighter-rouge"><pre class="highlight"><code>$ ls /tmp/counts*
</code></pre>
</div>
@@ -284,7 +300,27 @@ PAssert: 1
</code></pre>
</div>
-<div class="runner-flink highlighter-rouge"><pre class="highlight"><code>TODO BEAM-899
+<div class="runner-flink-local highlighter-rouge"><pre class="highlight"><code>$ more counts*
+The: 1
+api: 9
+old: 4
+Apache: 2
+limitations: 1
+bundled: 1
+Foundation: 1
+...
+</code></pre>
+</div>
+
+<div class="runner-flink-cluster highlighter-rouge"><pre class="highlight"><code>$ more /tmp/counts*
+The: 1
+api: 9
+old: 4
+Apache: 2
+limitations: 1
+bundled: 1
+Foundation: 1
+...
</code></pre>
</div>