You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by xi...@apache.org on 2018/12/21 18:54:17 UTC
samza git commit: SAMZA-2048: Add guide to run Beam wordcount example
Repository: samza
Updated Branches:
refs/heads/master 84d144c51 -> 091caef99
SAMZA-2048: Add guide to run Beam wordcount example
Use the maven archetype to generate the example project for beam wordcount examples. Add the steps to set it up and run the examples.
Author: xiliu <xi...@linkedin.com>
Reviewers: Hai Lu <lh...@apache.org>
Closes #867 from xinyuiscool/SAMZA-2048
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/091caef9
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/091caef9
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/091caef9
Branch: refs/heads/master
Commit: 091caef991236e49856c178f735777decbb49d8f
Parents: 84d144c
Author: Xinyu Liu <xi...@gmail.com>
Authored: Fri Dec 21 10:54:14 2018 -0800
Committer: xinyuiscool <xi...@gmail.com>
Committed: Fri Dec 21 10:54:14 2018 -0800
----------------------------------------------------------------------
docs/startup/code-examples/versioned/index.md | 64 ++++++++++++++++++++++
1 file changed, 64 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/091caef9/docs/startup/code-examples/versioned/index.md
----------------------------------------------------------------------
diff --git a/docs/startup/code-examples/versioned/index.md b/docs/startup/code-examples/versioned/index.md
index 2f4cb5c..96a78af 100644
--- a/docs/startup/code-examples/versioned/index.md
+++ b/docs/startup/code-examples/versioned/index.md
@@ -47,3 +47,67 @@ In addition to the cookbook, you can also consult these:
- [Amazon Kinesis](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/kinesis) and [Azure Eventhubs](https://github.com/apache/samza-hello-samza/tree/latest/src/main/java/samza/examples/azure) examples that cover how to consume input data from the respective systems.
+
+#### Apache Beam API examples
+
+The easiest way to get a copy of the WordCount examples in Beam API is to use [Apache Maven](http://maven.apache.org/download.cgi). After installing Maven, please run the following command:
+
+{% highlight bash %}
+> mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.beam \
+ -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
+ -DarchetypeVersion=2.9.0 \
+ -DgroupId=org.example \
+ -DartifactId=word-count-beam \
+ -Dversion="0.1" \
+ -Dpackage=org.apache.beam.examples \
+ -DinteractiveMode=false
+{% endhighlight %}
+
+This command creates a maven project `word-count-beam` which contains a series of example pipelines that count words in text files:
+
+{% highlight bash %}
+> cd word-count-beam/
+
+> ls src/main/java/org/apache/beam/examples/
+DebuggingWordCount.java WindowedWordCount.java common
+MinimalWordCount.java WordCount.java
+{% endhighlight %}
+
+To use SamzaRunner, please add the following `samza-runner` profile to `pom.xml` under the "profiles" section, same as in [here](https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml).
+
+{% highlight xml %}
+ ...
+ <profile>
+ <id>samza-runner</id>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-runners-samza</artifactId>
+ <version>${beam.version}</version>
+ <scope>runtime</scope>
+ </dependency>
+ </dependencies>
+ </profile>
+ ....
+{% endhighlight %}
+
+Now we can run the wordcount example with Samza using the following command:
+
+{% highlight bash %}
+>mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--inputFile=pom.xml --output=/tmp/counts --runner=SamzaRunner" -Psamza-runner
+{% endhighlight %}
+
+After the pipeline finishes, you can check out the output counts files in /tmp folder. Note Beam generates multiple output files for parallel processing. If you prefer a single output, please update the code to use TextIO.write().withoutSharding().
+
+{% highlight bash %}
+>more /tmp/counts*
+AS: 1
+IO: 2
+IS: 1
+OF: 1
+...
+{% endhighlight %}
+
+A walkthrough of the example code can be found [here](https://beam.apache.org/get-started/wordcount-example/). Feel free to play with other examples in the project or write your own. Please don't hesitate to [reach out](https://samza.apache.org/community/contact-us.html) if you encounter any issues.
\ No newline at end of file