You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by ja...@apache.org on 2018/11/27 09:48:29 UTC
[22/50] samza git commit: Revert "clean-up the quickstart page"
Revert "clean-up the quickstart page"
This reverts commit cd34d98bf638d84ec652f006844e05d343278d7a.
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/2615ad56
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/2615ad56
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/2615ad56
Branch: refs/heads/master
Commit: 2615ad5642b3cbcf9de1f79523c531c4204e6b09
Parents: cd34d98
Author: Jagadish <jv...@linkedin.com>
Authored: Tue Oct 23 23:18:01 2018 -0700
Committer: Jagadish <jv...@linkedin.com>
Committed: Tue Oct 23 23:18:01 2018 -0700
----------------------------------------------------------------------
docs/_docs/replace-versioned.sh | 5 +--
docs/_menu/index.html | 2 +-
docs/startup/code-examples/versioned/index.md | 49 ----------------------
docs/startup/quick-start/versioned/index.md | 26 +++++-------
4 files changed, 12 insertions(+), 70 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/2615ad56/docs/_docs/replace-versioned.sh
----------------------------------------------------------------------
diff --git a/docs/_docs/replace-versioned.sh b/docs/_docs/replace-versioned.sh
index c454cac..24bf7ae 100755
--- a/docs/_docs/replace-versioned.sh
+++ b/docs/_docs/replace-versioned.sh
@@ -44,7 +44,4 @@ echo "replaced startup/hello-samza/versioned to startup/hello-samza/"$version
mv -f $DIR/_site/startup/hello-samza/versioned $DIR/_site/startup/hello-samza/$version
echo "replaced startup/quick-start/versioned to startup/quick-start/"$version
-mv -f $DIR/_site/startup/quick-start/versioned $DIR/_site/startup/quick-start/$version
-
-echo "replaced startup/code-examples/versioned to startup/code-examples/"$version
-mv -f $DIR/_site/startup/code-examples/versioned $DIR/_site/startup/code-examples/$version
+mv -f $DIR/_site/startup/quick-start/versioned $DIR/_site/startup/quick-start/$version
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/samza/blob/2615ad56/docs/_menu/index.html
----------------------------------------------------------------------
diff --git a/docs/_menu/index.html b/docs/_menu/index.html
index a363bae..0d1750f 100644
--- a/docs/_menu/index.html
+++ b/docs/_menu/index.html
@@ -5,7 +5,7 @@ items:
- menu_title: QuickStart
url: /startup/quick-start/version/
- menu_title: Code Examples
- url: /startup/code-examples/version/
+ url: /learn/tutorials/version/
- menu_title: Documentation
has_sub: true
has_sub_subs: true
http://git-wip-us.apache.org/repos/asf/samza/blob/2615ad56/docs/startup/code-examples/versioned/index.md
----------------------------------------------------------------------
diff --git a/docs/startup/code-examples/versioned/index.md b/docs/startup/code-examples/versioned/index.md
deleted file mode 100644
index ba1cc3e..0000000
--- a/docs/startup/code-examples/versioned/index.md
+++ /dev/null
@@ -1,49 +0,0 @@
----
-layout: page
-title:
----
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-
-### Checking out our examples
-
-The [hello-samza](https://github.com/apache/samza-hello-samza) project contains several examples to help you create your Samza applications. To checkout the hello-samza project:
-
-{% highlight bash %}
-> git clone https://git.apache.org/samza-hello-samza.git hello-samza
-{% endhighlight %}
-
-#### High-level API examples
-[The Samza Cookbook](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/cookbook) contains various recipes using the Samza high-level API.
-These include:
-
-- The [Filter example](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/FilterExample.java) demonstrates how to perform stateless operations on a stream.
-
-- The [Join example](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/JoinExample.java]) demonstrates how you can join a Kafka stream of page-views with a stream of ad-clicks
-
-- The [Stream-Table Join example](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/RemoteTableJoinExample.java) demonstrates how the Samza Table API. It joins a Kafka stream with a remote dataset accessed through a REST service.
-
-- The [SessionWindow](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/SessionWindowExample.java) and [TumblingWindow](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/TumblingWindowExample.java) examples illustrate Samza's rich windowing and triggering capabilities.
-
-
-In addition to the cookbook, you can also consult these:
-
-- [Wikipedia Parser](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/wikipedia): An advanced example that builds a streaming pipeline consuming a live-feed of wikipedia edits, parsing each message and generating statistics from them.
-
-
-- [Amazon Kinesis](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/kinesis) and [Azure Eventhubs](https://github.com/apache/samza-hello-samza/tree/latest/src/main/java/samza/examples/azure) examples that cover how to consume input data from the respective systems.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/samza/blob/2615ad56/docs/startup/quick-start/versioned/index.md
----------------------------------------------------------------------
diff --git a/docs/startup/quick-start/versioned/index.md b/docs/startup/quick-start/versioned/index.md
index da20db2..a046ee7 100644
--- a/docs/startup/quick-start/versioned/index.md
+++ b/docs/startup/quick-start/versioned/index.md
@@ -65,7 +65,7 @@ public class WordCount implements StreamApplication {
The StreamApplication interface provides an API method named describe() for you to specify your streaming pipeline. Using [StreamApplicationDescriptor](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/application/StreamApplicationDescriptor.html), you can describe your entire data processing task from data inputs, operations and outputs.
-### Describe your inputs and outputs
+### Input data source using Kafka
In this example, we are going to use Kafka as the input data source and consume the text for word count line by line. We start by defining a KafkaSystemDescriptor, which specifies the properties to establishing the connection to the local Kafka cluster. Then we create a `KafkaInputDescriptor`/`KafkaOutputDescriptor` to set up the topic, Serializer and Deserializer. Finally we use this input in the [StreamApplicationDescriptor](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/application/StreamApplicationDescriptor.html) so we can consume from this topic. The code is in the following:
@@ -106,7 +106,7 @@ The resulting [MessageStream](/learn/documentation/{{site.version}}/api/javadocs
First we are going to extract the value from lines. This is a one-to-one transform and we can use the Samza map operator as following:
{% highlight java %}
-lines.map(kv -> kv.value)
+lines .map(kv -> kv.value)
{% endhighlight %}
Then we will split the line into words by using the flatmap operator:
@@ -152,29 +152,23 @@ lines
In this section we will configure the word count example to run locally in a single JVM. Please add a file named “word-count.properties” under the config folder. We will add the job configs in this file.
-In this section, we will configure our word count example to run locally in a single JVM. Let us add a file named “word-count.properties” under the config folder.
+Since there is only a single Samza processor, there is no coordination required. We use the PassthroughJobCoordinator for the example. We also group all Samza tasks into this single processor. As for the Kafka topic, we will consume from the beginning. Here is the full config needed for the job:
{% highlight jproperties %}
job.name=word-count
-# Use a PassthroughJobCoordinator since there is no coordination needed
job.coordinator.factory=org.apache.samza.standalone.PassthroughJobCoordinatorFactory
job.coordination.utils.factory=org.apache.samza.standalone.PassthroughCoordinationUtilsFactory
-
job.changelog.system=kafka
-
-# Use a single container to process all of the data
task.name.grouper.factory=org.apache.samza.container.grouper.task.SingleContainerGrouperFactory
processor.id=0
-
-# Read from the beginning of the topic
systems.kafka.default.stream.samza.offset.default=oldest
{% endhighlight %}
-For more details on Samza's configs, feel free to check out the latest [configuration reference](/learn/documentation/{{site.version}}/jobs/configuration-table.html).
+For more details about Samza config, feel free to check out the latest config [here](/learn/documentation/{{site.version}}/jobs/configuration-table.html).
### Run your application
-Let’s now add a `main()` function to the `WordCount` class. The function reads the config file and factory from the args, and creates a `LocalApplicationRunner` that run the application locally.
+Let’s add a `main()` function to `WordCount` class first. The function reads the config file and factory from the args, and create a `LocalApplicationRunner` to run the application locally. Here is the function details:
{% highlight java %}
public static void main(String[] args) {
@@ -195,7 +189,7 @@ apply plugin:'application'
mainClassName = "samzaapp.WordCount"
{% endhighlight %}
-Before running `main()`, we will create our input Kafka topic and populate it with sample data. You can download the scripts to interact with Kafka along with the sample data from [here](https://github.com/apache/samza-hello-samza/blob/latest/quickstart/wordcount.tar.gz).
+Before running `main()`, we need to create the input Kafka topic with some sample data. Let’s start a local kafka broker first. Samza examples provides a script named “grid” which you can use to start zookeeper, kafka broker and yarn. Your can download it [here](https://github.com/apache/samza-hello-samza/blob/master/bin/grid) and put it under scripts/ folder, then issue the following command:
{% highlight bash %}
> ./scripts/grid install zookeeper && ./scripts/grid start zookeeper
@@ -251,10 +245,10 @@ The [hello-samza](https://github.com/apache/samza-hello-samza) project contains
There are four main categories of examples in this project, including:
-1. [Wikipedia](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/wikipedia): this is a more complex example demonstrating the entire pipeline of consuming from the live feed from wikipedia edits, parsing the message and generating statistics from them.
+1. [wikipedia](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/wikipedia): this is a more complex example demonstrating the entire pipeline of consuming from the live feed from wikipedia edits, parsing the message and generating statistics from them.
-2. [Cookbook](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/cookbook): you will find various examples in this folder to demonstrate usage of Samza high-level API, such as windowing, join and aggregations.
+2. [cookbook](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/cookbook): you will find various examples in this folder to demonstrate usage of Samza high-level API, such as windowing, join and aggregations.
-3. [Azure](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/azure): This example shows how to build an application that consumes input streams from Azure EventHubs.
+3. [asure](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/azure): this example shows how to run your application on Microsoft Asure.
-4. [Kinesis](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/kinesis): This example shows how to consume from Kinesis streams.
\ No newline at end of file
+4. [kinesis](https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/kinesis): this example shows how to consume from Kinesis streams
\ No newline at end of file