You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by bh...@apache.org on 2020/06/03 21:36:11 UTC
[samza] branch master updated: SAMZA-2442: Update doc/tutorials
from config factory to config loader
This is an automated email from the ASF dual-hosted git repository.
bharathkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/samza.git
The following commit(s) were added to refs/heads/master by this push:
new fbe8349 SAMZA-2442: Update doc/tutorials from config factory to config loader
fbe8349 is described below
commit fbe8349714d5ba58b169384d11c8f856613d8919
Author: Ke Wu <kw...@linkedin.com>
AuthorDate: Wed Jun 3 14:36:00 2020 -0700
SAMZA-2442: Update doc/tutorials from config factory to config loader
Update documentations and left over scripts from config factory to config loader.
Tests:
1. Test samza rest following instructions on https://samza.apache.org/learn/documentation/latest/rest/overview.html
2.
API Changes:
None
Upgrade Instructions:
None
Usage Instructions:
None
Author: Ke Wu <kw...@linkedin.com>
Reviewers: Cameron Lee <ca...@linkedin.com>
Closes #1332 from kw2542/SAMZA-2442
---
bin/setup-int-test.sh | 2 +-
.../versioned/deployment/standalone.md | 2 +-
.../documentation/versioned/deployment/yarn.md | 2 +-
.../documentation/versioned/jobs/job-runner.md | 60 ----------------------
.../learn/documentation/versioned/rest/overview.md | 4 +-
.../tutorials/versioned/deploy-samza-to-CDH.md | 2 +-
.../versioned/hello-samza-high-level-yarn.md | 4 +-
.../tutorials/versioned/remote-debugging-samza.md | 2 +-
.../versioned/run-hello-samza-without-internet.md | 4 +-
.../tutorials/versioned/run-in-multi-node-yarn.md | 2 +-
.../versioned/samza-rest-getting-started.md | 6 +--
docs/learn/tutorials/versioned/samza-sql.md | 4 +-
docs/startup/hello-samza/versioned/index.md | 12 ++---
docs/startup/quick-start/versioned/samza-sql.md | 2 +-
docs/startup/quick-start/versioned/samza.md | 2 +-
.../stream/CoordinatorStreamWriter.java | 4 +-
samza-rest/src/main/bash/run-samza-rest-service.sh | 2 +-
.../org/apache/samza/rest/SamzaRestService.java | 3 +-
.../samza/rest/proxy/job/SimpleYarnJobProxy.java | 7 ++-
samza-test/src/main/python/configs/tests.json | 2 +-
samza-test/src/main/python/deployment.py | 2 +-
.../src/main/python/samza_job_yarn_deployer.py | 24 ++++-----
.../samza/validation/YarnJobValidationTool.java | 2 +-
23 files changed, 47 insertions(+), 109 deletions(-)
diff --git a/bin/setup-int-test.sh b/bin/setup-int-test.sh
index 130be5d..d0fa832 100755
--- a/bin/setup-int-test.sh
+++ b/bin/setup-int-test.sh
@@ -43,7 +43,7 @@ $KAFKA_DIR/bin/kafka-topics.sh --zookeeper localhost:2181 --partitions 1 --repli
# Start the jobs
for job in checker joiner emitter watcher
do
- $SAMZA_DIR/bin/run-job.sh --config job.config.loader.factory=org.apache.samza.config.loaders.PropertiesConfigLoaderFactory --config job.config.loader.properties.path=$SAMZA_DIR/config/join/$job.samza --config job.foo=$job
+ $SAMZA_DIR/bin/run-app.sh --config-path=$SAMZA_DIR/config/join/$job.samza --config job.foo=$job
done
diff --git a/docs/learn/documentation/versioned/deployment/standalone.md b/docs/learn/documentation/versioned/deployment/standalone.md
index 6b2d168..115e3f3 100644
--- a/docs/learn/documentation/versioned/deployment/standalone.md
+++ b/docs/learn/documentation/versioned/deployment/standalone.md
@@ -72,7 +72,7 @@ tar -xvf ./target/hello-samza-1.1.0-dist.tar.gz -C deploy/samza
We are ready to run the example application [WikipediaZkLocalApplication](https://github.com/apache/samza-hello-samza/blob/master/src/main/java/samza/examples/wikipedia/application/WikipediaZkLocalApplication.java). This application reads messages from the wikipedia-edits topic, and calculates counts, every ten seconds, for all edits that were made during that window. It emits these results to another topic named `wikipedia-stats`.
```bash
-./deploy/samza/bin/run-class.sh samza.examples.wikipedia.application.WikipediaZkLocalApplication --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-application-local-runner.properties
+./deploy/samza/bin/run-class.sh samza.examples.wikipedia.application.WikipediaZkLocalApplication --config job.config.loader.factory=org.apache.samza.config.loaders.PropertiesConfigLoaderFactory --config job.config.loader.properties.path=$PWD/deploy/samza/config/wikipedia-application-local-runner.properties
```
You can run the above command again to spin up a new instance of your application.
diff --git a/docs/learn/documentation/versioned/deployment/yarn.md b/docs/learn/documentation/versioned/deployment/yarn.md
index 3a46cea..36f88c0 100644
--- a/docs/learn/documentation/versioned/deployment/yarn.md
+++ b/docs/learn/documentation/versioned/deployment/yarn.md
@@ -97,7 +97,7 @@ samza-job-name-folder
Once the archive is built, the `run-app.sh` script can be used to submit the application to YARN's resource manager. The script takes 2 CLI parameters - the config factory and the config file for the application. As an example, lets run our [FilterExample](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/FilterExample.java) on YARN as follows:
```bash
-$ ./deploy/samza/bin/run-app.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path ./deploy/samza/config/filter-example.properties
+$ ./deploy/samza/bin/run-app.sh --config-path=./deploy/samza/config/filter-example.properties
```
Congratulations, you've successfully submitted your first job to YARN! You can view the YARN Web UI to view its status.
diff --git a/docs/learn/documentation/versioned/jobs/job-runner.md b/docs/learn/documentation/versioned/jobs/job-runner.md
deleted file mode 100644
index 5f0607f..0000000
--- a/docs/learn/documentation/versioned/jobs/job-runner.md
+++ /dev/null
@@ -1,60 +0,0 @@
----
-layout: page
-title: JobRunner
----
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-Samza jobs are started using a script called run-job.sh.
-
-{% highlight bash %}
-samza-example/target/bin/run-job.sh \
- --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory \
- --config-path=file://$PWD/config/hello-world.properties
-{% endhighlight %}
-
-You provide two parameters to the run-job.sh script. One is the config location, and the other is a factory class that is used to read your configuration file. The run-job.sh script is actually executing a Samza class called JobRunner. The JobRunner uses your ConfigFactory to get a Config object from the config path.
-
-{% highlight java %}
-public interface ConfigFactory {
- Config getConfig(URI configUri);
-}
-{% endhighlight %}
-
-The Config object is just a wrapper around Map<String, String>, with some nice helper methods. Out of the box, Samza ships with the PropertiesConfigFactory, but developers can implement any kind of ConfigFactory they wish.
-
-Once the JobRunner gets your configuration, it gives your configuration to the StreamJobFactory class defined by the "job.factory" property. Samza ships with three job factory implementations: ThreadJobFactory, ProcessJobFactory and YarnJobFactory. The StreamJobFactory's responsibility is to give the JobRunner a job that it can run.
-
-{% highlight java %}
-public interface StreamJob {
- StreamJob submit();
-
- StreamJob kill();
-
- ApplicationStatus waitForFinish(long timeoutMs);
-
- ApplicationStatus waitForStatus(ApplicationStatus status, long timeoutMs);
-
- ApplicationStatus getStatus();
-}
-{% endhighlight %}
-
-Once the JobRunner gets a job, it calls submit() on the job. This method is what tells the StreamJob implementation to start the SamzaContainer. In the case of LocalJobRunner, it uses a run-container.sh script to execute the SamzaContainer in a separate process, which will start one SamzaContainer locally on the machine that you ran run-job.sh on.
-
-This flow differs slightly when you use YARN, but we'll get to that later.
-
-## [Configuration »](configuration.html)
diff --git a/docs/learn/documentation/versioned/rest/overview.md b/docs/learn/documentation/versioned/rest/overview.md
index c382f03..2872bd0 100644
--- a/docs/learn/documentation/versioned/rest/overview.md
+++ b/docs/learn/documentation/versioned/rest/overview.md
@@ -37,8 +37,8 @@ Deploying the service is very similar to running a Samza job. First build the ta
Then from the extracted location, run the service using:
{% highlight bash %}
samza-example/target/bin/run-samza-rest-service.sh \
- --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory \
- --config-path=file://$PWD/config/samza-rest.properties
+ --config job.config.loader.factory=org.apache.samza.config.loaders.PropertiesConfigLoaderFactory \
+ --config job.config.loader.properties.path=$PWD/config/samza-rest.properties
{% endhighlight %}
The two config parameters have the same purpose as they do for [run-job.sh](../jobs/job-runner.html).
diff --git a/docs/learn/tutorials/versioned/deploy-samza-to-CDH.md b/docs/learn/tutorials/versioned/deploy-samza-to-CDH.md
index e3b920b..f215a6a 100644
--- a/docs/learn/tutorials/versioned/deploy-samza-to-CDH.md
+++ b/docs/learn/tutorials/versioned/deploy-samza-to-CDH.md
@@ -66,5 +66,5 @@ export HADOOP_CONF_DIR=/etc/hadoop/conf
### Run Samza Job
{% highlight bash %}
-bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/config/wikipedia-parser.properties
+bin/run-app.sh --config-path=$PWD/config/wikipedia-parser.properties
{% endhighlight %}
diff --git a/docs/learn/tutorials/versioned/hello-samza-high-level-yarn.md b/docs/learn/tutorials/versioned/hello-samza-high-level-yarn.md
index 69abf68..b0fc9fe 100644
--- a/docs/learn/tutorials/versioned/hello-samza-high-level-yarn.md
+++ b/docs/learn/tutorials/versioned/hello-samza-high-level-yarn.md
@@ -71,7 +71,7 @@ tar -xvf ./target/hello-samza-1.4.0-SNAPSHOT-dist.tar.gz -C deploy/samza
After you've built your Samza package, you can start the app on the grid using the run-app.sh script.
{% highlight bash %}
-./deploy/samza/bin/run-app.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-application.properties
+./deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/wikipedia-application.properties
{% endhighlight %}
The app will do all of the following:
@@ -115,7 +115,7 @@ This plan will make more sense after the [code walkthrough](hello-samza-high-lev
To shutdown the app, use the same _run-app.sh_ script with an extra _--operation=kill_ argument
{% highlight bash %}
-./deploy/samza/bin/run-app.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-application.properties --operation=kill
+./deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/wikipedia-application.properties --operation=kill
{% endhighlight %}
After you're done, you can clean everything up using the same grid script.
diff --git a/docs/learn/tutorials/versioned/remote-debugging-samza.md b/docs/learn/tutorials/versioned/remote-debugging-samza.md
index e0c7445..88c85fb 100644
--- a/docs/learn/tutorials/versioned/remote-debugging-samza.md
+++ b/docs/learn/tutorials/versioned/remote-debugging-samza.md
@@ -84,7 +84,7 @@ Once the grid starts, you can start the wikipedia-feed Samza job.
mvn clean package
mkdir -p deploy/samza
tar -xvf ./target/hello-samza-1.1.0-dist.tar.gz -C deploy/samza
-deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties
+deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/wikipedia-feed.properties
{% endhighlight %}
When the wikipedia-feed job starts up, a single Samza container will be created to process all incoming messages. This is the container that we'll want to connect to from the remote debugger.
diff --git a/docs/learn/tutorials/versioned/run-hello-samza-without-internet.md b/docs/learn/tutorials/versioned/run-hello-samza-without-internet.md
index e276cdb..487bd42 100644
--- a/docs/learn/tutorials/versioned/run-hello-samza-without-internet.md
+++ b/docs/learn/tutorials/versioned/run-hello-samza-without-internet.md
@@ -48,7 +48,7 @@ Otherwise, you may have the connection problem.
We provide an alternative to get wikipedia feed data. Instead of running
{% highlight bash %}
-deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties
+deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/wikipedia-feed.properties
{% endhighlight %}
You will run
@@ -70,7 +70,7 @@ Now you can go back to Generate Wikipedia Statistics section in [Hello Samza](..
The goal of
{% highlight bash %}
-deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties
+deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/wikipedia-feed.properties
{% endhighlight %}
is to deploy a Samza job which listens to wikipedia API, receives the feed in realtime and produces the feed to the Kafka topic wikipedia-raw. The alternative in this tutorial is reading local wikipedia feed in an infinite loop and producing the data to Kafka wikipedia-raw. The follow-up job, wikipedia-parser is getting data from Kafka topic wikipedia-raw, so as long as we have correct data in Kafka topic wikipedia-raw, we are fine. All Samza jobs are connected by the Kafka and do not de [...]
diff --git a/docs/learn/tutorials/versioned/run-in-multi-node-yarn.md b/docs/learn/tutorials/versioned/run-in-multi-node-yarn.md
index 837f6ed..e4745ce 100644
--- a/docs/learn/tutorials/versioned/run-in-multi-node-yarn.md
+++ b/docs/learn/tutorials/versioned/run-in-multi-node-yarn.md
@@ -167,7 +167,7 @@ cd /tmp/hello-samza && python -m SimpleHTTPServer
Go back to the original terminal (not the one running the HTTP server):
{% highlight bash %}
-deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties
+deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/wikipedia-feed.properties
{% endhighlight %}
Go to http://yourHostname:8088 and find the wikipedia-feed job. Click on the ApplicationMaster link to see that it's running.
diff --git a/docs/learn/tutorials/versioned/samza-rest-getting-started.md b/docs/learn/tutorials/versioned/samza-rest-getting-started.md
index bd8137a..d1a8b4b 100644
--- a/docs/learn/tutorials/versioned/samza-rest-getting-started.md
+++ b/docs/learn/tutorials/versioned/samza-rest-getting-started.md
@@ -48,7 +48,7 @@ Run the following commands:
{% highlight bash %}
cd samza-rest/build/distributions/
mkdir -p deploy/samza-rest
-tar -xvf ./samza-rest-1.4.0-SNAPSHOT.tgz -C deploy/samza-rest
+tar -xvf ./samza-rest_2.11-1.4.0-SNAPSHOT.tgz -C deploy/samza-rest
{% endhighlight %}
#### Configure the Installations Path
@@ -71,8 +71,8 @@ To deploy the service, run the run-samza-rest-service.sh script from the extract
{% highlight bash %}
cd deploy/samza-rest
./bin/run-samza-rest-service.sh \
- --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory \
- --config-path=file://$PWD/config/samza-rest.properties
+ --config job.config.loader.factory=org.apache.samza.config.loaders.PropertiesConfigLoaderFactory \
+ --config job.config.loader.properties.path=$PWD/config/samza-rest.properties
{% endhighlight %}
You provide two parameters to the run-samza-rest-service.sh script. One is the config location, and the other, optional, parameter is a factory class that is used to read your configuration file. The SamzaRestService uses your ConfigFactory to get a Config object from the config path. The ConfigFactory is covered in more detail on the [Job Runner page](../../documentation/{{site.version}}/jobs/job-runner.html). The run-samza-rest-service.sh script will block until the SamzaRestService te [...]
diff --git a/docs/learn/tutorials/versioned/samza-sql.md b/docs/learn/tutorials/versioned/samza-sql.md
index f905d4a..98a86b4 100644
--- a/docs/learn/tutorials/versioned/samza-sql.md
+++ b/docs/learn/tutorials/versioned/samza-sql.md
@@ -88,7 +88,7 @@ Before you can run a Samza application, you need to build a package for it. Plea
After you've built your Samza package, you can start the app on the grid using the run-app.sh script.
```bash
-./deploy/samza/bin/run-app.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/page-view-filter-sql.properties
+./deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/page-view-filter-sql.properties
```
The app executes the following SQL command :
@@ -115,7 +115,7 @@ Congratulations! You've now setup a local grid that includes YARN, Kafka, and Zo
To shutdown the app, use the same _run-app.sh_ script with an extra _--operation=kill_ argument
```bash
-./deploy/samza/bin/run-app.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/page-view-filter-sql.properties --operation=kill
+./deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/page-view-filter-sql.properties --operation=kill
```
Please follow the instructions from [Hello Samza High Level API - YARN Deployment](hello-samza-high-level-yarn.html) on how to shutdown and cleanup the app.
diff --git a/docs/startup/hello-samza/versioned/index.md b/docs/startup/hello-samza/versioned/index.md
index 3573e26..4b986ca 100644
--- a/docs/startup/hello-samza/versioned/index.md
+++ b/docs/startup/hello-samza/versioned/index.md
@@ -52,7 +52,7 @@ Before you can run a Samza job, you need to build a package for it. This package
**(Optional)** NOTE: if you want the hello-samza jobs to run with a local Samza build (e.g., if you are a Samza developer),
make sure that you run the following steps, otherwise skip them.
{% highlight bash %}
-In your hello-world project,
+In your hello-samza project,
git checkout latest
In your local Samza project,
./gradlew publishToMavenLocal
@@ -63,7 +63,7 @@ Then, you can continue w/ the following command in hello-samza project:
{% highlight bash %}
mvn clean package
mkdir -p deploy/samza
-tar -xvf ./target/hello-samza-1.4.0-SNAPSHOT-dist.tar.gz -C deploy/samza
+tar -xvf ./target/hello-samza-1.5.0-SNAPSHOT-dist.tar.gz -C deploy/samza
{% endhighlight %}
### Run a Samza Job
@@ -71,7 +71,7 @@ tar -xvf ./target/hello-samza-1.4.0-SNAPSHOT-dist.tar.gz -C deploy/samza
After you've built your Samza package, you can start a job on the grid using the run-app.sh script.
{% highlight bash %}
-deploy/samza/bin/run-app.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties
+deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/wikipedia-feed.properties
{% endhighlight %}
The job will consume a feed of real-time edits from Wikipedia, and produce them to a Kafka topic called "wikipedia-raw". Give the job a minute to startup, and then tail the Kafka topic:
@@ -89,8 +89,8 @@ If you can not see any output from Kafka consumer, you may have connection probl
Let's calculate some statistics based on the messages in the wikipedia-raw topic. Start two more jobs:
{% highlight bash %}
-deploy/samza/bin/run-app.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-parser.properties
-deploy/samza/bin/run-app.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-stats.properties
+deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/wikipedia-parser.properties
+deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/wikipedia-stats.properties
{% endhighlight %}
The first job (wikipedia-parser) parses the messages in wikipedia-raw, and extracts information about the size of the edit, who made the change, etc. You can take a look at its output with:
@@ -120,7 +120,7 @@ If you check the YARN UI, again, you'll see that all three jobs are now listed.
To shutdown one of the jobs, use the same script with an extra '--operation=kill' argument
{% highlight bash %}
-deploy/samza/bin/run-app.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties --operation=kill
+deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/wikipedia-feed.properties --operation=kill
{% endhighlight %}
After you're done, you can clean everything up using the same grid script.
diff --git a/docs/startup/quick-start/versioned/samza-sql.md b/docs/startup/quick-start/versioned/samza-sql.md
index b3bfdbd..4517698 100644
--- a/docs/startup/quick-start/versioned/samza-sql.md
+++ b/docs/startup/quick-start/versioned/samza-sql.md
@@ -76,7 +76,7 @@ configuration file and submit it to a YARN cluster.
```bash
-./deploy/samza/bin/run-app.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/page-view-filter-sql.properties
+./deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/page-view-filter-sql.properties
```
diff --git a/docs/startup/quick-start/versioned/samza.md b/docs/startup/quick-start/versioned/samza.md
index 4c3c32a..394f238 100644
--- a/docs/startup/quick-start/versioned/samza.md
+++ b/docs/startup/quick-start/versioned/samza.md
@@ -211,7 +211,7 @@ Let’s kick off our application and use gradle to run it. Alternately, you can
{% highlight bash %}
> export BASE_DIR=`pwd`
-> ./gradlew run --args="--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$BASE_DIR/src/main/config/word-count.properties"
+> ./gradlew run --args="--config job.config.loader.factory=org.apache.samza.config.loaders.PropertiesConfigLoaderFactory --config job.config.loader.properties.path=$BASE_DIR/src/main/config/word-count.properties"
{% endhighlight %}
diff --git a/samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamWriter.java b/samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamWriter.java
index 3d837ac..da65981 100644
--- a/samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamWriter.java
+++ b/samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamWriter.java
@@ -103,11 +103,9 @@ public class CoordinatorStreamWriter {
* Main function for using the CoordinatorStreamWriter. The main function starts a CoordinatorStreamWriter
* and sends control messages.
* To run the code use the following command:
- * {path to samza deployment}/samza/bin/run-coordinator-stream-writer.sh --config job.config.loader.factory={config--loader-factory} --config job.config.loader.properties{properties needed for config loader to load config} --type={type of the message} --key={[optional] key of the message} --value={[optional] value of the message}
+ * {path to samza deployment}/samza/bin/run-coordinator-stream-writer.sh --config job.config.loader.factory={config-loader-factory} --config job.config.loader.properties{properties needed for config loader to load config} --type={type of the message} --key={[optional] key of the message} --value={[optional] value of the message}
*
* @param args input arguments for running the writer. These arguments are:
- * "config-factory" = The config file factory
- * "config-path" = The path to config file of a job
* "type" = type of the message being written
* "key" = [optional] key of the message being written
* "value" = [optional] value of the message being written
diff --git a/samza-rest/src/main/bash/run-samza-rest-service.sh b/samza-rest/src/main/bash/run-samza-rest-service.sh
index bd52afd..a48daf3 100755
--- a/samza-rest/src/main/bash/run-samza-rest-service.sh
+++ b/samza-rest/src/main/bash/run-samza-rest-service.sh
@@ -16,7 +16,7 @@
# specific language governing permissions and limitations
# under the License.
-[[ $JAVA_OPTS != *-Dlog4j.configuration* ]] && export JAVA_OPTS="$JAVA_OPTS -Dlog4j.configuration=file:$(dirname $0)/log4j.xml"
+[[ $JAVA_OPTS != *-Dlog4j.configuration* ]] && export JAVA_OPTS="$JAVA_OPTS -Dlog4j.configuration=file:$(dirname $0)/log4j2.xml"
[[ -z "$SAMZA_LOG_DIR" ]] && export SAMZA_LOG_DIR="$PWD/logs"
exec $(dirname $0)/run-class.sh org.apache.samza.rest.SamzaRestService "$@"
diff --git a/samza-rest/src/main/java/org/apache/samza/rest/SamzaRestService.java b/samza-rest/src/main/java/org/apache/samza/rest/SamzaRestService.java
index a2db84b..b5a75d7 100644
--- a/samza-rest/src/main/java/org/apache/samza/rest/SamzaRestService.java
+++ b/samza-rest/src/main/java/org/apache/samza/rest/SamzaRestService.java
@@ -82,7 +82,8 @@ public class SamzaRestService {
* Command line interface to run the server.
*
* @param args arguments supported by {@link org.apache.samza.util.CommandLine}.
- * In particular, --config-path and --config-factory are used to read the Samza REST config file.
+ * In particular, --config job.config.loader.properties.path and
+ * --config job.config.loader.factory are used to read the Samza REST config file.
* @throws Exception if the server could not be successfully started.
*/
public static void main(String[] args)
diff --git a/samza-rest/src/main/java/org/apache/samza/rest/proxy/job/SimpleYarnJobProxy.java b/samza-rest/src/main/java/org/apache/samza/rest/proxy/job/SimpleYarnJobProxy.java
index 310c3e6..e982d0f 100644
--- a/samza-rest/src/main/java/org/apache/samza/rest/proxy/job/SimpleYarnJobProxy.java
+++ b/samza-rest/src/main/java/org/apache/samza/rest/proxy/job/SimpleYarnJobProxy.java
@@ -37,11 +37,10 @@ import org.slf4j.LoggerFactory;
public class SimpleYarnJobProxy extends ScriptJobProxy {
private static final Logger log = LoggerFactory.getLogger(SimpleYarnJobProxy.class);
- private static final String START_SCRIPT_NAME = "run-job.sh";
+ private static final String START_SCRIPT_NAME = "run-app.sh";
private static final String STOP_SCRIPT_NAME = "kill-yarn-job-by-name.sh";
- private static final String CONFIG_FACTORY_PARAM = "--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory";
- private static final String CONFIG_PATH_PARAM_FORMAT = "--config-path=file://%s";
+ private static final String CONFIG_PATH_PARAM_FORMAT = "--config-path=%s";
private final JobStatusProvider statusProvider;
@@ -64,7 +63,7 @@ public class SimpleYarnJobProxy extends ScriptJobProxy {
}
String scriptPath = getScriptPath(jobInstance, START_SCRIPT_NAME);
- int resultCode = scriptRunner.runScript(scriptPath, CONFIG_FACTORY_PARAM,
+ int resultCode = scriptRunner.runScript(scriptPath,
generateConfigPathParameter(jobInstance));
if (resultCode != 0) {
throw new SamzaException("Failed to start job. Result code: " + resultCode);
diff --git a/samza-test/src/main/python/configs/tests.json b/samza-test/src/main/python/configs/tests.json
index 7f3b6f0..1e54929 100644
--- a/samza-test/src/main/python/configs/tests.json
+++ b/samza-test/src/main/python/configs/tests.json
@@ -1,5 +1,5 @@
{
"samza_executable": "samza-test_2.11-1.5.0-SNAPSHOT.tgz",
"samza_install_path": "deploy/smoke_tests",
- "samza_config_factory": "org.apache.samza.config.factories.PropertiesConfigFactory"
+ "samza_config_loader_factory": "org.apache.samza.config.loaders.PropertiesConfigLoaderFactory"
}
diff --git a/samza-test/src/main/python/deployment.py b/samza-test/src/main/python/deployment.py
index 7cd3cac..0b72cc6 100644
--- a/samza-test/src/main/python/deployment.py
+++ b/samza-test/src/main/python/deployment.py
@@ -76,7 +76,7 @@ def setup_suite():
# Setup Samza job deployer.
samza_job_deployer = SamzaJobYarnDeployer({
- 'config_factory': c('samza_config_factory'),
+ 'config_loader_factory': c('samza_config_loader_factory'),
'yarn_site_template': c('yarn_site_template'),
'yarn_driver_configs': c('yarn_driver_configs'),
'yarn_nm_hosts': c('yarn_nm_hosts').values(),
diff --git a/samza-test/src/main/python/samza_job_yarn_deployer.py b/samza-test/src/main/python/samza_job_yarn_deployer.py
index 405678f..8a785ac 100644
--- a/samza-test/src/main/python/samza_job_yarn_deployer.py
+++ b/samza-test/src/main/python/samza_job_yarn_deployer.py
@@ -35,11 +35,11 @@ logger = logging.getLogger(__name__)
class SamzaJobYarnDeployer(Deployer):
def __init__(self, configs={}):
"""
- Instantiates a Samza job deployer that uses run-job.sh and kill-yarn-job.sh
+ Instantiates a Samza job deployer that uses run-job.sh and kill-yarn-job.sh
to start and stop Samza jobs in a YARN grid.
param: configs -- Map of config key/values pairs. These configs will be used
- as a default whenever overrides are not provided in the methods (install,
+ as a default whenever overrides are not provided in the methods (install,
start, stop, etc) below.
"""
logging.getLogger("paramiko").setLevel(logging.ERROR)
@@ -52,7 +52,7 @@ class SamzaJobYarnDeployer(Deployer):
def install(self, package_id, configs={}):
"""
- Installs a package (tarball, or zip) on to a list of remote hosts by
+ Installs a package (tarball, or zip) on to a list of remote hosts by
SFTP'ing the package to the remote install_path.
param: package_id -- A unique ID used to identify an installed YARN package.
@@ -104,29 +104,29 @@ class SamzaJobYarnDeployer(Deployer):
with a package_id, and a config file.
param: configs -- Map of config key/values pairs. Valid keys include:
- package_id: The package_id for the package that contains the code for job_id.
- Usually, the package_id refers to the .tgz job tarball that contains the
+ package_id: The package_id for the package that contains the code for job_id.
+ Usually, the package_id refers to the .tgz job tarball that contains the
code necessary to run job_id.
- config_factory: The config factory to use to decode the config_file.
+ config_loader_factory: The config loader factory to use to decode the config_file.
config_file: Path to the config file for the job to be run.
install_path: Path where the package for the job has been installed on remote NMs.
- properties: (optional) [(property-name,property-value)] Optional override
- properties for the run-job.sh script. These properties override the
+ properties: (optional) [(property-name,property-value)] Optional override
+ properties for the run-job.sh script. These properties override the
config_file's properties.
"""
configs = self._get_merged_configs(configs)
- self._validate_configs(configs, ['package_id', 'config_factory', 'config_file', 'install_path'])
+ self._validate_configs(configs, ['package_id', 'config_loader_factory', 'config_file', 'install_path'])
# Get configs.
package_id = configs.get('package_id')
- config_factory = configs.get('config_factory')
+ config_loader_factory = configs.get('config_loader_factory')
config_file = configs.get('config_file')
install_path = configs.get('install_path')
properties = configs.get('properties', {})
properties['yarn.package.path'] = 'file:' + os.path.join(install_path, self._get_package_tgz_name(package_id))
# Execute bin/run-job.sh locally from driver machine.
- command = "{0} --config-factory={1} --config-path={2}".format(os.path.join(package_id, "bin/run-job.sh"), config_factory, os.path.join(package_id, config_file))
+ command = "{0} --config job.config.loader.factory={1} --config job.config.loader.properties.path={2}".format(os.path.join(package_id, "bin/run-job.sh"), config_loader_factory, os.path.join(package_id, config_file))
env = self._get_env_vars(package_id)
for property_name, property_value in properties.iteritems():
command += " --config {0}={1}".format(property_name, property_value)
@@ -176,7 +176,7 @@ class SamzaJobYarnDeployer(Deployer):
def await(self, job_id, configs={}):
"""
- Waits for a Samza job to finish using bin/stat-yarn-job.sh. A job is
+ Waits for a Samza job to finish using bin/stat-yarn-job.sh. A job is
finished when its "Final State" is not "UNDEFINED".
param: job_id -- A unique ID used to idenitfy a Samza job.
diff --git a/samza-yarn/src/main/java/org/apache/samza/validation/YarnJobValidationTool.java b/samza-yarn/src/main/java/org/apache/samza/validation/YarnJobValidationTool.java
index bbcc976..23f0e4d 100644
--- a/samza-yarn/src/main/java/org/apache/samza/validation/YarnJobValidationTool.java
+++ b/samza-yarn/src/main/java/org/apache/samza/validation/YarnJobValidationTool.java
@@ -61,7 +61,7 @@ import org.slf4j.LoggerFactory;
*
* When running this tool, please provide the configuration URI of job. For example:
*
- * deploy/samza/bin/validate-yarn-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties [--metrics-validator=com.foo.bar.SomeMetricsValidator]
+ * deploy/samza/bin/validate-yarn-job.sh --config job.config.loader.factory=org.apache.samza.config.loaders.PropertiesConfigLoaderFactory --config job.config.loader.properties.path=$PWD/deploy/samza/config/wikipedia-feed.properties [--metrics-validator=com.foo.bar.SomeMetricsValidator]
*
* The tool prints out the validation result in each step and throws an exception when the
* validation fails.