You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fluo.apache.org by mw...@apache.org on 2016/07/25 14:18:47 UTC
[1/2] incubator-fluo git commit: Avoid guava beta dependency and
updated docs
Repository: incubator-fluo
Updated Branches:
refs/heads/master 8316f94f5 -> 6c988a72e
Avoid guava beta dependency and updated docs
* Update oracle and worker to avoid using Guava abstract service which is beta
* Update documentation to recommend certain versions of Hadoop, Accumulo, etc
* Renamed prod-fluo-setup.md to install.md
Project: http://git-wip-us.apache.org/repos/asf/incubator-fluo/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-fluo/commit/6aa0d438
Tree: http://git-wip-us.apache.org/repos/asf/incubator-fluo/tree/6aa0d438
Diff: http://git-wip-us.apache.org/repos/asf/incubator-fluo/diff/6aa0d438
Branch: refs/heads/master
Commit: 6aa0d4389ef627ae1b8ef84d98a703f25fe4b27e
Parents: 8316f94
Author: Mike Walch <mw...@gmail.com>
Authored: Fri Jul 22 13:05:48 2016 -0400
Committer: Mike Walch <mw...@gmail.com>
Committed: Mon Jul 25 09:53:31 2016 -0400
----------------------------------------------------------------------
README.md | 8 +-
docs/install.md | 227 +++++++++++++++++++
docs/prod-fluo-setup.md | 223 ------------------
.../apache/fluo/core/oracle/FluoOracleImpl.java | 67 ++----
.../apache/fluo/core/worker/FluoWorkerImpl.java | 57 ++---
modules/distribution/src/main/lib/ahz/pom.xml | 4 +-
modules/distribution/src/main/lib/fetch.sh | 2 +-
pom.xml | 1 +
8 files changed, 274 insertions(+), 315 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6aa0d438/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 5488013..e434a9b 100644
--- a/README.md
+++ b/README.md
@@ -14,9 +14,9 @@ news and general information.
There are several ways to run Fluo (listed in order of increasing difficulty):
* [quickstart] - Starts a MiniFluo instance that is configured to run a word count application
-* [fluo-dev] - Command-line tool for running Fluo and its dependencies on a single machine
-* [Zetten] - Command-line tool that launches an AWS cluster and sets up Fluo/Accumulo on it
-* [Production] - Sets up Fluo on a cluster where Accumulo, Hadoop & Zookeeper are running
+* [fluo-dev] - Automated tool that sets up Fluo and its dependencies on a single machine
+* [Zetten] - Automated tool that launches an AWS cluster and sets up Fluo/Accumulo on it
+* [Install instructions][install] - Manually set up Fluo on a cluster where Accumulo, Hadoop & Zookeeper are running
Except for [quickstart], all above will set up a Fluo application that will be idle unless you
create client & observer code for your application. You can either [create your own
@@ -50,7 +50,7 @@ Below are helpful resources for Fluo application developers:
[phrasecount]: https://github.com/fluo-io/phrasecount
[fluo-stress]: https://github.com/fluo-io/fluo-stress
[webindex]: https://github.com/fluo-io/webindex
-[Production]: docs/prod-fluo-setup.md
+[install]: docs/install.md
[apps]: docs/applications.md
[api]: https://fluo.apache.org/apidocs/
[recipes]: https://github.com/apache/incubator-fluo-recipes
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6aa0d438/docs/install.md
----------------------------------------------------------------------
diff --git a/docs/install.md b/docs/install.md
new file mode 100644
index 0000000..9856ff9
--- /dev/null
+++ b/docs/install.md
@@ -0,0 +1,227 @@
+Fluo Install Instructions
+=========================
+
+Install instructions for running Fluo on machine or cluster where Accumulo, Hadoop,
+and Zookeeper are installed and running. If you want to avoid setting up these
+dependencies, consider using [fluo-dev] or [Zetten].
+
+Requirements
+------------
+
+Before you install Fluo, the following software must be installed and running on
+your local machine or cluster:
+
+| Software | Recommended Version | Minimum Version |
+|-------------|---------------------|-----------------|
+| [Accumulo] | 1.7.2 | 1.6.1 |
+| [Hadoop] | 2.7.2 | 2.6.0 |
+| [Zookeeper] | 3.4.8 | |
+| [Java] | JDK 8 | JDK 8 |
+
+Obtain a distribution
+---------------------
+
+Before you can install Fluo, you will need to obtain a distribution tarball. It is
+recommended that you download the [latest release][release]. You can also build
+a distribution from the master branch by following these steps which create a tarball
+in `modules/distribution/target`:
+
+ git clone https://github.com/apache/incubator-fluo.git
+ cd fluo/
+ mvn package
+
+Install Fluo
+------------
+
+After you obtain a Fluo distribution tarball, follow these steps to install Fluo.
+
+1. Choose a directory with plenty of space and untar the distribution:
+
+ tar -xvzf fluo-1.0.0-incubating-bin.tar.gz
+
+2. Copy the example configuration to the base of your configuration directory to create
+the default configuration for your Fluo install:
+
+ cp conf/examples/* conf/
+
+ The default configuration will be used as the base configuration for each new application.
+
+3. Modify [fluo.properties] for your environment. However, you should not configure any
+application settings (like observers).
+
+ NOTE - All properties that have a default are set with it. Uncomment a property if you want
+to use a value different than the default. Properties that are unset and uncommented must be
+set by the user.
+
+4. Fluo needs to build its classpath using jars from the versions of Hadoop, Accumulo, and
+Zookeeper that you are using. Choose one of the two ways below to make these jars available
+to Fluo:
+
+ * Set `HADOOP_PREFIX`, `ACCUMULO_HOME`, and `ZOOKEEPER_HOME` in your environment or configure
+ these variables in [fluo-env.sh]. Fluo will look in these locations for jars.
+ * Run `./lib/fetch.sh ahz` to download Hadoop, Accumulo, and Zookeeper jars to `lib/ahz` and
+ configure [fluo-env.sh] to look in this directory. By default, this command will download the
+ default versions set in [lib/ahz/pom.xml]. If you are not using the default versions, you can
+ override them:
+
+ ./lib/fetch.sh ahz -Daccumulo.version=1.7.2 -Dhadoop.version=2.7.2 -Dzookeeper.version=3.4.8
+
+5. Fluo needs more dependencies than what is available from Hadoop, Accumulo, and Zookeeper.
+These extra dependencies need to be downloaded to `lib/` using the command below:
+
+ ./lib/fetch.sh extra
+
+You are now ready to use the Fluo command script.
+
+Fluo command script
+-------------------
+
+The Fluo command script is located at `bin/fluo` of your Fluo installation. All Fluo
+commands are invoked by this script.
+
+Modify and add the following to your `~/.bashrc` if you want to be able to execute the
+fluo script from any directory:
+
+ export PATH=/path/to/fluo-1.0.0-incubating/bin:$PATH
+
+Source your `.bashrc` for the changes to take effect and test the script
+
+ source ~/.bashrc
+ fluo
+
+Running the script without any arguments prints a description of all commands.
+
+ ./bin/fluo
+
+Configure a Fluo application
+----------------------------
+
+You are now ready to configure a Fluo application. Use the command below to create the
+configuration necessary for a new application. Feel free to pick a different name (other
+than `myapp`) for your application:
+
+ fluo new myapp
+
+This command will create a directory for your application at `apps/myapp` of your Fluo
+install which will contain a `conf` and `lib`.
+
+The `apps/myapp/conf` directory contains a copy of the `fluo.properties` from your default
+configuration. This should be configured for your application:
+
+ vim apps/myapp/fluo.properties
+
+When configuring the observer section in fluo.properties, you can configure your instance
+for the [phrasecount] application if you have not created your own application. See
+the [phrasecount] example for instructions. You can also choose not to configure any
+observers but your workers will be idle when started.
+
+The `apps/myapp/lib` directory should contain any observer jars for your application. If
+you configured [fluo.properties] for observers, copy any jars containing these
+observer classes to this directory.
+
+Initialize your application
+---------------------------
+
+After your application has been configured, use the command below to initialize it:
+
+ fluo init myapp
+
+This only needs to be called once and stores configuration in Zookeeper.
+
+Start your application
+----------------------
+
+A Fluo application consists of one oracle process and multiple worker processes.
+Before starting your application, you can configure the number of worker process
+in your [fluo.properties] file.
+
+When you are ready to start your Fluo application on your YARN cluster, run the
+command below:
+
+ fluo start myapp
+
+The start command above will work for a single-node or a large cluster. By
+using YARN, you do not need to deploy the Fluo binaries to every node on your
+cluster or start processes on every node.
+
+You can use the following command to check the status of your instance:
+
+ fluo status myapp
+
+For more detailed information on the YARN containers running Fluo:
+
+ fluo info myapp
+
+You can also use `yarn application -list` to check the status of your Fluo instance
+in YARN. Logs are viewable within YARN.
+
+When you have data in your fluo instance, you can view it using the command `fluo scan`.
+Pipe the output to `less` using the command `fluo scan | less` if you want to page
+through the data.
+
+Stop your Fluo application
+--------------------------
+
+Use the following command to stop your Fluo application:
+
+ fluo stop myapp
+
+If stop fails, there is also a kill command.
+
+ fluo kill myapp
+
+Tuning Accumulo
+---------------
+
+Fluo will reread the same data frequently when it checks conditions on
+mutations. When Fluo initializes a table it enables data caching to make
+this more efficient. However you may need to increase the amount of memory
+available for caching in the tserver by increasing `tserver.cache.data.size`.
+Increasing this may require increasing the maximum tserver java heap size in
+`accumulo-env.sh`.
+
+Fluo will run many client threads, will want to ensure the tablet server
+has enough threads. Should probably increase the
+`tserver.server.threads.minimum` Accumulo setting.
+
+Using at least Accumulo 1.6.1 is recommended because multiple performance bugs
+were fixed.
+
+Tuning YARN
+-----------
+
+When running Fluo oracles and workers in YARN, the number of instances, max memory, and number
+of cores for Fluo processes can be configured in [fluo.properties]. If YARN is killing processes
+consider increasing `twill.java.reserved.memory.mb` (which defaults to 200 and is set in yarn-site.xml).
+The `twill.java.reserved.memory.mb` config determines the gap between the YARN memory limit set in
+[fluo.properties] and the java -Xmx setting. For example, if max memory is 1024 and twill reserved
+memory is 200, the java -Xmx setting will be 1024-200 = 824 MB.
+
+Run locally without YARN
+------------------------
+
+If you do not have YARN set up, you can start the oracle and worker as a local
+Fluo process using the following commands:
+
+ local-fluo start-oracle
+ local-fluo start-worker
+
+Use the following commands to stop a local Fluo process:
+
+ local-fluo stop-worker
+ local-fluo stop-oracle
+
+In a distributed environment, you will need to deploy and configure a Fluo
+distribution on every node in your cluster.
+
+[fluo-dev]: https://github.com/fluo-io/fluo-dev
+[Zetten]: https://github.com/fluo-io/zetten
+[Accumulo]: https://accumulo.apache.org/
+[Hadoop]: http://hadoop.apache.org/
+[Zookeeper]: http://zookeeper.apache.org/
+[Java]: http://openjdk.java.net/
+[release]: https://fluo.apache.org/download/
+[phrasecount]: https://github.com/fluo-io/phrasecount
+[fluo.properties]: ../modules/distribution/src/main/config/fluo.properties
+[fluo-env.sh]: ../modules/distribution/src/main/config/fluo-env.sh
+[lib/ahz/pom.xml]: ../modules/distribution/src/main/lib/ahz/pom.xml
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6aa0d438/docs/prod-fluo-setup.md
----------------------------------------------------------------------
diff --git a/docs/prod-fluo-setup.md b/docs/prod-fluo-setup.md
deleted file mode 100644
index e21350b..0000000
--- a/docs/prod-fluo-setup.md
+++ /dev/null
@@ -1,223 +0,0 @@
-Production Setup
-================
-
-Below are instructions for running Fluo in a production environment where Accumulo,
-Hadoop & Zookeeper are installed and running. If you want to avoid setting up
-these dependencies, consider using [fluo-dev].
-
-Requirements
-------------
-
-Before you install Fluo, you will need the following installed and running on
-your local machine or cluster:
-
-* [Accumulo][Accumulo] (version 1.6+)
-* [Hadoop][Hadoop] (version 2.6+)
-* [Zookeeper]
-* [Java][Java] (version 8+)
-
-Obtain a distribution
----------------------
-
-Before you can install Fluo, you will need to obtain a distribution tarball. It is
-recommended that you download the [latest release][release]. You can also build
-a distribution from the master branch by following these steps which create a tarball
-in `modules/distribution/target`:
-
- git clone https://github.com/apache/incubator-fluo.git
- cd fluo/
- mvn package
-
-Install Fluo
-------------
-
-After you obtain a Fluo distribution tarball, follow these steps to install Fluo.
-
-1. Choose a directory with plenty of space and untar the distribution:
-
- tar -xvzf fluo-1.0.0-incubating-bin.tar.gz
-
-2. Copy the example configuration to the base of your configuration directory to create
-the default configuration for your Fluo install:
-
- cp conf/examples/* conf/
-
- The default configuration will be used as the base configuration for each new application.
-
-3. Modify [fluo.properties] for your environment. However, you should not configure any
-application settings (like observers).
-
- NOTE - All properties that have a default are set with it. Uncomment a property if you want
-to use a value different than the default. Properties that are unset and uncommented must be
-set by the user.
-
-4. Fluo needs build its classpath using jars from the version of Hadoop, Accumulo, and Zookeeper
-that you are using. Choose one of the two ways below to make these jars available to Fluo:
-
- * Set `HADOOP_PREFIX`, `ACCUMULO_HOME`, and `ZOOKEEPER_HOME` in your environment or configure
- these variables in [fluo-env.sh]. Fluo will look in these locations for jars.
- * Run `./lib/fetch.sh ahz` to download Hadoop, Accumulo, and Zookeeper jars to `lib/ahz` and
- configure [fluo-env.sh] to look in this directory. By default, this command will download the
- default versions set in [lib/ahz/pom.xml]. If you are not using the default versions, you can
- override them:
-
- ./lib/fetch.sh ahz -Daccumulo.version=1.7.1 -Dhadoop.version=2.6.3 -Dzookeeper.version=3.4.8
-
-5. Fluo needs more dependencies than what is available from Hadoop, Accumulo, and Zookeeper.
-These extra dependencies need to be downloaded to `lib/` using the command below:
-
- ./lib/fetch.sh extra
-
-You are now ready to use the Fluo command script.
-
-Fluo command script
--------------------
-
-The Fluo command script is located at `bin/fluo` of your Fluo installation. All Fluo
-commands are invoked by this script.
-
-Modify and add the following to your `~/.bashrc` if you want to be able to execute the
-fluo script from any directory:
-
- export PATH=/path/to/fluo-1.0.0-incubating/bin:$PATH
-
-Source your `.bashrc` for the changes to take effect and test the script
-
- source ~/.bashrc
- fluo
-
-Running the script without any arguments prints a description of all commands.
-
- ./bin/fluo
-
-Configure a Fluo application
-----------------------------
-
-You are now ready to configure a Fluo application. Use the command below to create the
-configuration necessary for a new application. Feel free to pick a different name (other
-than `myapp`) for your application:
-
- fluo new myapp
-
-This command will create a directory for your application at `apps/myapp` of your Fluo
-install which will contain a `conf` and `lib`.
-
-The `apps/myapp/conf` directory contains a copy of the `fluo.properties` from your default
-configuration. This should be configured for your application:
-
- vim apps/myapp/fluo.properties
-
-When configuring the observer section in fluo.properties, you can configure your instance
-for the [phrasecount] application if you have not created your own application. See
-the [phrasecount] example for instructions. You can also choose not to configure any
-observers but your workers will be idle when started.
-
-The `apps/myapp/lib` directory should contain any observer jars for your application. If
-you configured [fluo.properties] for observers, copy any jars containing these
-observer classes to this directory.
-
-Initialize your application
----------------------------
-
-After your application has been configured, use the command below to initialize it:
-
- fluo init myapp
-
-This only needs to be called once and stores configuration in Zookeeper.
-
-Start your application
-----------------------
-
-A Fluo application consists of one oracle process and multiple worker processes.
-Before starting your application, you can configure the number of worker process
-in your [fluo.properties] file.
-
-When you are ready to start your Fluo application on your YARN cluster, run the
-command below:
-
- fluo start myapp
-
-The start command above will work for a single-node or a large cluster. By
-using YARN, you do not need to deploy the Fluo binaries to every node on your
-cluster or start processes on every node.
-
-You can use the following command to check the status of your instance:
-
- fluo status myapp
-
-For more detailed information on the YARN containers running Fluo:
-
- fluo info myapp
-
-You can also use `yarn application -list` to check the status of your Fluo instance
-in YARN. Logs are viewable within YARN.
-
-When you have data in your fluo instance, you can view it using the command `fluo scan`.
-Pipe the output to `less` using the command `fluo scan | less` if you want to page
-through the data.
-
-Stop your Fluo application
---------------------------
-
-Use the following command to stop your Fluo application:
-
- fluo stop myapp
-
-If stop fails, there is also a kill command.
-
- fluo kill myapp
-
-Tuning Accumulo
----------------
-
-Fluo will reread the same data frequently when it checks conditions on
-mutations. When Fluo initializes a table it enables data caching to make
-this more efficient. However you may need to increase the amount of memory
-available for caching in the tserver by increasing `tserver.cache.data.size`.
-Increasing this may require increasing the maximum tserver java heap size in
-`accumulo-env.sh`.
-
-Fluo will run many client threads, will want to ensure the tablet server
-has enough threads. Should probably increase the
-`tserver.server.threads.minimum` Accumulo setting.
-
-Using at least Accumulo 1.6.1 is recommended because multiple performance bugs
-were fixed.
-
-Tuning YARN
------------
-
-When running Fluo oracles and workers in YARN, the number of instances, max memory, and number
-of cores for Fluo processes can be configured in [fluo.properties]. If YARN is killing processes
-consider increasing `twill.java.reserved.memory.mb` (which defaults to 200 and is set in yarn-site.xml).
-The `twill.java.reserved.memory.mb` config determines the gap between the YARN memory limit set in
-[fluo.properties] and the java -Xmx setting. For example, if max memory is 1024 and twill reserved
-memory is 200, the java -Xmx setting will be 1024-200 = 824 MB.
-
-Run locally without YARN
-------------------------
-
-If you do not have YARN set up, you can start the oracle and worker as a local
-Fluo process using the following commands:
-
- local-fluo start-oracle
- local-fluo start-worker
-
-Use the following commands to stop a local Fluo process:
-
- local-fluo stop-worker
- local-fluo stop-oracle
-
-In a distributed environment, you will need to deploy and configure a Fluo
-distribution on every node in your cluster.
-
-[fluo-dev]: https://github.com/fluo-io/fluo-dev
-[Accumulo]: https://accumulo.apache.org/
-[Hadoop]: http://hadoop.apache.org/
-[Zookeeper]: http://zookeeper.apache.org/
-[Java]: http://openjdk.java.net/
-[release]: https://fluo.apache.org/download/
-[phrasecount]: https://github.com/fluo-io/phrasecount
-[fluo.properties]: ../modules/distribution/src/main/config/fluo.properties
-[fluo-env.sh]: ../modules/distribution/src/main/config/fluo-env.sh
-[lib/ahz/pom.xml]: ../modules/distribution/src/main/lib/ahz/pom.xml
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6aa0d438/modules/core/src/main/java/org/apache/fluo/core/oracle/FluoOracleImpl.java
----------------------------------------------------------------------
diff --git a/modules/core/src/main/java/org/apache/fluo/core/oracle/FluoOracleImpl.java b/modules/core/src/main/java/org/apache/fluo/core/oracle/FluoOracleImpl.java
index 6851e01..c0f5f11 100644
--- a/modules/core/src/main/java/org/apache/fluo/core/oracle/FluoOracleImpl.java
+++ b/modules/core/src/main/java/org/apache/fluo/core/oracle/FluoOracleImpl.java
@@ -19,8 +19,6 @@ import java.io.File;
import java.util.Objects;
import com.google.common.base.Preconditions;
-import com.google.common.util.concurrent.AbstractIdleService;
-import com.google.common.util.concurrent.UncheckedExecutionException;
import org.apache.curator.framework.recipes.cache.NodeCache;
import org.apache.fluo.api.config.FluoConfiguration;
import org.apache.fluo.api.exceptions.FluoException;
@@ -36,55 +34,28 @@ public class FluoOracleImpl implements FluoOracle {
private static final Logger log = LoggerFactory.getLogger(FluoOracleImpl.class);
- private OracleService oracleService;
+ private FluoConfiguration config;
+ private Environment env;
+ private AutoCloseable reporters;
+ private OracleServer oracleServer;
+ private NodeCache appIdCache;
public FluoOracleImpl(FluoConfiguration config) {
- this.oracleService = new OracleService(config);
- }
-
- @Override
- public void start() {
+ Objects.requireNonNull(config);
+ Preconditions.checkArgument(config.hasRequiredOracleProps());
+ // any client in oracle should retry forever
+ config.setClientRetryTimeout(-1);
try {
- oracleService.startAndWait();
- } catch (UncheckedExecutionException e) {
- throw new FluoException(e);
+ config.validate();
+ } catch (Exception e) {
+ throw new IllegalArgumentException("Invalid FluoConfiguration", e);
}
+ this.config = config;
}
@Override
- public void stop() {
+ public void start() {
try {
- oracleService.stopAndWait();
- } catch (UncheckedExecutionException e) {
- throw new FluoException(e);
- }
- }
-
- private static class OracleService extends AbstractIdleService {
-
- private static final Logger log = LoggerFactory.getLogger(OracleService.class);
-
- private FluoConfiguration config;
- private Environment env;
- private AutoCloseable reporters;
- private OracleServer oracleServer;
- private NodeCache appIdCache;
-
- OracleService(FluoConfiguration config) {
- Objects.requireNonNull(config);
- Preconditions.checkArgument(config.hasRequiredOracleProps());
- // any client in oracle should retry forever
- config.setClientRetryTimeout(-1);
- try {
- config.validate();
- } catch (Exception e) {
- throw new IllegalArgumentException("Invalid FluoConfiguration", e);
- }
- this.config = config;
- }
-
- @Override
- protected void startUp() throws Exception {
env = new Environment(config);
reporters = ReporterUtil.setupReporters(env);
appIdCache = CuratorUtil.startAppIdWatcher(env);
@@ -95,14 +66,20 @@ public class FluoOracleImpl implements FluoOracle {
oracleServer = new OracleServer(env);
oracleServer.start();
+ } catch (Exception e) {
+ throw new FluoException(e);
}
+ }
- @Override
- protected void shutDown() throws Exception {
+ @Override
+ public void stop() {
+ try {
oracleServer.stop();
appIdCache.close();
reporters.close();
env.close();
+ } catch (Exception e) {
+ throw new FluoException(e);
}
}
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6aa0d438/modules/core/src/main/java/org/apache/fluo/core/worker/FluoWorkerImpl.java
----------------------------------------------------------------------
diff --git a/modules/core/src/main/java/org/apache/fluo/core/worker/FluoWorkerImpl.java b/modules/core/src/main/java/org/apache/fluo/core/worker/FluoWorkerImpl.java
index e5b1bf2..47c9333 100644
--- a/modules/core/src/main/java/org/apache/fluo/core/worker/FluoWorkerImpl.java
+++ b/modules/core/src/main/java/org/apache/fluo/core/worker/FluoWorkerImpl.java
@@ -19,8 +19,6 @@ import java.io.File;
import java.util.Objects;
import com.google.common.base.Preconditions;
-import com.google.common.util.concurrent.AbstractIdleService;
-import com.google.common.util.concurrent.UncheckedExecutionException;
import org.apache.curator.framework.recipes.cache.NodeCache;
import org.apache.fluo.api.config.FluoConfiguration;
import org.apache.fluo.api.exceptions.FluoException;
@@ -36,49 +34,22 @@ public class FluoWorkerImpl implements FluoWorker {
private static final Logger log = LoggerFactory.getLogger(FluoWorkerImpl.class);
- private WorkerService workerService;
+ private FluoConfiguration config;
+ private Environment env;
+ private AutoCloseable reporters;
+ private NotificationProcessor np;
+ private NotificationFinder notificationFinder;
+ private NodeCache appIdCache;
public FluoWorkerImpl(FluoConfiguration config) {
- this.workerService = new WorkerService(config);
+ Objects.requireNonNull(config);
+ Preconditions.checkArgument(config.hasRequiredWorkerProps());
+ this.config = config;
}
@Override
public void start() {
try {
- workerService.startAndWait();
- } catch (UncheckedExecutionException e) {
- throw new FluoException(e);
- }
- }
-
- @Override
- public void stop() {
- try {
- workerService.stopAndWait();
- } catch (UncheckedExecutionException e) {
- throw new FluoException(e);
- }
- }
-
- private static class WorkerService extends AbstractIdleService {
-
- private static final Logger log = LoggerFactory.getLogger(WorkerService.class);
-
- private FluoConfiguration config;
- private Environment env;
- private AutoCloseable reporters;
- private NotificationProcessor np;
- private NotificationFinder notificationFinder;
- private NodeCache appIdCache;
-
- WorkerService(FluoConfiguration config) {
- Objects.requireNonNull(config);
- Preconditions.checkArgument(config.hasRequiredWorkerProps());
- this.config = config;
- }
-
- @Override
- protected void startUp() throws Exception {
env = new Environment(config);
reporters = ReporterUtil.setupReporters(env);
appIdCache = CuratorUtil.startAppIdWatcher(env);
@@ -91,15 +62,21 @@ public class FluoWorkerImpl implements FluoWorker {
notificationFinder = NotificationFinderFactory.newNotificationFinder(env.getConfiguration());
notificationFinder.init(env, np);
notificationFinder.start();
+ } catch (Exception e) {
+ throw new FluoException(e);
}
+ }
- @Override
- protected void shutDown() throws Exception {
+ @Override
+ public void stop() {
+ try {
notificationFinder.stop();
np.close();
appIdCache.close();
reporters.close();
env.close();
+ } catch (Exception e) {
+ throw new FluoException(e);
}
}
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6aa0d438/modules/distribution/src/main/lib/ahz/pom.xml
----------------------------------------------------------------------
diff --git a/modules/distribution/src/main/lib/ahz/pom.xml b/modules/distribution/src/main/lib/ahz/pom.xml
index d08c597..ef8a6bd 100644
--- a/modules/distribution/src/main/lib/ahz/pom.xml
+++ b/modules/distribution/src/main/lib/ahz/pom.xml
@@ -20,8 +20,8 @@
<version>a</version>
<properties>
- <accumulo.version>1.7.1</accumulo.version>
- <hadoop.version>2.6.3</hadoop.version>
+ <accumulo.version>1.7.2</accumulo.version>
+ <hadoop.version>2.7.2</hadoop.version>
<zookeeper.version>3.4.8</zookeeper.version>
</properties>
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6aa0d438/modules/distribution/src/main/lib/fetch.sh
----------------------------------------------------------------------
diff --git a/modules/distribution/src/main/lib/fetch.sh b/modules/distribution/src/main/lib/fetch.sh
index 37fbff2..95765a2 100755
--- a/modules/distribution/src/main/lib/fetch.sh
+++ b/modules/distribution/src/main/lib/fetch.sh
@@ -108,6 +108,6 @@ extra)
echo -e " extra Download extra Fluo dependencies\n"
echo "For 'ahz', the versions of Hadoop, Accumulo, & Zookeeper are specifed in ahz/pom.xml."
echo -e "However, you can override them using the command below:\n"
- echo "./fetch.sh ahz -Daccumulo.version=1.7.1 -Dhadoop.version=2.6.3 -Dzookeeper.version=3.4.8"
+ echo "./fetch.sh ahz -Daccumulo.version=1.7.2 -Dhadoop.version=2.7.2 -Dzookeeper.version=3.4.8"
esac
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6aa0d438/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 2d0b100..9904e49 100644
--- a/pom.xml
+++ b/pom.xml
@@ -83,6 +83,7 @@
<version>1.32</version>
</dependency>
<dependency>
+ <!-- Guava 13.0.1 is required by Twill (due to beta method usage) -->
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>13.0.1</version>
[2/2] incubator-fluo git commit: Merge branch 'avoid-guava-beta'
Posted by mw...@apache.org.
Merge branch 'avoid-guava-beta'
Project: http://git-wip-us.apache.org/repos/asf/incubator-fluo/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-fluo/commit/6c988a72
Tree: http://git-wip-us.apache.org/repos/asf/incubator-fluo/tree/6c988a72
Diff: http://git-wip-us.apache.org/repos/asf/incubator-fluo/diff/6c988a72
Branch: refs/heads/master
Commit: 6c988a72ed45d6dc61043bcb06590fa470791298
Parents: 8316f94 6aa0d43
Author: Mike Walch <mw...@gmail.com>
Authored: Mon Jul 25 10:17:05 2016 -0400
Committer: Mike Walch <mw...@gmail.com>
Committed: Mon Jul 25 10:17:05 2016 -0400
----------------------------------------------------------------------
README.md | 8 +-
docs/install.md | 227 +++++++++++++++++++
docs/prod-fluo-setup.md | 223 ------------------
.../apache/fluo/core/oracle/FluoOracleImpl.java | 67 ++----
.../apache/fluo/core/worker/FluoWorkerImpl.java | 57 ++---
modules/distribution/src/main/lib/ahz/pom.xml | 4 +-
modules/distribution/src/main/lib/fetch.sh | 2 +-
pom.xml | 1 +
8 files changed, 274 insertions(+), 315 deletions(-)
----------------------------------------------------------------------