You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fluo.apache.org by kt...@apache.org on 2016/06/27 22:20:15 UTC
incubator-fluo git commit: fixes #247 Made Accumulo, Hadoop,
and ZK jars configurable in tarball
Repository: incubator-fluo
Updated Branches:
refs/heads/master 2b2d784fc -> 6fd86298c
fixes #247 Made Accumulo,Hadoop,and ZK jars configurable in tarball
Project: http://git-wip-us.apache.org/repos/asf/incubator-fluo/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-fluo/commit/6fd86298
Tree: http://git-wip-us.apache.org/repos/asf/incubator-fluo/tree/6fd86298
Diff: http://git-wip-us.apache.org/repos/asf/incubator-fluo/diff/6fd86298
Branch: refs/heads/master
Commit: 6fd86298c7d6c4759df98e9d6633fe91ec036f99
Parents: 2b2d784
Author: Keith Turner <kt...@apache.org>
Authored: Fri Jun 17 19:50:59 2016 -0400
Committer: Keith Turner <kt...@apache.org>
Committed: Mon Jun 27 18:16:37 2016 -0400
----------------------------------------------------------------------
README.md | 2 -
docs/mini-fluo-setup.md | 143 -----------------
docs/prod-fluo-setup.md | 29 ++--
.../fluo/cluster/command/FluoCommand.java | 5 +-
.../fluo/cluster/command/MiniFluoCommand.java | 123 --------------
.../apache/fluo/cluster/runner/AppRunner.java | 62 -------
.../fluo/cluster/runner/ClasspathOptions.java | 54 -------
.../fluo/cluster/runner/MiniAppRunner.java | 44 -----
.../fluo/cluster/runner/it/MiniAppRunnerIT.java | 89 -----------
modules/distribution/pom.xml | 15 +-
modules/distribution/src/main/assembly/bin.xml | 80 +++-------
.../distribution/src/main/config/fluo-env.sh | 62 +++++++
modules/distribution/src/main/deps/README.md | 14 ++
modules/distribution/src/main/deps/pom.xml | 64 ++++++++
modules/distribution/src/main/scripts/fluo | 23 +--
.../src/main/scripts/impl/config.sh | 8 +-
.../distribution/src/main/scripts/local-fluo | 10 +-
modules/distribution/src/main/scripts/mini-fluo | 160 -------------------
18 files changed, 207 insertions(+), 780 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 0e89697..5488013 100644
--- a/README.md
+++ b/README.md
@@ -14,7 +14,6 @@ news and general information.
There are several ways to run Fluo (listed in order of increasing difficulty):
* [quickstart] - Starts a MiniFluo instance that is configured to run a word count application
-* [MiniFluo] - Sets up a minimal Fluo instance that writes its data to single directory
* [fluo-dev] - Command-line tool for running Fluo and its dependencies on a single machine
* [Zetten] - Command-line tool that launches an AWS cluster and sets up Fluo/Accumulo on it
* [Production] - Sets up Fluo on a cluster where Accumulo, Hadoop & Zookeeper are running
@@ -51,7 +50,6 @@ Below are helpful resources for Fluo application developers:
[phrasecount]: https://github.com/fluo-io/phrasecount
[fluo-stress]: https://github.com/fluo-io/fluo-stress
[webindex]: https://github.com/fluo-io/webindex
-[MiniFluo]: docs/mini-fluo-setup.md
[Production]: docs/prod-fluo-setup.md
[apps]: docs/applications.md
[api]: https://fluo.apache.org/apidocs/
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/docs/mini-fluo-setup.md
----------------------------------------------------------------------
diff --git a/docs/mini-fluo-setup.md b/docs/mini-fluo-setup.md
deleted file mode 100644
index 3ee3f56..0000000
--- a/docs/mini-fluo-setup.md
+++ /dev/null
@@ -1,143 +0,0 @@
-MiniFluo Setup
-==============
-
-Below are instructions for running a test/development instance of Fluo
-called MiniFluo. MiniFluo runs its own Accumulo and Zookeeper but can
-only run on one machine. While MiniFluo is easy to set up and has
-all of the features of Fluo, it is not recommended for production use as
-all of its data is lost when its stopped.
-
-If you looking to run Fluo in a production environment, read the
-[production setup instructions][prod-fluo].
-
-Obtaining a distribution
-------------------------
-
-Before you can install MiniFluo, you will need to obtain a distribution tarball. You
-can either download the [latest release][release] or build a distribution tarball
-by following these steps:
-
-First, clone Fluo:
-
- git clone https://github.com/apache/incubator-fluo.git
- cd fluo/
-
-Optionally, check out a stable tag (if you don't want to build a release from master):
-
- git checkout 1.0.0-beta-2
-
-Next, build a distribution which will be created in `modules/distribution/target`.
-
- mvn package
-
-Install Fluo distribution
--------------------------
-
-When you have a distribution tarball built for your environment, follow these steps
-to install Fluo.
-
-First, choose a directory with plenty of space and untar the distribution:
-
- tar -xvzf fluo-1.0.0-beta-2-bin.tar.gz
-
-Verify that your distribution has the same versions of Hadoop & Accumulo as your environment:
-
- cd fluo-1.0.0-beta-2
- ls lib/hadoop-* lib/accumulo-*
-
-Next, copy the example configuration to the base of your configuration directory to create
-the default configuration for your Fluo install:
-
- cp conf/examples/* conf/
-
-The default configuration will be used as the base configuration for each new application.
-Therefore, you should modify [fluo.properties] for your environment. However, you should
-not configure any application settings (like observers).
-
-As you are running a MiniFluo instance, most properties in [fluo.properties] do not need to be
-set by you. For example, all "Client properties" can be left unset as MiniFluo will start up
-its own Accumulo and Zookeeper. If you would rather have MiniFluo connect to an existing
-Accumulo instance, change `org.apache.fluo.mini.start.accumulo` to `false` and set all "Client properties".
-If you have an existing Accumulo instance, you should also consider running a Fluo instance by
-following the [production setup instructions][prod-fluo].
-
-MiniFluo command script
------------------------
-
-The MiniFluo command script is located at `bin/mini-fluo` of your Fluo installation. All
-MiniFluo commands are invoked by this script.
-
-Modify and add the following to your `~/.bashrc` if you want to be able to execute the
-`mini-fluo` script from any directory:
-
- export PATH=/path/to/fluo-1.0.0-beta-2/bin:$PATH
-
-Source your `.bashrc` for the changes to take effect and test the script
-
- source ~/.bashrc
- mini-fluo
-
-Running the script without any arguments prints a description of all commands.
-
-Configure a MiniFluo application
---------------------------------
-
-You are now ready to configure a MiniFluo application. Use the command below to create the
-configuration necessary for a new application. Feel free to pick a different name (other
-than `myapp`) for your application:
-
- mini-fluo new myapp
-
-This command will create a directory for your application at `apps/myapp` of your Fluo
-install which will contain a `conf` and `lib`.
-
-The `apps/myapp/conf` directory contains a copy of the `fluo.properties` from your default
-configuration. This should be configured for your application:
-
- vim apps/myapp/fluo.properties
-
-When configuring the observer section in [fluo.properties], you can configure your application
-to run the [phrasecount] example if you have not developed your own application. See
-the [phrasecount] example for instructions. You can also choose not to configure any
-observers but MiniFluo will be idle when started.
-
-The `apps/myapp/lib` directory should contain any observer jars for your application. If
-you configured [fluo.properties] for observers, copy any jars containing these
-observer classes this directory.
-
-Run a MiniFluo application
---------------------------
-
-A MiniFluo application can be started with the following command:
-
- mini-fluo start myapp
-
-The application will output its logs to `app/myapp/logs/` directory of your Fluo installation.
-
-MiniFluo starts its own cluster of Accumulo and Zookeeper. All data for this
-cluster is written by default to `app/myapp/mini/` directory of your Fluo installation
-but this can be configured in `fluo.properties`. Logs for underlying cluster
-can be viewed in `mini/logs`.
-
-Due to MiniFluo starting its own cluster, it generates and writes a `client.properties`
-file to its data directory. This file can be used by Fluo clients to connect
-to the MiniFluo instance.
-
-When you have data in your fluo instance, you can view it using the command `mini-fluo scan myapp`.
-Pipe the output to `less` using the command `mini-fluo scan myapp | less` if you want to page
-through the data.
-
-Stop a MiniFluo application
----------------------------
-
-A MiniFluo application can be stopped by running the following command:
-
- mini-fluo stop myapp
-
-If MiniFluo started its own Accumulo cluster, it will also remove the `mini/` data
-directory and any data in your instance will be lost.
-
-[prod-fluo]: prod-fluo-setup.md
-[release]: https://github.com/apache/incubator-fluo/releases
-[phrasecount]: https://github.com/fluo-io/phrasecount
-[fluo.properties]: ../modules/distribution/src/main/config/fluo.properties
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/docs/prod-fluo-setup.md
----------------------------------------------------------------------
diff --git a/docs/prod-fluo-setup.md b/docs/prod-fluo-setup.md
index 0a8f3e1..dd633a0 100644
--- a/docs/prod-fluo-setup.md
+++ b/docs/prod-fluo-setup.md
@@ -3,7 +3,7 @@ Production Setup
Below are instructions for running Fluo in a production environment where Accumulo,
Hadoop & Zookeeper are installed and running. If you want to avoid setting up
-these dependencies, consider using [fluo-dev] or [MiniFluo].
+these dependencies, consider using [fluo-dev].
Requirements
------------
@@ -19,25 +19,14 @@ your local machine or cluster:
Obtain a distribution
---------------------
-Before you can install Fluo, you will need to obtain a distribution tarball that
-works for your environment. Fluo distributions are built for specific releases
-of Hadoop and Accumulo. If you are using Accumulo 1.7.1 and Hadoop 2.6.3,
-you can download the [latest release][release]. If you need a release for a
-different environment or one built from the master branch, follow these steps:
-
-First, clone Fluo:
+Before you can install Fluo, you will need to obtain a distribution tarball. It is
+recommended that you download the [latest release][release]. You can also build
+a distribution from the master branch by following these steps which create a tarball
+in `modules/distribution/target`:
git clone https://github.com/apache/incubator-fluo.git
cd fluo/
-
-Optionally, check out a stable tag (if you don't want to build a release from master):
-
- git checkout 1.0.0-beta-2
-
-Next, build a distribution for your environment. The tarball will be created in
-`modules/distribution/target`.
-
- mvn package -Daccumulo.version=1.7.1 -Dhadoop.version=2.7.1
+ mvn package
Install Fluo
------------
@@ -67,6 +56,11 @@ NOTE - All properties that have a default are set with it. Uncomment a property
to use a value different than the default. Properties that are unset and uncommented must be
set by the user.
+To run, Fluo requires the Hadoop, Accumulo, and Zookeeper jars for the versions
+running on the cluster where Fluo will run. In order to configure this a
+classpath must be constructed with these jars for Fluo. This can be done by
+editing `conf/fluo-env.sh`, which also contains further documentation.
+
Fluo command script
-------------------
@@ -208,7 +202,6 @@ Use the following commands to stop a local Fluo process:
In a distributed environment, you will need to deploy and configure a Fluo
distribution on every node in your cluster.
-[MiniFluo]: mini-fluo-setup.md
[fluo-dev]: https://github.com/fluo-io/fluo-dev
[Accumulo]: https://accumulo.apache.org/
[Hadoop]: http://hadoop.apache.org/
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/cluster/src/main/java/org/apache/fluo/cluster/command/FluoCommand.java
----------------------------------------------------------------------
diff --git a/modules/cluster/src/main/java/org/apache/fluo/cluster/command/FluoCommand.java b/modules/cluster/src/main/java/org/apache/fluo/cluster/command/FluoCommand.java
index dbf5939..24852f1 100644
--- a/modules/cluster/src/main/java/org/apache/fluo/cluster/command/FluoCommand.java
+++ b/modules/cluster/src/main/java/org/apache/fluo/cluster/command/FluoCommand.java
@@ -61,9 +61,6 @@ public class FluoCommand {
try (YarnAppRunner runner = new YarnAppRunner(hadoopPrefix)) {
switch (command.toLowerCase()) {
- case "classpath":
- runner.classpath(fluoHomeDir, remainArgs);
- break;
case "init":
runner.init(fluoInstall.getAppConfiguration(appName),
fluoInstall.getAppPropsPath(appName), remainArgs);
@@ -102,7 +99,7 @@ public class FluoCommand {
runner.waitUntilFinished(fluoInstall.resolveFluoConfiguration(appName));
break;
case "exec":
- runner.exec(fluoInstall.resolveFluoConfiguration(appName), remainArgs);
+ runner.exec(fluoInstall.resolveFluoConfiguration(appName, false), remainArgs);
break;
default:
System.err.println("Unknown command: " + command);
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/cluster/src/main/java/org/apache/fluo/cluster/command/MiniFluoCommand.java
----------------------------------------------------------------------
diff --git a/modules/cluster/src/main/java/org/apache/fluo/cluster/command/MiniFluoCommand.java b/modules/cluster/src/main/java/org/apache/fluo/cluster/command/MiniFluoCommand.java
deleted file mode 100644
index 713c3fe..0000000
--- a/modules/cluster/src/main/java/org/apache/fluo/cluster/command/MiniFluoCommand.java
+++ /dev/null
@@ -1,123 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
- * agreements. See the NOTICE file distributed with this work for additional information regarding
- * copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance with the License. You may obtain a
- * copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software distributed under the License
- * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
- * or implied. See the License for the specific language governing permissions and limitations under
- * the License.
- */
-
-package org.apache.fluo.cluster.command;
-
-import java.io.File;
-import java.util.Arrays;
-
-import ch.qos.logback.classic.Level;
-import ch.qos.logback.classic.Logger;
-import org.apache.fluo.api.config.FluoConfiguration;
-import org.apache.fluo.cluster.runner.MiniAppRunner;
-import org.apache.fluo.cluster.util.FluoInstall;
-import org.apache.fluo.mini.MiniFluoImpl;
-import org.slf4j.LoggerFactory;
-
-public class MiniFluoCommand {
-
- static FluoInstall fluoInstall;
-
- public static void verifyNoArgs(String[] remainArgs) {
- if (remainArgs.length != 0) {
- System.err.println("ERROR - Received unexpected command-line arguments: "
- + Arrays.toString(remainArgs));
- System.exit(-1);
- }
- }
-
- public static FluoConfiguration chooseConfig(String appName) {
-
- FluoConfiguration chosenConfig = null;
- String propsPath;
- FluoConfiguration fluoConfig = fluoInstall.resolveFluoConfiguration(appName, false);
- if (fluoConfig.hasRequiredClientProps()) {
- chosenConfig = fluoConfig;
- propsPath = fluoInstall.getFluoPropsPath();
- } else {
- File miniDataDir = new File(fluoConfig.getMiniDataDir());
- if (!miniDataDir.exists()) {
- System.err.println("Cannot connect to Fluo '" + fluoConfig.getApplicationName()
- + "' application! Client properties are not set in fluo.properties and "
- + " a MiniAccumuloCluster is not running at " + miniDataDir.getAbsolutePath());
- System.exit(-1);
- }
- if (!fluoConfig.hasRequiredMiniFluoProps()) {
- System.err.println("Fluo properties are not configured correctly!");
- System.exit(-1);
- }
- propsPath = MiniFluoImpl.clientPropsPath(fluoConfig);
- File clientPropsFile = new File(propsPath);
- if (!clientPropsFile.exists()) {
- System.err.println("MiniFluo client.properties do not exist at "
- + clientPropsFile.getAbsolutePath());
- System.exit(-1);
- }
- chosenConfig = new FluoConfiguration(clientPropsFile);
- }
-
- System.out.println("Connecting to MiniFluo instance (" + chosenConfig.getInstanceZookeepers()
- + ") using config (" + fluoInstall.stripFluoHomeDir(propsPath) + ")");
- return chosenConfig;
- }
-
- public static void main(String[] args) throws Exception {
-
- if (args.length < 3) {
- System.err.println("ERROR - Expected at least two arguments. "
- + "Usage: MiniFluoCommand <fluoHomeDir> <command> <appName> ...");
- System.exit(-1);
- }
-
- String fluoHomeDir = args[0];
- String command = args[1];
- String appName = args[2];
- String[] remainArgs = Arrays.copyOfRange(args, 3, args.length);
-
- if (command.equalsIgnoreCase("scan")) {
- for (String logger : new String[] {Logger.ROOT_LOGGER_NAME, "org.apache.fluo"}) {
- ((Logger) LoggerFactory.getLogger(logger)).setLevel(Level.ERROR);
- }
- } else if (command.equalsIgnoreCase("wait")) {
- ((Logger) LoggerFactory.getLogger(FluoConfiguration.class)).setLevel(Level.ERROR);
- }
-
- fluoInstall = new FluoInstall(fluoHomeDir);
- MiniAppRunner runner = new MiniAppRunner();
-
- switch (command.toLowerCase()) {
- case "classpath":
- runner.classpath(fluoHomeDir, remainArgs);
- break;
- case "scan":
- runner.scan(chooseConfig(appName), remainArgs);
- break;
- case "cleanup":
- verifyNoArgs(remainArgs);
- runner.cleanup(fluoInstall.resolveFluoConfiguration(appName, false));
- break;
- case "wait":
- verifyNoArgs(remainArgs);
- runner.waitUntilFinished(chooseConfig(appName));
- break;
- case "exec":
- runner.exec(chooseConfig(appName), remainArgs);
- break;
- default:
- System.err.println("Unknown command: " + command);
- break;
- }
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/cluster/src/main/java/org/apache/fluo/cluster/runner/AppRunner.java
----------------------------------------------------------------------
diff --git a/modules/cluster/src/main/java/org/apache/fluo/cluster/runner/AppRunner.java b/modules/cluster/src/main/java/org/apache/fluo/cluster/runner/AppRunner.java
index c906ac5..ac0da34 100644
--- a/modules/cluster/src/main/java/org/apache/fluo/cluster/runner/AppRunner.java
+++ b/modules/cluster/src/main/java/org/apache/fluo/cluster/runner/AppRunner.java
@@ -15,7 +15,6 @@
package org.apache.fluo.cluster.runner;
-import java.io.File;
import java.lang.reflect.Method;
import java.util.Arrays;
import java.util.Map;
@@ -234,67 +233,6 @@ public abstract class AppRunner {
return entriesFound;
}
- private static void appendLib(StringBuilder classpath, String libDirName, boolean useLibJarsFormat) {
- File libDir = new File(libDirName);
- if (!libDir.exists()) {
- System.err.println("ERROR - Directory needed for classpath does not exist: " + libDirName);
- System.exit(-1);
- }
-
- if (useLibJarsFormat) {
- File[] files = libDir.listFiles();
- if (files != null) {
- Arrays.sort(files);
- for (File f : files) {
- if (f.isFile() && f.getName().endsWith(".jar")) {
- if (classpath.length() != 0) {
- classpath.append(",");
- }
- classpath.append(f.getAbsolutePath());
- }
- }
- }
- } else {
- if (classpath.length() != 0) {
- classpath.append(":");
- }
- classpath.append(libDir.getAbsolutePath()).append("/*");
- }
- }
-
- public void classpath(String fluoHomeDir, String[] args) {
- ClasspathOptions options = new ClasspathOptions();
- JCommander jcommand = new JCommander(options);
- jcommand.setProgramName(scriptName + " classpath");
- try {
- jcommand.parse(args);
- } catch (ParameterException e) {
- System.err.println(e.getMessage());
- jcommand.usage();
- System.exit(-1);
- }
-
- if (options.help) {
- jcommand.usage();
- System.exit(0);
- }
-
- StringBuilder classpath = new StringBuilder();
-
- appendLib(classpath, fluoHomeDir + "/lib/fluo-client", options.getLibJars());
- if (options.getAccumulo()) {
- appendLib(classpath, fluoHomeDir + "/lib/accumulo", options.getLibJars());
- }
- if (options.getZookeepers()) {
- appendLib(classpath, fluoHomeDir + "/lib/zookeeper", options.getLibJars());
- }
- if (options.getHadoop()) {
- appendLib(classpath, fluoHomeDir + "/lib/hadoop-client", options.getLibJars());
- }
-
- System.out.println(classpath.toString());
- }
-
private long calculateSleep(long notifyCount, long numWorkers) {
long sleep = notifyCount / numWorkers / 100;
if (sleep < MIN_SLEEP_SEC) {
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/cluster/src/main/java/org/apache/fluo/cluster/runner/ClasspathOptions.java
----------------------------------------------------------------------
diff --git a/modules/cluster/src/main/java/org/apache/fluo/cluster/runner/ClasspathOptions.java b/modules/cluster/src/main/java/org/apache/fluo/cluster/runner/ClasspathOptions.java
deleted file mode 100644
index 61727bb..0000000
--- a/modules/cluster/src/main/java/org/apache/fluo/cluster/runner/ClasspathOptions.java
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
- * agreements. See the NOTICE file distributed with this work for additional information regarding
- * copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance with the License. You may obtain a
- * copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software distributed under the License
- * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
- * or implied. See the License for the specific language governing permissions and limitations under
- * the License.
- */
-
-package org.apache.fluo.cluster.runner;
-
-import com.beust.jcommander.Parameter;
-
-public class ClasspathOptions {
-
- @Parameter(names = {"-a", "--accumulo"}, description = "Additionally prints Accumulo client jars")
- private boolean accumulo = false;
-
- @Parameter(names = {"-z", "--zookeeper"},
- description = "Additionally prints Zookeeper client jars")
- private boolean zookeeper = false;
-
- @Parameter(names = {"-H", "--hadoop"}, description = "Additionally prints Hadoop client jars")
- private boolean hadoop = false;
-
- @Parameter(names = {"-l", "--lib-jars"},
- description = "Prints classpath in format suitable for Hadoop -libjars")
- private boolean libJars = false;
-
- @Parameter(names = {"-h", "-help", "--help"}, help = true, description = "Prints help")
- public boolean help;
-
- public boolean getAccumulo() {
- return accumulo;
- }
-
- public boolean getHadoop() {
- return hadoop;
- }
-
- public boolean getZookeepers() {
- return zookeeper;
- }
-
- public boolean getLibJars() {
- return libJars;
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/cluster/src/main/java/org/apache/fluo/cluster/runner/MiniAppRunner.java
----------------------------------------------------------------------
diff --git a/modules/cluster/src/main/java/org/apache/fluo/cluster/runner/MiniAppRunner.java b/modules/cluster/src/main/java/org/apache/fluo/cluster/runner/MiniAppRunner.java
deleted file mode 100644
index bbffa37..0000000
--- a/modules/cluster/src/main/java/org/apache/fluo/cluster/runner/MiniAppRunner.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
- * agreements. See the NOTICE file distributed with this work for additional information regarding
- * copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance with the License. You may obtain a
- * copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software distributed under the License
- * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
- * or implied. See the License for the specific language governing permissions and limitations under
- * the License.
- */
-
-package org.apache.fluo.cluster.runner;
-
-import java.io.File;
-import java.io.IOException;
-
-import org.apache.commons.io.FileUtils;
-import org.apache.fluo.api.config.FluoConfiguration;
-
-/**
- * Runs applications in MiniFluo
- */
-public class MiniAppRunner extends AppRunner {
-
- public MiniAppRunner() {
- super("mini-fluo");
- }
-
- public void cleanup(FluoConfiguration appConfig) {
- File dataDir = new File(appConfig.getMiniDataDir());
- if (dataDir.exists() && appConfig.getMiniStartAccumulo()) {
- try {
- FileUtils.deleteDirectory(dataDir);
- } catch (IOException e) {
- throw new IllegalStateException(e);
- }
- }
- }
-
-}
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/cluster/src/test/java/org/apache/fluo/cluster/runner/it/MiniAppRunnerIT.java
----------------------------------------------------------------------
diff --git a/modules/cluster/src/test/java/org/apache/fluo/cluster/runner/it/MiniAppRunnerIT.java b/modules/cluster/src/test/java/org/apache/fluo/cluster/runner/it/MiniAppRunnerIT.java
deleted file mode 100644
index 12948c3..0000000
--- a/modules/cluster/src/test/java/org/apache/fluo/cluster/runner/it/MiniAppRunnerIT.java
+++ /dev/null
@@ -1,89 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
- * agreements. See the NOTICE file distributed with this work for additional information regarding
- * copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance with the License. You may obtain a
- * copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software distributed under the License
- * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
- * or implied. See the License for the specific language governing permissions and limitations under
- * the License.
- */
-
-package org.apache.fluo.cluster.runner.it;
-
-import java.util.Collections;
-import java.util.List;
-
-import org.apache.fluo.api.config.ObserverConfiguration;
-import org.apache.fluo.api.data.Bytes;
-import org.apache.fluo.api.data.Column;
-import org.apache.fluo.api.types.StringEncoder;
-import org.apache.fluo.api.types.TypeLayer;
-import org.apache.fluo.api.types.TypedObserver;
-import org.apache.fluo.api.types.TypedTransaction;
-import org.apache.fluo.api.types.TypedTransactionBase;
-import org.apache.fluo.cluster.runner.MiniAppRunner;
-import org.apache.fluo.core.impl.Environment;
-import org.apache.fluo.integration.ITBaseMini;
-import org.junit.Assert;
-import org.junit.Test;
-
-public class MiniAppRunnerIT extends ITBaseMini {
-
- private static TypeLayer tl = new TypeLayer(new StringEncoder());
-
- public static class TestObserver extends TypedObserver {
-
- @Override
- public ObservedColumn getObservedColumn() {
- return new ObservedColumn(new Column("ingest", "num"), NotificationType.STRONG);
- }
-
- @Override
- public void process(TypedTransactionBase tx, Bytes row, Column col) {
- try {
- Thread.sleep(1000);
- } catch (InterruptedException e) {
- throw new IllegalStateException(e);
- }
- Column countColumn = new Column(Bytes.of("count"), col.getQualifier());
- int ingest = tx.get().row(row).col(col).toInteger(0);
- int count = tx.get().row(row).col(countColumn).toInteger(0);
- tx.mutate().row(row).col(countColumn).set(count + ingest);
- tx.mutate().row(row).col(col).delete();
- }
- }
-
- @Override
- protected List<ObserverConfiguration> getObservers() {
- return Collections.singletonList(new ObserverConfiguration(TestObserver.class.getName()));
- }
-
- @Test
- public void testSimple() {
- try (TypedTransaction tx1 = tl.wrap(client.newTransaction())) {
- tx1.mutate().row("row1").fam("ingest").qual("num").set(1);
- tx1.mutate().row("row2").fam("ingest").qual("num").set(1);
- tx1.mutate().row("row3").fam("ingest").qual("num").set(1);
- tx1.commit();
- }
-
- Environment env = new Environment(config);
- MiniAppRunner runner = new MiniAppRunner();
-
- Assert.assertEquals(3, runner.countNotifications(env));
-
- runner.waitUntilFinished(config);
-
- Assert.assertEquals(0, runner.countNotifications(env));
- Assert.assertEquals(3, runner.scan(config, new String[] {}));
- Assert.assertEquals(1, runner.scan(config, new String[] {"-r", "row1"}));
- Assert.assertEquals(1, runner.scan(config, new String[] {"-r", "row2"}));
- Assert.assertEquals(3, runner.scan(config, new String[] {"-c", "count"}));
- Assert.assertEquals(0, runner.scan(config, new String[] {"-c", "ingest"}));
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/distribution/pom.xml
----------------------------------------------------------------------
diff --git a/modules/distribution/pom.xml b/modules/distribution/pom.xml
index 2a1ed43..bc2aacf 100644
--- a/modules/distribution/pom.xml
+++ b/modules/distribution/pom.xml
@@ -15,21 +15,17 @@
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
-
<parent>
<groupId>org.apache.fluo</groupId>
<artifactId>fluo-parent</artifactId>
<version>1.0.0-beta-3-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
-
<artifactId>fluo-distribution</artifactId>
<packaging>pom</packaging>
-
<name>Fluo Distribution</name>
<description>This module produces a tarball distribution of Fluo. It contains all of the default configuration
and scripts required for the distribution.</description>
-
<!-- NOTE: These dependencies are bundled in this assembly -->
<dependencies>
<dependency>
@@ -65,7 +61,6 @@
<artifactId>slf4j-log4j12</artifactId>
</dependency>
</dependencies>
-
<build>
<plugins>
<plugin>
@@ -86,9 +81,17 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <groupId>org.apache.rat</groupId>
+ <artifactId>apache-rat-plugin</artifactId>
+ <configuration>
+ <excludes>
+ <exclude>src/main/deps/**</exclude>
+ </excludes>
+ </configuration>
+ </plugin>
</plugins>
</build>
-
<profiles>
<profile>
<id>full</id>
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/distribution/src/main/assembly/bin.xml
----------------------------------------------------------------------
diff --git a/modules/distribution/src/main/assembly/bin.xml b/modules/distribution/src/main/assembly/bin.xml
index 522c465..da9d0e0 100644
--- a/modules/distribution/src/main/assembly/bin.xml
+++ b/modules/distribution/src/main/assembly/bin.xml
@@ -27,22 +27,6 @@
<scope>runtime</scope>
<outputFileNameMapping>${artifact.artifactId}${dashClassifier?}-${artifact.version}.${artifact.extension}</outputFileNameMapping>
<useTransitiveDependencies>true</useTransitiveDependencies>
- <excludes>
- <exclude>*:slf4j-log4j12</exclude>
- <exclude>*:log4j</exclude>
- <exclude>*:logback-classic</exclude>
- <exclude>*:logback-core</exclude>
- <exclude>*:log4j-over-slf4j</exclude>
- </excludes>
- </dependencySet>
- <dependencySet>
- <outputDirectory>lib/fluo-client</outputDirectory>
- <directoryMode>0755</directoryMode>
- <fileMode>0644</fileMode>
- <useProjectArtifact>true</useProjectArtifact>
- <scope>runtime</scope>
- <outputFileNameMapping>${artifact.artifactId}${dashClassifier?}-${artifact.version}.${artifact.extension}</outputFileNameMapping>
- <useTransitiveDependencies>true</useTransitiveDependencies>
<useTransitiveFiltering>true</useTransitiveFiltering>
<includes>
<include>org.apache.fluo:fluo-api</include>
@@ -50,59 +34,32 @@
<include>org.apache.fluo:fluo-core</include>
<include>org.apache.fluo:fluo-mini</include>
<include>org.apache.fluo:fluo-mapreduce</include>
+ <include>org.apache.fluo:fluo-cluster</include>
<include>org.slf4j:slf4j-api</include>
- <!--fluo-core has direct dependencies on following, but for some reason maven does not include -->
<include>org.apache.curator:curator-framework</include>
<include>org.apache.curator:curator-recipes</include>
</includes>
<excludes>
<exclude>org.apache.accumulo:*</exclude>
+ <exclude>org.apache.thrift:*</exclude>
<exclude>org.apache.hadoop:*</exclude>
<exclude>org.apache.zookeeper:zookeeper</exclude>
+ <exclude>*:slf4j-log4j12</exclude>
+ <exclude>*:log4j</exclude>
+ <exclude>*:logback-classic</exclude>
+ <exclude>*:logback-core</exclude>
+ <exclude>*:log4j-over-slf4j</exclude>
</excludes>
</dependencySet>
<dependencySet>
- <outputDirectory>lib/accumulo</outputDirectory>
- <directoryMode>0755</directoryMode>
- <fileMode>0644</fileMode>
- <useProjectArtifact>false</useProjectArtifact>
- <scope>runtime</scope>
- <outputFileNameMapping>${artifact.artifactId}${dashClassifier?}-${artifact.version}.${artifact.extension}</outputFileNameMapping>
- <useTransitiveDependencies>true</useTransitiveDependencies>
- <useTransitiveFiltering>true</useTransitiveFiltering>
- <includes>
- <include>org.apache.accumulo:accumulo-core</include>
- <include>org.apache.accumulo:accumulo-minicluster</include>
- </includes>
- <excludes>
- <exclude>org.apache.hadoop:hadoop-minicluster</exclude>
- </excludes>
- </dependencySet>
- <dependencySet>
- <outputDirectory>lib/hadoop-client</outputDirectory>
- <directoryMode>0755</directoryMode>
- <fileMode>0644</fileMode>
- <useProjectArtifact>false</useProjectArtifact>
- <scope>runtime</scope>
- <outputFileNameMapping>${artifact.artifactId}${dashClassifier?}-${artifact.version}.${artifact.extension}</outputFileNameMapping>
- <useTransitiveDependencies>true</useTransitiveDependencies>
- <useTransitiveFiltering>true</useTransitiveFiltering>
- <includes>
- <include>org.apache.hadoop:hadoop-client</include>
- </includes>
- </dependencySet>
- <dependencySet>
- <outputDirectory>lib/zookeeper</outputDirectory>
- <directoryMode>0755</directoryMode>
- <fileMode>0644</fileMode>
- <useProjectArtifact>false</useProjectArtifact>
- <scope>runtime</scope>
- <outputFileNameMapping>${artifact.artifactId}${dashClassifier?}-${artifact.version}.${artifact.extension}</outputFileNameMapping>
- <useTransitiveDependencies>true</useTransitiveDependencies>
- <useTransitiveFiltering>true</useTransitiveFiltering>
- <includes>
- <include>org.apache.zookeeper:zookeeper</include>
- </includes>
+ <!-- this is in separate section because it was transitively filtered -->
+ <outputDirectory>lib</outputDirectory>
+ <directoryMode>0755</directoryMode>
+ <fileMode>0644</fileMode>
+ <!-- was having a problem with version 2.2.2 from Accumulo-->
+ <includes>
+ <include>com.google.code.gson:gson</include>
+ </includes>
</dependencySet>
<dependencySet>
<outputDirectory>lib/log4j</outputDirectory>
@@ -160,6 +117,13 @@
</includes>
</fileSet>
<fileSet>
+ <directory>src/main/deps</directory>
+ <outputDirectory>lib/deps</outputDirectory>
+ <includes>
+ <include>*</include>
+ </includes>
+ </fileSet>
+ <fileSet>
<directory>${project.basedir}/../../docs</directory>
<outputDirectory>docs</outputDirectory>
<includes>
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/distribution/src/main/config/fluo-env.sh
----------------------------------------------------------------------
diff --git a/modules/distribution/src/main/config/fluo-env.sh b/modules/distribution/src/main/config/fluo-env.sh
index 8594c78..b344775 100755
--- a/modules/distribution/src/main/config/fluo-env.sh
+++ b/modules/distribution/src/main/config/fluo-env.sh
@@ -14,4 +14,66 @@
# Sets HADOOP_PREFIX if it is not already set. Please modify the
# export statement to use the correct directory. Remove the test
# statement to override any previously set environment.
+
+
test -z "$HADOOP_PREFIX" && export HADOOP_PREFIX=/path/to/hadoop
+
+# The classpath for Fluo must be defined. The Fluo tarball does not include
+# jars for Accumulo, Zookeeper, or Hadoop. This example env file offers two
+# ways to setup the classpath with these jars. Go to the end of the file for
+# more info.
+
+addToClasspath()
+{
+ local dir=$1
+ local filterRegex=$2
+
+ if [ ! -d "$dir" ]; then
+ echo "ERROR $dir does not exist or not a directory"
+ exit 1
+ fi
+
+ for jar in $dir/*.jar; do
+ if ! [[ $jar =~ $filterRegex ]]; then
+ CLASSPATH="$CLASSPATH:$jar"
+ fi
+ done
+}
+
+
+# This function attemps to obtain Accumulo, Hadoop, and Zookeeper jars from the
+# location where those dependencies are installed on the system.
+setupClasspathFromSystem()
+{
+ test -z "$ACCUMULO_HOME" && ACCUMULO_HOME=/path/to/accumulo
+ test -z "$ZOOKEEPER_HOME" && ZOOKEEPER_HOME=/path/to/zookeeper
+
+ CLASSPATH="$FLUO_HOME/lib/*:$FLUO_HOME/lib/logback/*"
+
+ #any jars matching this pattern is excluded from classpath
+ EXCLUDE_RE="(.*log4j.*)|(.*asm.*)|(.*guava.*)|(.*gson.*)"
+
+ addToClasspath "$ACCUMULO_HOME/lib" $EXCLUDE_RE
+ addToClasspath "$ZOOKEEPER_HOME" $EXCLUDE_RE
+ addToClasspath "$ZOOKEEPER_HOME/lib" $EXCLUDE_RE
+ addToClasspath "$HADOOP_PREFIX/share/hadoop/common" $EXCLUDE_RE;
+ addToClasspath "$HADOOP_PREFIX/share/hadoop/common/lib" $EXCLUDE_RE;
+ addToClasspath "$HADOOP_PREFIX/share/hadoop/hdfs" $EXCLUDE_RE;
+ addToClasspath "$HADOOP_PREFIX/share/hadoop/hdfs/lib" $EXCLUDE_RE;
+ addToClasspath "$HADOOP_PREFIX/share/hadoop/yarn" $EXCLUDE_RE;
+ addToClasspath "$HADOOP_PREFIX/share/hadoop/yarn/lib" $EXCLUDE_RE;
+}
+
+
+# This function obtains Accumulo, Hadoop, and Zookeeper jars from
+# $FLUO_HOME/lib/deps/. Before using this function, go to that directoy and
+# follow the readme to download the dependencies.
+setupClasspathFromLib(){
+ CLASSPATH="$FLUO_HOME/lib/*:$FLUO_HOME/lib/logback/*:$FLUO_HOME/lib/deps/*"
+}
+
+# Call one of the following functions to setup the classpath or write your own
+# bash code to setup the classpath for Fluo.
+
+setupClasspathFromSystem
+#setupClasspathFromLib
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/distribution/src/main/deps/README.md
----------------------------------------------------------------------
diff --git a/modules/distribution/src/main/deps/README.md b/modules/distribution/src/main/deps/README.md
new file mode 100644
index 0000000..0457fe8
--- /dev/null
+++ b/modules/distribution/src/main/deps/README.md
@@ -0,0 +1,14 @@
+Fluo Dependencies
+-----------------
+
+This directory provides a simple maven pom that can be used to fetch Accumulo,
+Hadoop, and Zookeeper depedencies. You should fetch the versions of these
+dependencies that are installed on your system.
+
+ cd $FLUO_HOME/lib/deps/
+ #edit pom to reflect versions of Software on your system
+ vim pom.xml
+ rm *.jar;mvn package clean
+
+After doing this edit `$FLUO_HOME/conf/fluo-env.sh` to add
+`$FLUO_HOME/lib/deps/*` to the classpath.
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/distribution/src/main/deps/pom.xml
----------------------------------------------------------------------
diff --git a/modules/distribution/src/main/deps/pom.xml b/modules/distribution/src/main/deps/pom.xml
new file mode 100644
index 0000000..68c2576
--- /dev/null
+++ b/modules/distribution/src/main/deps/pom.xml
@@ -0,0 +1,64 @@
+<project>
+
+ <modelVersion>4.0.0</modelVersion>
+ <packaging>jar</packaging>
+ <groupId>a</groupId>
+ <artifactId>a</artifactId>
+ <version>a</version>
+
+ <properties>
+ <accumulo.version>1.7.1</accumulo.version>
+ <hadoop.version>2.6.3</hadoop.version>
+ <zookeeper.version>3.4.8</zookeeper.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.accumulo</groupId>
+ <artifactId>accumulo-core</artifactId>
+ <version>${accumulo.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.zookeeper</groupId>
+ <artifactId>zookeeper</artifactId>
+ <version>${zookeeper.version}</version>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <version>2.10</version>
+ <executions>
+ <execution>
+ <id>copy-dependencies</id>
+ <phase>package</phase>
+ <goals>
+ <goal>copy-dependencies</goal>
+ </goals>
+ <configuration>
+ <outputDirectory>${project.build.directory}/..</outputDirectory>
+ <overWriteReleases>false</overWriteReleases>
+ <overWriteSnapshots>false</overWriteSnapshots>
+ <overWriteIfNewer>true</overWriteIfNewer>
+ <excludeArtifactIds>guava,gson,asm,log4j,slf4j-api,slf4j-log4j12</excludeArtifactIds>
+ <includeScope>runtime</includeScope>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/distribution/src/main/scripts/fluo
----------------------------------------------------------------------
diff --git a/modules/distribution/src/main/scripts/fluo b/modules/distribution/src/main/scripts/fluo
index ae24c64..b9cd984 100755
--- a/modules/distribution/src/main/scripts/fluo
+++ b/modules/distribution/src/main/scripts/fluo
@@ -26,6 +26,8 @@ script=$( basename "$SOURCE" )
. "$bin"/impl/config.sh
+export CLASSPATH
+
# stop if any command fails
set -e
@@ -57,7 +59,7 @@ function print_usage {
echo " status <app> Prints status of Fluo application"
echo " info <app> Prints information about containers of Fluo application"
echo " scan <app> Prints snapshot of data in Fluo application"
- echo " classpath Prints the classpath needed for fluo clients. Run with -h to learn more."
+ echo " classpath Prints the classpath setup in fluo-env.sh"
echo " wait <app> Waits until all notifications are processed"
echo " exec <app> <class> {<argument>}";
echo " "
@@ -76,7 +78,7 @@ function validate_app {
function validate_app_full {
validate_app
- java -cp "$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*" org.apache.fluo.cluster.util.ValidateAppName $APP
+ java org.apache.fluo.cluster.util.ValidateAppName $APP
if [[ ! -d $APP_DIR || ! -d $APP_CONF_DIR || ! -d $APP_LIB_DIR ]]; then
echo "ERROR - The Fluo '$APP' application needs to be configured in apps/ with a conf/ and lib/ directory. Use 'fluo new $APP' to create this configuration"
exit 1
@@ -97,7 +99,7 @@ function check_hadoop {
case "$1" in
new)
validate_app
- java -cp "$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*" org.apache.fluo.cluster.util.ValidateAppName $APP
+ java org.apache.fluo.cluster.util.ValidateAppName $APP
if [ -d $APP_DIR ]; then
echo "The Fluo '$APP' application already has a directory in apps/"
exit 1
@@ -120,27 +122,30 @@ init)
echo "Copying `ls $FLUO_HOME/lib/fluo-accumulo-*.jar` to HDFS"
$HADOOP_PREFIX/bin/hdfs dfs -copyFromLocal -f $FLUO_HOME/lib/fluo-accumulo-*.jar /fluo/lib/
fi
- java -cp "$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*:$APP_LIB_DIR/*" org.apache.fluo.cluster.command.FluoCommand $FLUO_HOME $HADOOP_PREFIX "$@"
+ export CLASSPATH="$APP_LIB_DIR/*:$CLASSPATH"
+ java org.apache.fluo.cluster.command.FluoCommand $FLUO_HOME $HADOOP_PREFIX "$@"
;;
start)
validate_app_full
check_hadoop
- java -cp "$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*:$APP_LIB_DIR/*" org.apache.fluo.cluster.command.FluoCommand $FLUO_HOME $HADOOP_PREFIX "$@"
+ export CLASSPATH="$APP_LIB_DIR/*:$CLASSPATH"
+ java org.apache.fluo.cluster.command.FluoCommand $FLUO_HOME $HADOOP_PREFIX "$@"
;;
stop|kill|status|info|scan|wait)
validate_app
check_hadoop
- java -cp "$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*" org.apache.fluo.cluster.command.FluoCommand $FLUO_HOME $HADOOP_PREFIX "$@"
+ java org.apache.fluo.cluster.command.FluoCommand $FLUO_HOME $HADOOP_PREFIX "$@"
;;
list)
check_hadoop
- java -cp "$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*" org.apache.fluo.cluster.command.FluoCommand $FLUO_HOME $HADOOP_PREFIX $1 app ${*:2}
+ java org.apache.fluo.cluster.command.FluoCommand $FLUO_HOME $HADOOP_PREFIX $1 app ${*:2}
;;
classpath)
- java -cp "$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*" org.apache.fluo.cluster.command.FluoCommand $FLUO_HOME none $1 app ${*:2}
+ echo "$CLASSPATH"
;;
exec)
- java -cp "$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*:$APP_LIB_DIR/*" org.apache.fluo.cluster.command.FluoCommand $FLUO_HOME $HADOOP_PREFIX "$@"
+ export CLASSPATH="$APP_LIB_DIR/*:$CLASSPATH"
+ java org.apache.fluo.cluster.command.FluoCommand $FLUO_HOME $HADOOP_PREFIX "$@"
;;
*)
print_usage
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/distribution/src/main/scripts/impl/config.sh
----------------------------------------------------------------------
diff --git a/modules/distribution/src/main/scripts/impl/config.sh b/modules/distribution/src/main/scripts/impl/config.sh
index b386a77..7abbbe5 100755
--- a/modules/distribution/src/main/scripts/impl/config.sh
+++ b/modules/distribution/src/main/scripts/impl/config.sh
@@ -25,10 +25,6 @@ bin="$( cd -P "$( dirname "$impl" )" && pwd )"
script=$( basename "$SOURCE" )
# Stop: Resolve Script Directory
-if [ -f "$bin"/../conf/fluo-env.sh ]; then
- . "$bin"/../conf/fluo-env.sh
-fi
-
# Determine FLUO_HOME - Use env variable set by user. If none set, calculate using bin dir
FLUO_HOME="${FLUO_HOME:-$( cd -P ${bin}/.. && pwd )}"
export FLUO_HOME
@@ -56,6 +52,10 @@ then
exit 1
fi
+if [ -f "$bin"/../conf/fluo-env.sh ]; then
+ . "$bin"/../conf/fluo-env.sh
+fi
+
if [[ "$OSTYPE" == "darwin"* ]]; then
export MD5=md5
export SED="sed -i .bak"
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/distribution/src/main/scripts/local-fluo
----------------------------------------------------------------------
diff --git a/modules/distribution/src/main/scripts/local-fluo b/modules/distribution/src/main/scripts/local-fluo
index 748620d..16ce77e 100755
--- a/modules/distribution/src/main/scripts/local-fluo
+++ b/modules/distribution/src/main/scripts/local-fluo
@@ -26,6 +26,8 @@ script=$( basename "$SOURCE" )
. "$bin"/impl/config.sh
+export CLASSPATH
+
# stop if any command fails
set -e
@@ -54,7 +56,7 @@ function validate_app_full {
if [[ $APP = *"-h"* ]]; then
print_usage
fi
- java -cp "$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*" org.apache.fluo.cluster.util.ValidateAppName $APP
+ java org.apache.fluo.cluster.util.ValidateAppName $APP
}
function check_dirs {
@@ -78,14 +80,13 @@ function kill_if_running {
LOGHOST=$(hostname)
LOCAL_OPTS="$APP_CONF_DIR/fluo.properties"
-LOCAL_LIB="$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*"
case "$1" in
start-oracle)
validate_app_full
check_dirs
SERVICE="oracle"
- java -Dfluo.app=$APP -cp "$LOCAL_LIB" org.apache.fluo.core.oracle.FluoOracleImpl $LOCAL_OPTS >${APP_LOG_DIR}/${SERVICE}_${LOGHOST}.out 2>${APP_LOG_DIR}/${SERVICE}_${LOGHOST}.err &
+ java -Dfluo.app=$APP org.apache.fluo.core.oracle.FluoOracleImpl $LOCAL_OPTS >${APP_LOG_DIR}/${SERVICE}_${LOGHOST}.out 2>${APP_LOG_DIR}/${SERVICE}_${LOGHOST}.err &
echo "Started oracle for Fluo '$APP' application. View its logs at $APP_LOG_DIR"
;;
kill-oracle)
@@ -95,7 +96,8 @@ start-worker)
validate_app_full
check_dirs
SERVICE="worker"
- java -Dfluo.app=$APP -cp "$LOCAL_LIB:$APP_LIB_DIR/*" org.apache.fluo.core.worker.FluoWorkerImpl $LOCAL_OPTS >${APP_LOG_DIR}/${SERVICE}_${LOGHOST}.out 2>${APP_LOG_DIR}/${SERVICE}_${LOGHOST}.err &
+ export CLASSPATH="$APP_LIB_DIR/*:$CLASSPATH"
+ java -Dfluo.app=$APP org.apache.fluo.core.worker.FluoWorkerImpl $LOCAL_OPTS >${APP_LOG_DIR}/${SERVICE}_${LOGHOST}.out 2>${APP_LOG_DIR}/${SERVICE}_${LOGHOST}.err &
echo "Started worker for Fluo '$APP' application. View its logs at $APP_LOG_DIR"
;;
kill-worker)
http://git-wip-us.apache.org/repos/asf/incubator-fluo/blob/6fd86298/modules/distribution/src/main/scripts/mini-fluo
----------------------------------------------------------------------
diff --git a/modules/distribution/src/main/scripts/mini-fluo b/modules/distribution/src/main/scripts/mini-fluo
deleted file mode 100755
index 074b872..0000000
--- a/modules/distribution/src/main/scripts/mini-fluo
+++ /dev/null
@@ -1,160 +0,0 @@
-#! /usr/bin/env bash
-
-# Licensed to the Apache Software Foundation (ASF) under one or more contributor license
-# agreements. See the NOTICE file distributed with this work for additional information regarding
-# copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with the License. You may
-# obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
-# or implied. See the License for the specific language governing permissions and limitations under
-# the License.
-
-# Start: Resolve Script Directory
-SOURCE="${BASH_SOURCE[0]}"
-while [ -h "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symlink
- bin="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
- SOURCE="$(readlink "$SOURCE")"
- [[ $SOURCE != /* ]] && SOURCE="$bin/$SOURCE" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
-done
-bin="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
-script=$( basename "$SOURCE" )
-# Stop: Resolve Script Directory
-
-. "$bin"/impl/config.sh
-
-# stop if any command fails
-set -e
-
-APP=$2
-APP_DIR=$FLUO_HOME/apps/$APP
-APP_LIB_DIR=$APP_DIR/lib
-APP_CONF_DIR=$APP_DIR/conf
-APP_LOG_DIR=$APP_DIR/logs
-APP_MINI_DIR=$APP_DIR/mini
-APP_CLIENT_PROPS=$APP_MINI_DIR/client.properties
-
-function print_usage {
- echo -e "Usage: mini-fluo <command> (<argument> ..)\n"
- echo -e "Possible commands:\n"
- echo " new <app> Creates configuration for new MiniFluo application in apps/"
- echo " start <app> Starts application in MiniFluo on local machine"
- echo " kill <app> Kills application in MiniFluo on local machine"
- echo " scan <app> Prints snapshot of data in MiniFluo application"
- echo " classpath Prints the classpath needed for fluo clients. Run with -h to learn more."
- echo " wait <app> Waits until all notifications are processed"
- echo " exec <app> <class> {<argument>}";
- echo " "
- exit 1
-}
-
-function validate_app {
- if [ -z "$APP" ]; then
- echo -e "The application name (set by <app>) cannot be an empty string!\n"
- print_usage
- fi
- if [[ $APP = *"-h"* ]]; then
- print_usage
- fi
-}
-
-function validate_app_full {
- validate_app
- java -cp "$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*" org.apache.fluo.cluster.util.ValidateAppName $APP
-}
-
-function check_dirs {
- if [[ ! -d $APP_DIR || ! -d $APP_CONF_DIR || ! -d $APP_LIB_DIR ]]; then
- echo "ERROR - The Fluo '$APP' application needs to be configured in apps/ with a conf/ and lib/ directory. Use 'mini-fluo new $APP' to create this configuration"
- exit 1
- fi
- mkdir -p $APP_LOG_DIR 2>/dev/null
-}
-
-function copy_config {
- if [ -f $FLUO_CONF_DIR/$1 ]; then
- cp $FLUO_CONF_DIR/$1 $APP_CONF_DIR/
- elif [ -f $FLUO_CONF_DIR/examples/$1 ]; then
- cp FLUO_CONF_DIR/examples/$1 $APP_CONF_DIR
- else
- echo "Config file $1 not found in $FLUO_CONF_DIR or $FLUO_CONF_DIR/examples"
- exit 1
- fi
-}
-
-function kill_if_running {
- PID=`jps -m | grep $1 | grep $APP | cut -f 1 -d ' '`
- if [ -z "$PID" ]; then
- echo "ERROR - A $1 process for the Fluo application '$APP' is not running"
- exit 1
- else
- kill $PID
- fi
-}
-
-FLUO_LOG_HOST=$(hostname)
-FLUO_APP="mini"
-MINI_OPTS="$APP_CONF_DIR/fluo.properties"
-MINI_OUT=${APP_LOG_DIR}/${FLUO_APP}_${FLUO_LOG_HOST}.out
-MINI_ERR=${APP_LOG_DIR}/${FLUO_APP}_${FLUO_LOG_HOST}.err
-MINI_LIB="$APP_CONF_DIR/*:$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/log4j/*"
-
-START_OPTS="-Dlog4j.configuration=file:///$APP_CONF_DIR/log4j.xml -Dorg.apache.fluo.log.app=$FLUO_APP -Dorg.apache.fluo.log.host=$FLUO_LOG_HOST -Dorg.apache.fluo.log.dir=$APP_LOG_DIR"
-
-case "$1" in
-new)
- if [ -d $APP_DIR ]; then
- echo "The Fluo '$APP' application already has a directory in apps/"
- exit 1
- fi
- validate_app_full
- mkdir -p $APP_DIR
- mkdir -p $APP_CONF_DIR
- mkdir -p $APP_LIB_DIR
- copy_config fluo.properties
- $SED "s/org.apache.fluo.client.application.name=/org.apache.fluo.client.application.name=$APP/g" $APP_CONF_DIR/fluo.properties
- $SED "s/org.apache.fluo.client.accumulo/#org.apache.fluo.client.accumulo/g" $APP_CONF_DIR/fluo.properties
- $SED "s/org.apache.fluo.admin.accumulo.classpath/#org.apache.fluo.admin.accumulo.classpath/g" $APP_CONF_DIR/fluo.properties
- $SED "s/#org.apache.fluo.mini/org.apache.fluo.mini/g" $APP_CONF_DIR/fluo.properties
- copy_config log4j.xml
- ;;
-start)
- validate_app
- check_dirs
- java $START_OPTS -cp "$MINI_LIB:$FLUO_LIB_DIR/logback/*:$APP_LIB_DIR/*" org.apache.fluo.mini.MiniFluoImpl $MINI_OPTS >$MINI_OUT 2>$MINI_ERR &
- echo "Started '$APP' application in MiniFluo. View logs at $APP_LOG_DIR"
- echo "MiniFluo will create properties file for Fluo clients at $APP_CLIENT_PROPS"
- echo -n "Waiting for MiniFluo to create client.properties..."
- while [ ! -f $APP_CLIENT_PROPS ]
- do
- echo -n "."
- sleep 1
- done
- echo "."
- echo "Client properties were created. MiniFluo has started successfully!"
- ;;
-kill)
- validate_app
- check_dirs
- kill_if_running MiniFluoMain
- java -cp "$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*" org.apache.fluo.cluster.command.MiniFluoCommand $FLUO_HOME cleanup $APP
- echo "Killed '$APP' application running in MiniFluo"
- ;;
-scan|wait)
- validate_app
- check_dirs
- java -cp "$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*" org.apache.fluo.cluster.command.MiniFluoCommand $FLUO_HOME "$@"
- ;;
-classpath)
- java -cp "$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*:$APP_LIB_DIR/*" org.apache.fluo.cluster.command.MiniFluoCommand $FLUO_HOME $1 app ${*:2}
- ;;
-exec)
- java -cp "$FLUO_LIB_DIR/*:$FLUO_LIB_DIR/logback/*:$APP_LIB_DIR/*" org.apache.fluo.cluster.command.MiniFluoCommand $FLUO_HOME "$@"
- ;;
-
-*)
- print_usage
-esac