You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fluo.apache.org by mw...@apache.org on 2017/10/06 15:37:20 UTC
[fluo-website] branch gh-pages updated: Major refactor of Fluo docs
(#99)
This is an automated email from the ASF dual-hosted git repository.
mwalch pushed a commit to branch gh-pages
in repository https://gitbox.apache.org/repos/asf/fluo-website.git
The following commit(s) were added to refs/heads/gh-pages by this push:
new 203a281 Major refactor of Fluo docs (#99)
203a281 is described below
commit 203a281dca0029feac488d98e4c7d5de29efe192
Author: Mike Walch <mw...@apache.org>
AuthorDate: Fri Oct 6 11:37:18 2017 -0400
Major refactor of Fluo docs (#99)
* Added docker and yarn instructions
* Split up applications page
* Create quick start
---
_config.yml | 1 +
_fluo-1-2/administration/initialize.md | 81 ++++++
_fluo-1-2/administration/manage-applications.md | 107 +++++++
_fluo-1-2/administration/metrics.md | 2 +-
_fluo-1-2/administration/run-fluo-in-docker.md | 117 ++++++++
_fluo-1-2/administration/run-fluo-in-yarn.md | 101 +++++++
_fluo-1-2/administration/run-fluo-processes.md | 46 +++
_fluo-1-2/development/applications.md | 314 ---------------------
_fluo-1-2/getting-started/create-application.md | 139 +++++++++
.../{install.md => install-fluo.md} | 8 +-
_fluo-1-2/getting-started/quick-start.md | 27 ++
_layouts/fluo-1.2.html | 2 +-
12 files changed, 625 insertions(+), 320 deletions(-)
diff --git a/_config.yml b/_config.yml
index 35302ab..952b790 100644
--- a/_config.yml
+++ b/_config.yml
@@ -55,6 +55,7 @@ defaults:
docs_base: "/docs/fluo/1.2"
javadoc_base: "https://static.javadoc.io/org.apache.fluo/fluo-api/1.1.0-incubating"
github_base: "https://github.com/apache/fluo/blob/master"
+ github_yarn: "https://github.com/apache/fluo-yarn/blob/master"
-
scope:
path: "_recipes-1-2"
diff --git a/_fluo-1-2/administration/initialize.md b/_fluo-1-2/administration/initialize.md
new file mode 100644
index 0000000..5aaeaf1
--- /dev/null
+++ b/_fluo-1-2/administration/initialize.md
@@ -0,0 +1,81 @@
+---
+title: Initialize Application
+category: administration
+order: 1
+---
+
+## Overview
+
+Before a Fluo Application can run, it must be initialized. Below is an overview of what
+initialization does and some of the properties that may be set for initialization.
+
+ * **Initialize ZooKeeper** - Each application has its own area in ZooKeeper used for configuration,
+ Oracle state, and worker coordination. All properties, except `fluo.connections.*`, are copied
+ into ZooKeeper. For example, if `fluo.worker.num.threads=128` was set, then when a worker process
+ starts it will read this from ZooKeeper.
+ * **Copy Observer jars to DFS** - Fluo workers processes need the jars containing observers. These
+ are provided in one of the following ways.
+ * Set the property `fluo.observer.init.dir` to a local directory containing observer jars. The
+ jars in this directory are copied to DFS under `<fluo.dfs.root>/<app name>`. When a worker is
+ started, the jars are pulled from DFS and added to its classpath.
+ * Set the property `fluo.observer.jars.url` to a directory in DFS containing observer jars. No
+ copying is done. When a worker is started, the jars are pulled from this location and added to
+ its classpath.
+ * Do not set any of the properties above and have the mechanism that starts the worker process
+ add the needed jars to the classpath.
+ * **Create Accumulo table** - Each Fluo application creates and configures an Accumulo table. The
+ `fluo.accumulo.*` properties determine which Accumulo instance is used. For performance reasons,
+ Fluo runs its own code in Accumulo tablet servers. Fluo attempts to copy Fluo jars into DFS and
+ configure Accumulo to use them. Fluo first checks the property `fluo.accumulo.jars` and if set,
+ copies the jars listed there. If that property is not set, then Fluo looks on the classpath to
+ find jars. Jars are copied to a location under `<fluo.dfs.root>/<app name>`.
+
+## Instructions
+
+Below are the steps to initialize an application from the command line. It is also possible to
+initialize an application using Fluo's Java API.
+
+1. Create a copy of [fluo-app.properties] for your Fluo application.
+
+ cp $FLUO_HOME/conf/fluo-app.properties /path/to/myapp/fluo-app.properties
+
+2. Edit your copy of [fluo-app.properties] and make sure to set the following:
+
+ * Class name of your ObserverProvider
+ * Paths to your Fluo observer jars
+ * Accumulo configuration
+ * DFS configuration
+
+ When configuring the observer section of fluo-app.properties, you can configure your instance for the
+ [phrasecount] application if you have not created your own application. See the [phrasecount]
+ example for instructions. You can also choose not to configure any observers but your workers will
+ be idle when started.
+
+3. Run the command below to initialize your Fluo application. Change `myapp` to your application name:
+
+ fluo init myapp /path/to/myapp/fluo-app.properties
+
+ A Fluo application only needs to be initialized once. After initialization, the Fluo application
+ name is used to start/stop the application and scan the Fluo table.
+
+4. Run `fluo list` which connects to Fluo and lists applications to verify initialization.
+
+5. Run `fluo config myapp` to see what configuration is stored in ZooKeeper.
+
+## Next Steps
+
+Run your Fluo application using one of the methods below:
+
+* [Run Fluo processes][process]
+* [Run Fluo in YARN][yarn]
+* [Run Fluo in Docker][docker] which enables running in Mesos and Kubernetes
+
+Also, learn how to [manage Fluo applications][manage].
+
+[fluo-app.properties]: {{ page.github_base}}/modules/distribution/src/main/config/fluo-app.properties
+[fluo-conn.properties]: {{ page.github_base}}/modules/distribution/src/main/config/fluo-conn.properties
+[phrasecount]: https://github.com/astralway/phrasecount
+[process]: {{ page.docs_base }}/administration/run-fluo-processes/
+[yarn]: {{ page.docs_base }}/administration/run-fluo-in-yarn/
+[docker]: {{ page.docs_base }}/administration/run-fluo-in-docker/
+[manage]: {{ page.docs_base }}/administration/manage-applications/
diff --git a/_fluo-1-2/administration/manage-applications.md b/_fluo-1-2/administration/manage-applications.md
new file mode 100644
index 0000000..5e3fcce
--- /dev/null
+++ b/_fluo-1-2/administration/manage-applications.md
@@ -0,0 +1,107 @@
+---
+title: Manage Applications
+category: administration
+order: 5
+---
+
+This page contains information on managing Fluo applications.
+
+## Viewing data
+
+When you have data in your Fluo application, you can view it using the command `fluo scan myapp`.
+Pipe the output to `less` using the command `fluo scan myapp | less` if you want to page through the data.
+
+## Get status of all applications
+
+To list all Fluo applications, run `fluo list`.
+
+## Running application code
+
+The `fluo exec <app name> <class> {arguments}` provides an easy way to execute application code. It
+will execute a class with a main method if a jar containing the class is included with the observer
+jars configured at initialization. When the class is run, Fluo classes and dependencies will be on
+the classpath. The `fluo exec` command can inject the applications configuration if the class is
+written in the following way. Defining the injection point is optional.
+
+```java
+import javax.inject.Inject;
+
+public class AppCommand {
+
+ //when run with fluo exec command, the applications configuration will be injected
+ @Inject
+ private static FluoConfiguration fluoConfig;
+
+ public static void main(String[] args) throws Exception {
+ try(FluoClient fluoClient = FluoFactory.newClient(fluoConfig)) {
+ //do stuff with Fluo
+ }
+ }
+}
+```
+
+## Application Configuration
+
+For configuring observers, fluo provides a simple mechanism to set and access application specific
+configuration. See the javadoc on [FluoClient].getAppConfiguration() for more details.
+
+## Debugging Applications
+
+While monitoring [Fluo metrics][metrics] can detect problems (like too many transaction collisions)
+in a Fluo application, [metrics][metrics] may not provide enough information to debug the root cause
+of the problem. To help debug Fluo applications, low-level logging of transactions can be turned on
+by setting the following loggers to TRACE:
+
+| Logger | Level | Information |
+|----------------------|-------|----------------------------------------------------------------------------------------------------|
+| fluo.tx | TRACE | Provides detailed information about what transactions read and wrote |
+| fluo.tx.summary | TRACE | Provides a one line summary about each transaction executed |
+| fluo.tx.collisions | TRACE | Provides details about what data was involved When a transaction collides with another transaction |
+| fluo.tx.scan | TRACE | Provides logging for each cell read by a scan. Scan summary logged at `fluo.tx` level. This allows suppression of `fluo.tx.scan` while still seeing summary. |
+
+Below is an example log after setting `fluo.tx` to TRACE. The number following `txid: ` is the
+transactions start timestamp from the Oracle.
+
+```
+2015-02-11 18:24:05,341 [fluo.tx ] TRACE: txid: 3 begin() thread: 198
+2015-02-11 18:24:05,343 [fluo.tx ] TRACE: txid: 3 class: com.SimpleLoader
+2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 3 get(4333, stat count ) -> null
+2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 3 set(4333, stat count , 1)
+2015-02-11 18:24:05,441 [fluo.tx ] TRACE: txid: 3 commit() -> SUCCESSFUL commitTs: 4
+2015-02-11 18:24:05,341 [fluo.tx ] TRACE: txid: 5 begin() thread: 198
+2015-02-11 18:24:05,442 [fluo.tx ] TRACE: txid: 3 close()
+2015-02-11 18:24:05,343 [fluo.tx ] TRACE: txid: 5 class: com.SimpleLoader
+2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 5 get(4333, stat count ) -> 1
+2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 5 set(4333, stat count , 2)
+2015-02-11 18:24:05,441 [fluo.tx ] TRACE: txid: 5 commit() -> SUCCESSFUL commitTs: 6
+2015-02-11 18:24:05,442 [fluo.tx ] TRACE: txid: 5 close()
+```
+
+The log above traces the following sequence of events.
+
+* Transaction T1 has a start timestamp of `3`
+* Thread with id `198` is executing T1, its running code from the class `com.SimpleLoader`
+* T1 reads row `4333` and column `stat count` which does not exist
+* T1 sets row `4333` and column `stat count` to `1`
+* T1 commits successfully and its commit timestamp from the Oracle is `4`.
+* Transaction T2 has a start timestamp of `5` (because its `5` > `4` it can see what T1 wrote).
+* T2 reads a value of `1` for row `4333` and column `stat count`
+* T2 sets row `4333` and column `stat count` to `2`
+* T2 commits successfully with a commit timestamp of `6`
+
+Below is an example log after only setting `fluo.tx.collisions` to TRACE. This setting will only log
+trace information when a collision occurs. Unlike the previous example, what the transaction read
+and wrote is not logged. This shows that a transaction with a start timestamp of `106` and a class
+name of `com.SimpleLoader` collided with another transaction on row `r1` and column `fam1 qual1`.
+
+```
+2015-02-11 18:17:02,639 [tx.collisions] TRACE: txid: 106 class: com.SimpleLoader
+2015-02-11 18:17:02,639 [tx.collisions] TRACE: txid: 106 collisions: {r1=[fam1 qual1 ]}
+```
+
+When applications read and write arbitrary binary data, this does not log so well. In order to make
+the trace logs human readable, non ASCII chars are escaped using hex. The convention used it `\xDD`
+where D is a hex digit. Also the `\` character is escaped to make the output unambiguous.
+
+[FluoClient]: {{ page.javadoc_base}}/org/apache/fluo/api/client/FluoClient.html
+[metrics]: {{ page.docs_base }}/administration/metrics/
diff --git a/_fluo-1-2/administration/metrics.md b/_fluo-1-2/administration/metrics.md
index 9576ee6..4c7ed79 100644
--- a/_fluo-1-2/administration/metrics.md
+++ b/_fluo-1-2/administration/metrics.md
@@ -1,7 +1,7 @@
---
title: Metrics
category: administration
-order: 1
+order: 6
---
A Fluo application can be configured (in [fluo-app.properties]) to report metrics. When metrics are
diff --git a/_fluo-1-2/administration/run-fluo-in-docker.md b/_fluo-1-2/administration/run-fluo-in-docker.md
new file mode 100644
index 0000000..361a836
--- /dev/null
+++ b/_fluo-1-2/administration/run-fluo-in-docker.md
@@ -0,0 +1,117 @@
+---
+title: Run Fluo in Docker
+category: administration
+order: 4
+---
+
+## Obtain the Docker image
+
+To obtain the docker image created by this project, you can either pull it from DockerHub at
+`apache/fluo` or build it yourself. To pull the image from DockerHub, run the command below:
+
+ docker pull apache/fluo
+
+While it is easier to pull from DockerHub, the image will default to the software versions below:
+
+| Software | Version |
+|-------------|---------|
+| [Fluo] | 1.2.0 |
+| [Accumulo] | 1.8.1 |
+| [Hadoop] | 2.7.3 |
+| [Zookeeper] | 3.4.9 |
+
+If these versions do not match what is running on your cluster, you should consider [building
+your own image][build-image] with matching versions.
+
+## Image basics
+
+The entrypoint for the Fluo docker image is the `fluo` script. While the primary use
+case for this image is to start an oracle or worker, you can run other commands in the
+`fluo` script to test out the image:
+
+```bash
+# No arguments prints Fluo command usage
+docker run apache/fluo
+# Print Fluo version
+docker run apache/fluo version
+# Print Fluo classpath
+docker run apache/fluo classpath
+```
+
+## Run application in Docker
+
+Before starting a Fluo oracle and worker using the Fluo Docker image, [initialize your Fluo application][initialize].
+Next, choose a method below to run the oracle and worker(s) of your Fluo application. In the examples below, the Fluo
+application is named `myapp` and was initialized using a Zookeeper node on `zkhost`.
+
+### Docker engine
+
+Use the `docker` command to start local docker containers.
+
+1. Start a Fluo oracle for `myapp` application connecting to a Zookeeper at `zkhost`.
+
+ docker run -d --network="host" apache/fluo oracle -a myapp -o fluo.connection.zookeepers=zkhost/fluo
+
+ The command above uses `-d` to run the container in the background and `--network="host"` to run the container
+ on the host's network stack.
+
+2. Start Fluo worker(s). Execute this command multiple times to start multiple workers.
+
+ docker run -d --network="host" apache/fluo worker -a myapp -o fluo.connection.zookeepers=zkhost/fluo
+
+### Marathon
+
+Using the [Marathon] UI, you can create applications using JSON configuration.
+
+The JSON below can be used to start a Fluo oracle.
+
+```json
+{
+ "id": "myapp-fluo-oracle",
+ "cmd": "fluo oracle -a myapp -o fluo.connection.zookeepers=zkhost/fluo",
+ "cpus": 1,
+ "mem": 256,
+ "disk": 0,
+ "instances": 1,
+ "container": {
+ "docker": {
+ "image": "apache/fluo",
+ "network": "HOST"
+ },
+ "type": "DOCKER"
+ }
+}
+```
+
+The JSON below can be used to start Fluo worker(s). Modify instances to start multiple workers.
+
+```json
+{
+ "id": "myapp-fluo-worker",
+ "cmd": "fluo worker -a myapp -o fluo.connection.zookeepers=zkhost/fluo",
+ "cpus": 1,
+ "mem": 512,
+ "disk": 0,
+ "instances": 1,
+ "container": {
+ "docker": {
+ "image": "apache/fluo",
+ "network": "HOST"
+ },
+ "type": "DOCKER"
+ }
+}
+```
+
+## Next Steps
+
+Learn how to [manage Fluo applications][manage].
+
+[build-image]: https://github.com/apache/fluo-docker/blob/master/README.md#build-the-docker-image
+[manage]: {{ page.docs_base }}/administration/manage-applications/
+[Fluo]: https://fluo.apache.org/
+[Accumulo]: https://accumulo.apache.org/
+[Hadoop]: https://hadoop.apache.org/
+[Zookeeper]: https://zookeeper.apache.org/
+[intialize]: {{ page.docs_base }}/administration/initialize/
+[Marathon]: https://mesosphere.github.io/marathon/
diff --git a/_fluo-1-2/administration/run-fluo-in-yarn.md b/_fluo-1-2/administration/run-fluo-in-yarn.md
new file mode 100644
index 0000000..eb71b90
--- /dev/null
+++ b/_fluo-1-2/administration/run-fluo-in-yarn.md
@@ -0,0 +1,101 @@
+---
+title: Run Fluo in YARN
+category: administration
+order: 3
+---
+
+An Apache Fluo application can be started in Hadoop YARN using the Fluo YARN launcher.
+
+## Requirements
+
+The Fluo YARN launcher requires [Apache Fluo][Fluo] and [Hadoop YARN][YARN] to be installed:
+
+| Software | Recommended Version | Minimum Version |
+|-------------|---------------------|-----------------|
+| [Fluo] | 1.2.0 | 1.2.0 |
+| [YARN] | 2.7.2 | 2.6.0 |
+
+See the [related projects page][related] for external projects that may help in setting up these dependencies.
+
+## Set up your Fluo application
+
+Before you can launch a Fluo application in YARN, you should [install Fluo][install] and [initialize your
+application][initialize]. After your application has been initialized, follow the instructions below to install
+the Fluo YARN launcher and run your application in YARN. Avoid using the `fluo` command to start local oracle
+and worker processes if you are running in YARN.
+
+## Install and Configure Fluo YARN launcher
+
+Before you can install the Fluo YARN launcher, you will need to obtain a distribution tarball. It is
+recommended that you download the [latest release][release]. You can also build a distribution from the
+master branch by following these steps which create a tarball in `distribution/target`:
+
+ git clone https://github.com/apache/fluo-yarn.git
+ cd fluo-yarn/
+ mvn package
+
+After you obtain a Fluo YARN distribution tarball, follow these steps to install Fluo.
+
+1. Choose a directory with plenty of space, untar the distribution, and run `fetch.sh` to retrieve dependencies:
+
+ tar -xvzf fluo-yarn-1.0.0-bin.tar.gz
+ cd fluo-yarn-1.0.0
+ ./lib/fetch.sh
+
+ The distribution contains a `fluo-yarn` script in `bin/` that administers Fluo and the
+ following configuration files in `conf/`:
+
+ | Configuration file | Description |
+ |-----------------------------|-------------------------------------------------------------------------|
+ | [fluo-yarn-env.sh] | Configures classpath for `fluo-yarn` script. Required for all commands. |
+ | [fluo-yarn.properties] | Configures how application runs in YARN. Required for `start` command. |
+ | [log4j.properties] | Configures logging |
+
+2. Configure [fluo-yarn-env.sh]
+
+ * Set `FLUO_HOME` if it is not in your environment
+ * Modify `FLUO_CONN_PROPS` if you don't want use the default.
+
+3. Configure [fluo-yarn.properties] to set how your application will be launched in YARN:
+
+ * YARN resource manager hostname
+ * Number of oracle and worker instances
+ * Max memory usage per oracle/worker
+
+ If you are managing multiple Fluo applications in YARN, you can copy this file and configure it for
+ each application.
+
+## Start Fluo application in YARN
+
+Follow the instructions below to start your application in YARN. If you have not done so already, you should [initialize
+your Fluo application][initialize] before following these instructions.
+
+1. Configure [fluo-yarn-env.sh] and [fluo-yarn.properties] if you have not already.
+
+2. Run the commands below to start your Fluo application in YARN.
+
+ fluo-yarn start myapp conf/fluo-yarn.properties
+
+ The commands will retrieve your application configuration and observer jars (using your application name) before
+ starting the application in YARN. The command will output a YARN application ID that can be used to find your
+ application in the YARN resource manager and view its logs.
+
+## Manage Fluo application in YARN
+
+Except for stopping your application in YARN, the `fluo` script can be used to [manage your application][manage] using the
+`scan` and `wait` commands.
+
+When you want you stop your Fluo application, use the the YARN resource manager or the
+`yarn application -kill <App ID>` to stop the application in YARN.
+
+[Fluo]: https://fluo.apache.org/
+[YARN]: https://hadoop.apache.org/
+[related]: /related-projects/
+[install]: {{ page.docs_base }}/getting-started/install-fluo/
+[application]: {{ page.docs_base }}/getting-started/create-application/
+[initialize]: {{ page.docs_base }}/administration/initialize/
+[manage]: {{ page.docs_base }}/administration/manage-applications/
+[release]: /download/
+[fluo-yarn-env.sh]: {{ page.github_yarn}}/distribution/conf/fluo-yarn-env.sh
+[fluo-yarn.properties]: {{ page.github_yarn}}/distribution/conf/fluo-yarn.properties
+[log4j.properties]: {{ page.github_yarn}}/distribution/conf/log4j.properties
diff --git a/_fluo-1-2/administration/run-fluo-processes.md b/_fluo-1-2/administration/run-fluo-processes.md
new file mode 100644
index 0000000..0f100ee
--- /dev/null
+++ b/_fluo-1-2/administration/run-fluo-processes.md
@@ -0,0 +1,46 @@
+---
+title: Run Fluo processes
+category: administration
+order: 2
+---
+
+An Apache Fluo application consists of an oracle process and multiple worker processes. These processes
+can be manually started by users on a single node or distributed across a cluster.
+
+## Requirements
+
+Each node where Fluo is run must have [Fluo installed][install-fluo].
+
+## Start Fluo processes
+
+Follow the instructions below to start Fluo processes.
+
+1. Configure [fluo-env.sh] and [fluo-conn.properties] if you have not already.
+
+2. Run Fluo application processes using the `fluo oracle` and `fluo worker` commands. Fluo applications
+ are typically run with one oracle process and multiple worker processes. The commands below will start
+ a Fluo oracle and two workers on your local machine:
+
+ fluo oracle myapp &> oracle.log &
+ fluo worker myapp &> worker1.log &
+ fluo worker myapp &> worker2.log &
+
+ The commands will retrieve your application configuration and observer jars (using your
+ application name) before starting the oracle or worker process.
+
+If you want to distribute the processes of your Fluo application across a cluster, you will need install
+Fluo on every node where you want to run a Fluo process. It is recommended that you use a cluster management
+tool like Salt, Ansible, or Chef to install and run Fluo on a cluster.
+
+## Stop Fluo processes
+
+To stop your Fluo application, run `jps -m | grep Fluo` to find process IDs and use `kill` to stop them.
+
+## Next Steps
+
+Learn how to [manage Fluo applications][manage].
+
+[install-fluo]: {{ page.docs_base }}/getting-started/install-fluo/
+[manage]: {{ page.docs_base }}/administration/manage-applications/
+[fluo-env.sh]: {{ page.github_base}}/modules/distribution/src/main/config/fluo-env.sh
+[fluo-conn.properties]: {{ page.github_base}}/modules/distribution/src/main/config/fluo-conn.properties
diff --git a/_fluo-1-2/development/applications.md b/_fluo-1-2/development/applications.md
deleted file mode 100644
index 4f554f7..0000000
--- a/_fluo-1-2/development/applications.md
+++ /dev/null
@@ -1,314 +0,0 @@
----
-title: Applications
-category: development
-order: 1
----
-
-Once you have Fluo installed and running on your cluster, you can run Fluo applications consisting
-of [clients and observers][design]. This documentations will shows how to :
-
- * Create a Fluo client
- * Create a Fluo observer
- * Initialize a Fluo Application
- * Start and stop a Fluo application (which consists of Oracle and Worker processes)
-
-## Fluo Maven Dependencies
-
-For both clients and observers, you will need to include the following in your Maven pom:
-
-```xml
-<dependency>
- <groupId>org.apache.fluo</groupId>
- <artifactId>fluo-api</artifactId>
- <version>{{ page.version }}</version>
-</dependency>
-<dependency>
- <groupId>org.apache.fluo</groupId>
- <artifactId>fluo-core</artifactId>
- <version>{{ page.version }}</version>
- <scope>runtime</scope>
-</dependency>
-```
-
-Fluo provides a classpath command to help users build a runtime classpath. This command along with
-the `hadoop jar` command is useful when writing scripts to run Fluo client code. These commands
-allow the scripts to use the versions of Hadoop, Accumulo, and Zookeeper installed on a cluster.
-
-## Creating a Fluo client
-
-To create a [FluoClient], you will need to provide it with a [FluoConfiguration] object that is
-configured to connect to your Fluo instance.
-
-If you have access to the [fluo-conn.properties] file that was used to configure your Fluo instance, you
-can use it to build a [FluoConfiguration] object with all necessary properties:
-
-```java
-FluoConfiguration config = new FluoConfiguration(new File("fluo-conn.properties"));
-config.setApplicationName("myapp");
-```
-
-You can also create an empty [FluoConfiguration] object and set properties using Java:
-
-```java
-FluoConfiguration config = new FluoConfiguration();
-config.setInstanceZookeepers("localhost/fluo");
-config.setApplicationName("myapp");
-```
-
-Once you have [FluoConfiguration] object, pass it to the `newClient()` method of [FluoFactory] to
-create a [FluoClient]:
-
-```java
-try(FluoClient client = FluoFactory.newClient(config)){
-
- try (Transaction tx = client.newTransaction()) {
- // read and write some data
- tx.commit();
- }
-
- try (Snapshot snapshot = client.newSnapshot()) {
- //read some data
- }
-}
-```
-
-It may help to reference the [API javadocs][API] while you are learning the Fluo API.
-
-## Creating a Fluo observer
-
-To create an observer, follow these steps:
-
-1. Create one or more classes that extend [Observer] like the example below. Please use [slf4j] for
- any logging in observers as [slf4j] supports multiple logging implementations. This is
- necessary as Fluo applications have a hard requirement on [logback] when running in YARN.
-
- ```java
- public class InvertObserver implements Observer {
-
- @Override
- public void process(TransactionBase tx, Bytes row, Column col) throws Exception {
- // read value
- Bytes value = tx.get(row, col);
- // invert row and value
- tx.set(value, new Column("inv", "data"), row);
- }
- }
- ```
-
-2. Create a class that implements [ObserverProvider] like the example below. The purpose of this
- class is associate a set Observers with columns that trigger the observers. The class can
- register multiple observers.
-
- ```java
- class AppObserverProvider implements ObserverProvider {
- @Override
- public void provide(Registry or, Context ctx) {
- //setup InvertObserver to be triggered when the column obs:data is modified
- or.forColumn(new Column("obs", "data"), NotificationType.STRONG)
- .useObserver(new InvertObserver());
-
- //Observer is a Functional interface. So Observers can be written as lambdas.
- or.forColumn(new Column("new","data"), NotificationType.WEAK)
- .useObserver((tx,row,col) -> {
- Bytes combined = combineNewAndOld(tx,row);
- tx.set(row, new Column("current","data"), combined);
- });
- }
- }
- ```
-
-3. Build a jar containing these classes and include this jar in the `lib/` directory of your Fluo
- application.
-4. Configure your Fluo application to use this observer provider by modifying the Application section of
- [fluo-app.properties]. Set `fluo.observer.provider` to the observer provider class name.
-5. Initialize your Fluo application as described in the next section. During initialization Fluo
- will obtain the observed columns from the ObserverProvider and persist the columns in Zookeeper.
- These columns persisted in Zookeeper are used by transactions to know when to trigger observers.
-
-## Initializing a Fluo Application
-
-Before a Fluo Application can run, it must be initiaized. Below is an overview of what
-initialization does and some of the properties that may be set for initialization.
-
- * **Initialize ZooKeeper** : Each application has its own area in ZooKeeper used for configuration,
- Oracle state, and worker coordination. All properties, except `fluo.connections.*`, are copied
- into ZooKeeper. For example, if `fluo.worker.num.threads=128` was set, then when a worker process
- starts it will read this from ZooKeeper.
- * **Copy Observer jars to DFS** : Fluo workers processes need the jars containing observers. These
- are provided in one of the following ways.
- * Set the property `fluo.observer.init.dir` to a local directory containing observer jars. The
- jars in this directory are copied to DFS under `<fluo.dfs.root>/<app name>`. When a worker is
- started, the jars are pulled from DFS and added to its classpath.
- * Set the property `fluo.observer.jars.url` to a directory in DFS containing observer jars. No
- copying is done. When a worker is started, the jars are pulled from this location and added to
- its classpath.
- * Do not set any of the properties above and have the mechanism that starts the worker process
- add the needed jars to the classpath.
- * **Create Accumulo table** : Each Fluo application creates and configures an Accumulo table. The
- `fluo.accumulo.*` properties determine which Accumulo instance is used. For performance reasons,
- Fluo runs its own code in Accumulo tablet servers. Fluo attempts to copy Fluo jars into DFS and
- configure Accumulo to use them. Fluo first checks the property `fluo.accumulo.jars` and if set,
- copies the jars listed there. If that property is not set, then Fluo looks on the classpath to
- find jars. Jars are copied to a location under `<fluo.dfs.root>/<app name>`.
-
-Below are the steps to initialize an application from the command line. It is also possible to
-initialize an application using Fluo's Java API.
-
-1. Create a copy of [fluo-app.properties] for your Fluo application.
-
- cp $FLUO_HOME/conf/fluo-app.properties /path/to/myapp/fluo-app.properties
-
-2. Edit your copy of [fluo-app.properties] and make sure to set the following:
-
- * Class name of your ObserverProvider
- * Paths to your Fluo observer jars
- * Accumulo configuration
- * DFS configuration
-
- When configuring the observer section of fluo-app.properties, you can configure your instance for the
- [phrasecount] application if you have not created your own application. See the [phrasecount]
- example for instructions. You can also choose not to configure any observers but your workers will
- be idle when started.
-
-3. Run the command below to initialize your Fluo application. Change `myapp` to your application name:
-
- fluo init myapp /path/to/myapp/fluo-app.properties
-
- A Fluo application only needs to be initialized once. After initialization, the Fluo application
- name is used to start/stop the application and scan the Fluo table.
-
-4. Run `fluo list` which connects to Fluo and lists applications to verify initialization.
-
-5. Run `fluo config myapp` to see what configuration is stored in ZooKeeper.
-
-## Starting your Fluo application
-
-Follow the instructions below to start a Fluo application which contains an oracle and multiple workers.
-
-1. Configure [fluo-env.sh] and [fluo-conn.properties] if you have not already.
-
-2. Run Fluo application processes using the `fluo oracle` and `fluo worker` commands. Fluo applications
- are typically run with one oracle process and multiple worker processes. The commands below will start
- a Fluo oracle and two workers on your local machine:
-
- fluo oracle myapp &> oracle.log &
- fluo worker myapp &> worker1.log &
- fluo worker myapp &> worker2.log &
-
- The commands will retrieve your application configuration and observer jars (using your
- application name) before starting the oracle or worker process.
-
-If you want to distribute the processes of your Fluo application across a cluster, you will need install
-Fluo on every node where you want to run a Fluo process and follow the instructions above on each node.
-
-## Managing your Fluo application
-
-When you have data in your Fluo application, you can view it using the command `fluo scan myapp`.
-Pipe the output to `less` using the command `fluo scan myapp | less` if you want to page through the data.
-
-To list all Fluo applications, run `fluo list`.
-
-To stop your Fluo application, run `jps -m | grep Fluo` to find process IDs and use `kill` to stop them.
-
-## Running application code
-
-The `fluo exec <app name> <class> {arguments}` provides an easy way to execute application code. It
-will execute a class with a main method if a jar containing the class is included with the observer
-jars configured at initialization. When the class is run, Fluo classes and dependencies will be on
-the classpath. The `fluo exec` command can inject the applications configuration if the class is
-written in the following way. Defining the injection point is optional.
-
-```java
-import javax.inject.Inject;
-
-public class AppCommand {
-
- //when run with fluo exec command, the applications configuration will be injected
- @Inject
- private static FluoConfiguration fluoConfig;
-
- public static void main(String[] args) throws Exception {
- try(FluoClient fluoClient = FluoFactory.newClient(fluoConfig)) {
- //do stuff with Fluo
- }
- }
-}
-```
-
-## Application Configuration
-
-For configuring observers, fluo provides a simple mechanism to set and access application specific
-configuration. See the javadoc on [FluoClient].getAppConfiguration() for more details.
-
-## Debugging Applications
-
-While monitoring [Fluo metrics][metrics] can detect problems (like too many transaction collisions)
-in a Fluo application, [metrics][metrics] may not provide enough information to debug the root cause
-of the problem. To help debug Fluo applications, low-level logging of transactions can be turned on
-by setting the following loggers to TRACE:
-
-| Logger | Level | Information |
-|----------------------|-------|----------------------------------------------------------------------------------------------------|
-| fluo.tx | TRACE | Provides detailed information about what transactions read and wrote |
-| fluo.tx.summary | TRACE | Provides a one line summary about each transaction executed |
-| fluo.tx.collisions | TRACE | Provides details about what data was involved When a transaction collides with another transaction |
-| fluo.tx.scan | TRACE | Provides logging for each cell read by a scan. Scan summary logged at `fluo.tx` level. This allows suppression of `fluo.tx.scan` while still seeing summary. |
-
-Below is an example log after setting `fluo.tx` to TRACE. The number following `txid: ` is the
-transactions start timestamp from the Oracle.
-
-```
-2015-02-11 18:24:05,341 [fluo.tx ] TRACE: txid: 3 begin() thread: 198
-2015-02-11 18:24:05,343 [fluo.tx ] TRACE: txid: 3 class: com.SimpleLoader
-2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 3 get(4333, stat count ) -> null
-2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 3 set(4333, stat count , 1)
-2015-02-11 18:24:05,441 [fluo.tx ] TRACE: txid: 3 commit() -> SUCCESSFUL commitTs: 4
-2015-02-11 18:24:05,341 [fluo.tx ] TRACE: txid: 5 begin() thread: 198
-2015-02-11 18:24:05,442 [fluo.tx ] TRACE: txid: 3 close()
-2015-02-11 18:24:05,343 [fluo.tx ] TRACE: txid: 5 class: com.SimpleLoader
-2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 5 get(4333, stat count ) -> 1
-2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 5 set(4333, stat count , 2)
-2015-02-11 18:24:05,441 [fluo.tx ] TRACE: txid: 5 commit() -> SUCCESSFUL commitTs: 6
-2015-02-11 18:24:05,442 [fluo.tx ] TRACE: txid: 5 close()
-```
-
-The log above traces the following sequence of events.
-
-* Transaction T1 has a start timestamp of `3`
-* Thread with id `198` is executing T1, its running code from the class `com.SimpleLoader`
-* T1 reads row `4333` and column `stat count` which does not exist
-* T1 sets row `4333` and column `stat count` to `1`
-* T1 commits successfully and its commit timestamp from the Oracle is `4`.
-* Transaction T2 has a start timestamp of `5` (because its `5` > `4` it can see what T1 wrote).
-* T2 reads a value of `1` for row `4333` and column `stat count`
-* T2 sets row `4333` and column `stat count` to `2`
-* T2 commits successfully with a commit timestamp of `6`
-
-Below is an example log after only setting `fluo.tx.collisions` to TRACE. This setting will only log
-trace information when a collision occurs. Unlike the previous example, what the transaction read
-and wrote is not logged. This shows that a transaction with a start timestamp of `106` and a class
-name of `com.SimpleLoader` collided with another transaction on row `r1` and column `fam1 qual1`.
-
-```
-2015-02-11 18:17:02,639 [tx.collisions] TRACE: txid: 106 class: com.SimpleLoader
-2015-02-11 18:17:02,639 [tx.collisions] TRACE: txid: 106 collisions: {r1=[fam1 qual1 ]}
-```
-
-When applications read and write arbitrary binary data, this does not log so well. In order to make
-the trace logs human readable, non ASCII chars are escaped using hex. The convention used it `\xDD`
-where D is a hex digit. Also the `\` character is escaped to make the output unambiguous.
-
-[design]: {{ page.docs_base }}/getting-started/design/
-[FluoFactory]: {{ page.javadoc_base}}/org/apache/fluo/api/client/FluoFactory.html
-[FluoClient]: {{ page.javadoc_base}}/org/apache/fluo/api/client/FluoClient.html
-[FluoConfiguration]: {{ page.javadoc_base}}/org/apache/fluo/api/config/FluoConfiguration.html
-[Observer]: {{ page.javadoc_base}}/org/apache/fluo/api/observer/Observer.html
-[ObserverProvider]: {{ page.javadoc_base}}/org/apache/fluo/api/observer/ObserverProvider.html
-[fluo-conn.properties]: {{ page.github_base}}/modules/distribution/src/main/config/fluo-conn.properties
-[fluo-app.properties]: {{ page.github_base}}/modules/distribution/src/main/config/fluo-app.properties
-[API]: https://fluo.apache.org/apidocs/
-[metrics]: {{ page.docs_base }}/administration/metrics/
-[slf4j]: http://www.slf4j.org/
-[logback]: http://logback.qos.ch/
-[phrasecount]: https://github.com/astralway/phrasecount
-[fluo-env.sh]: {{ page.github_base}}/modules/distribution/src/main/config/fluo-env.sh
diff --git a/_fluo-1-2/getting-started/create-application.md b/_fluo-1-2/getting-started/create-application.md
new file mode 100644
index 0000000..baae343
--- /dev/null
+++ b/_fluo-1-2/getting-started/create-application.md
@@ -0,0 +1,139 @@
+---
+title: Create Application
+category: getting-started
+order: 4
+---
+
+Once you have [Fluo installed][install-fluo], you can create and run Fluo applications consisting
+of [clients and observers][design]. This documentation shows how to create a Fluo client and observer.
+
+## Fluo Maven Dependencies
+
+For both clients and observers, you will need to include the following in your Maven pom:
+
+```xml
+<dependency>
+ <groupId>org.apache.fluo</groupId>
+ <artifactId>fluo-api</artifactId>
+ <version>{{ page.version }}</version>
+</dependency>
+<dependency>
+ <groupId>org.apache.fluo</groupId>
+ <artifactId>fluo-core</artifactId>
+ <version>{{ page.version }}</version>
+ <scope>runtime</scope>
+</dependency>
+```
+
+Fluo provides a classpath command to help users build a runtime classpath. This command along with
+the `hadoop jar` command is useful when writing scripts to run Fluo client code. These commands
+allow the scripts to use the versions of Hadoop, Accumulo, and Zookeeper installed on a cluster.
+
+## Creating a Fluo client
+
+To create a [FluoClient], you will need to provide it with a [FluoConfiguration] object that is
+configured to connect to your Fluo instance.
+
+If you have access to the [fluo-conn.properties] file that was used to configure your Fluo instance, you
+can use it to build a [FluoConfiguration] object with all necessary properties:
+
+```java
+FluoConfiguration config = new FluoConfiguration(new File("fluo-conn.properties"));
+config.setApplicationName("myapp");
+```
+
+You can also create an empty [FluoConfiguration] object and set properties using Java:
+
+```java
+FluoConfiguration config = new FluoConfiguration();
+config.setInstanceZookeepers("localhost/fluo");
+config.setApplicationName("myapp");
+```
+
+Once you have [FluoConfiguration] object, pass it to the `newClient()` method of [FluoFactory] to
+create a [FluoClient]:
+
+```java
+try(FluoClient client = FluoFactory.newClient(config)){
+
+ try (Transaction tx = client.newTransaction()) {
+ // read and write some data
+ tx.commit();
+ }
+
+ try (Snapshot snapshot = client.newSnapshot()) {
+ //read some data
+ }
+}
+```
+
+It may help to reference the [API javadocs][API] while you are learning the Fluo API.
+
+## Creating a Fluo observer
+
+To create an observer, follow these steps:
+
+1. Create one or more classes that extend [Observer] like the example below. Please use [slf4j] for
+ any logging in observers as [slf4j] supports multiple logging implementations. This is
+ necessary as Fluo applications have a hard requirement on [logback] when running in YARN.
+
+ ```java
+ public class InvertObserver implements Observer {
+
+ @Override
+ public void process(TransactionBase tx, Bytes row, Column col) throws Exception {
+ // read value
+ Bytes value = tx.get(row, col);
+ // invert row and value
+ tx.set(value, new Column("inv", "data"), row);
+ }
+ }
+ ```
+
+2. Create a class that implements [ObserverProvider] like the example below. The purpose of this
+ class is associate a set Observers with columns that trigger the observers. The class can
+ register multiple observers.
+
+ ```java
+ class AppObserverProvider implements ObserverProvider {
+ @Override
+ public void provide(Registry or, Context ctx) {
+ //setup InvertObserver to be triggered when the column obs:data is modified
+ or.forColumn(new Column("obs", "data"), NotificationType.STRONG)
+ .useObserver(new InvertObserver());
+
+ //Observer is a Functional interface. So Observers can be written as lambdas.
+ or.forColumn(new Column("new","data"), NotificationType.WEAK)
+ .useObserver((tx,row,col) -> {
+ Bytes combined = combineNewAndOld(tx,row);
+ tx.set(row, new Column("current","data"), combined);
+ });
+ }
+ }
+ ```
+
+3. Build a jar containing these classes and include this jar in the `lib/` directory of your Fluo
+ application.
+4. Configure your Fluo application to use this observer provider by modifying the Application section of
+ [fluo-app.properties]. Set `fluo.observer.provider` to the observer provider class name.
+5. Initialize your Fluo application as described in the next section. During initialization Fluo
+ will obtain the observed columns from the ObserverProvider and persist the columns in Zookeeper.
+ These columns persisted in Zookeeper are used by transactions to know when to trigger observers.
+
+## Next Steps
+
+[Initialize] your Fluo application before running it.
+
+[design]: {{ page.docs_base }}/getting-started/design/
+[install-fluo]: {{ page.docs_base }}/getting-started/install-fluo/
+[Initialize]: {{ page.docs_base }}/administration/initialize/
+[FluoFactory]: {{ page.javadoc_base}}/org/apache/fluo/api/client/FluoFactory.html
+[FluoClient]: {{ page.javadoc_base}}/org/apache/fluo/api/client/FluoClient.html
+[FluoConfiguration]: {{ page.javadoc_base}}/org/apache/fluo/api/config/FluoConfiguration.html
+[Observer]: {{ page.javadoc_base}}/org/apache/fluo/api/observer/Observer.html
+[ObserverProvider]: {{ page.javadoc_base}}/org/apache/fluo/api/observer/ObserverProvider.html
+[fluo-conn.properties]: {{ page.github_base}}/modules/distribution/src/main/config/fluo-conn.properties
+[fluo-app.properties]: {{ page.github_base}}/modules/distribution/src/main/config/fluo-app.properties
+[API]: https://fluo.apache.org/apidocs/
+[slf4j]: http://www.slf4j.org/
+[logback]: http://logback.qos.ch/
diff --git a/_fluo-1-2/getting-started/install.md b/_fluo-1-2/getting-started/install-fluo.md
similarity index 97%
rename from _fluo-1-2/getting-started/install.md
rename to _fluo-1-2/getting-started/install-fluo.md
index c38441a..33466e3 100644
--- a/_fluo-1-2/getting-started/install.md
+++ b/_fluo-1-2/getting-started/install-fluo.md
@@ -1,7 +1,7 @@
---
-title: Installation
+title: Install Fluo
category: getting-started
-order: 2
+order: 3
---
Instructions for installing Apache Fluo and starting a Fluo application on a cluster where
@@ -105,8 +105,8 @@ Using at least Accumulo 1.6.1 is recommended because multiple performance bugs w
[Hadoop]: http://hadoop.apache.org/
[Zookeeper]: http://zookeeper.apache.org/
[Java]: http://openjdk.java.net/
-[related]: https://fluo.apache.org/related-projects/
-[release]: https://fluo.apache.org/download/
+[related]: /related-projects/
+[release]: /download/
[fluo-conn.properties]: {{ page.github_base }}/modules/distribution/src/main/config/fluo-conn.properties
[fluo-app.properties]: {{ page.github_base }}/modules/distribution/src/main/config/fluo-app.properties
[log4j.properties]: {{ page.github_base }}/modules/distribution/src/main/config/log4j.properties
diff --git a/_fluo-1-2/getting-started/quick-start.md b/_fluo-1-2/getting-started/quick-start.md
new file mode 100644
index 0000000..3037e91
--- /dev/null
+++ b/_fluo-1-2/getting-started/quick-start.md
@@ -0,0 +1,27 @@
+---
+title: Quick Start
+category: getting-started
+order: 2
+---
+
+Follow the steps below to run a Fluo application:
+
+1. [Install Fluo][install]
+
+2. [Create a Fluo application][create] or use [an example application][example].
+
+3. [Initialize your Fluo application][initialize]
+
+4. Pick a method below to run your Fluo application
+
+ * [Run Fluo processes][process]
+ * [Run Fluo in YARN][yarn]
+ * [Run Fluo in Docker][docker] which enables running in Mesos and Kubernetes
+
+[install]: {{ page.docs_base }}/getting-started/install-fluo/
+[create]: {{ page.docs_base }}/getting-started/create-application/
+[example]: /related-projects/
+[initialize]: {{ page.docs_base }}/administration/initialize/
+[process]: {{ page.docs_base }}/administration/run-fluo-processes/
+[yarn]: {{ page.docs_base }}/administration/run-fluo-in-yarn/
+[docker]: {{ page.docs_base }}/administration/run-fluo-in-docker/
diff --git a/_layouts/fluo-1.2.html b/_layouts/fluo-1.2.html
index 28e774b..1eaf114 100644
--- a/_layouts/fluo-1.2.html
+++ b/_layouts/fluo-1.2.html
@@ -37,7 +37,7 @@ layout: default
<p>Fluo {{ page.version }} documentation >> {{ page.category | capitalize | replace: "-", " " }} >> {{ page.title }}</p>
{% endif %}
- <div class="alert alert-danger" style="margin-bottom: 0px;" role="alert">This documentation is for a future release of Fluo! <a href="{{ site.baseurl }}/docs/fluo/{{ site.latest_fluo_release }}/">View documentation for the latest release</a>.</div>
+ <div class="alert alert-warning" style="margin-bottom: 0px;" role="alert">This documentation is for a future release of Apache Fluo! <a href="{{ site.baseurl }}/docs/fluo/{{ site.latest_fluo_release }}/">View documentation for the latest release</a>.</div>
{% unless page.nodoctitle %}
<div class="row">
--
To stop receiving notification emails like this one, please contact
['"commits@fluo.apache.org" <co...@fluo.apache.org>'].