You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by el...@apache.org on 2014/12/19 19:44:46 UTC
[5/8] accumulo git commit: Merge branch '1.6'
Merge branch '1.6'
Conflicts:
docs/src/main/latex/accumulo_user_manual/accumulo_user_manual.tex
docs/src/main/latex/accumulo_user_manual/chapters/administration.tex
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/06946905
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/06946905
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/06946905
Branch: refs/heads/master
Commit: 06946905c3dac58d27c82e2bf9aa4fb3e68dc33d
Parents: 4e83745 c7f4502
Author: Josh Elser <el...@apache.org>
Authored: Fri Dec 19 13:43:06 2014 -0500
Committer: Josh Elser <el...@apache.org>
Committed: Fri Dec 19 13:43:06 2014 -0500
----------------------------------------------------------------------
.../main/asciidoc/accumulo_user_manual.asciidoc | 2 +
.../main/asciidoc/chapters/administration.txt | 20 ++++++
.../main/asciidoc/chapters/implementation.txt | 75 ++++++++++++++++++++
3 files changed, 97 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/06946905/docs/src/main/asciidoc/accumulo_user_manual.asciidoc
----------------------------------------------------------------------
diff --cc docs/src/main/asciidoc/accumulo_user_manual.asciidoc
index b958c9d,0000000..ec8e538
mode 100644,000000..100644
--- a/docs/src/main/asciidoc/accumulo_user_manual.asciidoc
+++ b/docs/src/main/asciidoc/accumulo_user_manual.asciidoc
@@@ -1,62 -1,0 +1,64 @@@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+Apache Accumulo User Manual Version 1.6
+=======================================
+:author: Apache Accumulo Project
+:email: dev@accumulo.apache.org
+:toc2:
+:toclevels: 4
+:toc-title: Apache Accumulo 1.6
+:numbered:
+:website: http://accumulo.apache.org/
+
+image::accumulo-logo.png[]
+
+Copyright © 2011-2014 The Apache Software Foundation, Licensed under the Apache
+License, Version 2.0. Apache Accumulo, Accumulo, Apache, and the Apache
+Accumulo project logo are trademarks of the Apache Software Foundation.
+
+include::chapters/introduction.txt[]
+
+include::chapters/design.txt[]
+
+include::chapters/shell.txt[]
+
+include::chapters/clients.txt[]
+
+include::chapters/development_clients.txt[]
+
+include::chapters/table_configuration.txt[]
+
+include::chapters/table_design.txt[]
+
+include::chapters/high_speed_ingest.txt[]
+
+include::chapters/analytics.txt[]
+
+include::chapters/security.txt[]
+
+include::chapters/replication.txt[]
+
++include::chapters/implementation.txt[]
++
+include::chapters/administration.txt[]
+
+include::chapters/multivolume.txt[]
+
+include::chapters/troubleshooting.txt[]
+
+[appendix]
+include::appendices/config.txt[]
+
http://git-wip-us.apache.org/repos/asf/accumulo/blob/06946905/docs/src/main/asciidoc/chapters/administration.txt
----------------------------------------------------------------------
diff --cc docs/src/main/asciidoc/chapters/administration.txt
index 2044653,0000000..f21e7a9
mode 100644,000000..100644
--- a/docs/src/main/asciidoc/chapters/administration.txt
+++ b/docs/src/main/asciidoc/chapters/administration.txt
@@@ -1,633 -1,0 +1,653 @@@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+== Administration
+
+=== Hardware
+
+Because we are running essentially two or three systems simultaneously layered
+across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to
+consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have
+at least one core and 2 - 4 GB each.
+
+One core running HDFS can typically keep 2 to 4 disks busy, so each machine may
+typically have as little as 2 x 300GB disks and as much as 4 x 1TB or 2TB disks.
+
+It is possible to do with less than this, such as with 1u servers with 2 cores and 4GB
+each, but in this case it is recommended to only run up to two processes per
+machine -- i.e. DataNode and TabletServer or DataNode and MapReduce worker but
+not all three. The constraint here is having enough available heap space for all the
+processes on a machine.
+
+=== Network
+
+Accumulo communicates via remote procedure calls over TCP/IP for both passing
+data and control messages. In addition, Accumulo uses HDFS clients to
+communicate with HDFS. To achieve good ingest and query performance, sufficient
+network bandwidth must be available between any two machines.
+
+In addition to needing access to ports associated with HDFS and ZooKeeper, Accumulo will
+use the following default ports. Please make sure that they are open, or change
+their value in conf/accumulo-site.xml.
+
+.Accumulo default ports
+[width="75%",cols=">,^2,^2"]
+[options="header"]
+|====
+|Port | Description | Property Name
+|4445 | Shutdown Port (Accumulo MiniCluster) | n/a
+|4560 | Accumulo monitor (for centralized log display) | monitor.port.log4j
+|9997 | Tablet Server | tserver.port.client
+|9999 | Master Server | master.port.client
+|12234 | Accumulo Tracer | trace.port.client
+|42424 | Accumulo Proxy Server | n/a
+|50091 | Accumulo GC | gc.port.client
+|50095 | Accumulo HTTP monitor | monitor.port.client
+|10001 | Master Replication service | master.replication.coordinator.port
+|10002 | TabletServer Replication service | replication.receipt.service.port
+|====
+
+In addition, the user can provide +0+ and an ephemeral port will be chosen instead. This
+ephemeral port is likely to be unique and not already bound. Thus, configuring ports to
+use +0+ instead of an explicit value, should, in most cases, work around any issues of
+running multiple distinct Accumulo instances (or any other process which tries to use the
+same default ports) on the same hardware.
+
+=== Installation
+Choose a directory for the Accumulo installation. This directory will be referenced
+by the environment variable +$ACCUMULO_HOME+. Run the following:
+
+ $ tar xzf accumulo-1.6.0-bin.tar.gz # unpack to subdirectory
+ $ mv accumulo-1.6.0 $ACCUMULO_HOME # move to desired location
+
+Repeat this step at each machine within the cluster. Usually all machines have the
+same +$ACCUMULO_HOME+.
+
+=== Dependencies
+Accumulo requires HDFS and ZooKeeper to be configured and running
+before starting. Password-less SSH should be configured between at least the
+Accumulo master and TabletServer machines. It is also a good idea to run Network
+Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of
+sync, which can cause problems with automatically timestamped data.
+
+=== Configuration
+
+Accumulo is configured by editing several Shell and XML files found in
++$ACCUMULO_HOME/conf+. The structure closely resembles Hadoop's configuration
+files.
+
+==== Edit conf/accumulo-env.sh
+
+Accumulo needs to know where to find the software it depends on. Edit accumulo-env.sh
+and specify the following:
+
+. Enter the location of the installation directory of Accumulo for +$ACCUMULO_HOME+
+. Enter your system's Java home for +$JAVA_HOME+
+. Enter the location of Hadoop for +$HADOOP_PREFIX+
+. Choose a location for Accumulo logs and enter it for +$ACCUMULO_LOG_DIR+
+. Enter the location of ZooKeeper for +$ZOOKEEPER_HOME+
+
+By default Accumulo TabletServers are set to use 1GB of memory. You may change
+this by altering the value of +$ACCUMULO_TSERVER_OPTS+. Note the syntax is that of
+the Java JVM command line options. This value should be less than the physical
+memory of the machines running TabletServers.
+
+There are similar options for the master's memory usage and the garbage collector
+process. Reduce these if they exceed the physical RAM of your hardware and
+increase them, within the bounds of the physical RAM, if a process fails because of
+insufficient memory.
+
+Note that you will be specifying the Java heap space in accumulo-env.sh. You should
+make sure that the total heap space used for the Accumulo tserver and the Hadoop
+DataNode and TaskTracker is less than the available memory on each slave node in
+the cluster. On large clusters, it is recommended that the Accumulo master, Hadoop
+NameNode, secondary NameNode, and Hadoop JobTracker all be run on separate
+machines to allow them to use more heap space. If you are running these on the
+same machine on a small cluster, likewise make sure their heap space settings fit
+within the available memory.
+
+==== Native Map
+
+The tablet server uses a data structure called a MemTable to store sorted key/value
+pairs in memory when they are first received from the client. When a minor compaction
+occurs, this data structure is written to HDFS. The MemTable will default to using
+memory in the JVM but a JNI version, called the native map, can be used to significantly
+speed up performance by utilizing the memory space of the native operating system. The
+native map also avoids the performance implications brought on by garbage collection
+in the JVM by causing it to pause much less frequently.
+
+===== Building
+
+32-bit and 64-bit Linux and Mac OS X versions of the native map can be built
+from the Accumulo bin package by executing
++$ACCUMULO_HOME/bin/build_native_library.sh+. If your system's
+default compiler options are insufficient, you can add additional compiler
+options to the command line, such as options for the architecture. These will be
+passed to the Makefile in the environment variable +USERFLAGS+.
+
+Examples:
+
+. +$ACCUMULO_HOME/bin/build_native_library.sh+
+. +$ACCUMULO_HOME/bin/build_native_library.sh -m32+
+
+After building the native map from the source, you will find the artifact in
++$ACCUMULO_HOME/lib/native+. Upon starting up, the tablet server will look
+in this directory for the map library. If the file is renamed or moved from its
+target directory, the tablet server may not be able to find it. The system can
+also locate the native maps shared library by setting +LD_LIBRARY_PATH+
+(or +DYLD_LIBRARY_PATH+ on Mac OS X) in +$ACCUMULO_HOME/conf/accumulo-env.sh+.
+
+===== Configuration
+
+As mentioned, Accumulo will use the native libraries if they are found in the expected
+location and if it is not configured to ignore them. Using the native maps over JVM
+Maps nets a noticable improvement in ingest rates; however, certain configuration
+variables are important to modify when increasing the size of the native map.
+
+To adjust the size of the native map, increase the value of +tserver.memory.maps.max+.
+By default, the maximum size of the native map is 1GB. When increasing this value, it is
+also important to adjust the values of +table.compaction.minor.logs.threshold+ and
++tserver.walog.max.size+. +table.compaction.minor.logs.threshold+ is the maximum
+number of write-ahead log files that a tablet can reference before they will be automatically
+minor compacted. +tserver.walog.max.size+ is the maximum size of a write-ahead log.
+
+The maximum size of the native maps for a server should be less than the product
+of the write-ahead log maximum size and minor compaction threshold for log files:
+
++$table.compaction.minor.logs.threshold * $tserver.walog.max.size >= $tserver.memory.maps.max+
+
+This formula ensures that minor compactions won't be automatically triggered before the native
+maps can be completely saturated.
+
+Subsequently, when increasing the size of the write-ahead logs, it can also be important
+to increase the HDFS block size that Accumulo uses when creating the files for the write-ahead log.
+This is controlled via +tserver.wal.blocksize+. A basic recommendation is that when
++tserver.walog.max.size+ is larger than 2GB in size, set +tserver.wal.blocksize+ to 2GB.
+Increasing the block size to a value larger than 2GB can result in decreased write
+performance to the write-ahead log file which will slow ingest.
+
+==== Cluster Specification
+
+On the machine that will serve as the Accumulo master:
+
+. Write the IP address or domain name of the Accumulo Master to the +$ACCUMULO_HOME/conf/masters+ file.
+. Write the IP addresses or domain name of the machines that will be TabletServers in +$ACCUMULO_HOME/conf/slaves+, one per line.
+
+Note that if using domain names rather than IP addresses, DNS must be configured
+properly for all machines participating in the cluster. DNS can be a confusing source
+of errors.
+
+==== Accumulo Settings
+Specify appropriate values for the following settings in
++$ACCUMULO_HOME/conf/accumulo-site.xml+ :
+
+[source,xml]
+<property>
+ <name>instance.zookeeper.host</name>
+ <value>zooserver-one:2181,zooserver-two:2181</value>
+ <description>list of zookeeper servers</description>
+</property>
+
+This enables Accumulo to find ZooKeeper. Accumulo uses ZooKeeper to coordinate
+settings between processes and helps finalize TabletServer failure.
+
+[source,xml]
+<property>
+ <name>instance.secret</name>
+ <value>DEFAULT</value>
+</property>
+
+The instance needs a secret to enable secure communication between servers. Configure your
+secret and make sure that the +accumulo-site.xml+ file is not readable to other users.
+For alternatives to storing the +instance.secret+ in plaintext, please read the
++Sensitive Configuration Values+ section.
+
+Some settings can be modified via the Accumulo shell and take effect immediately, but
+some settings require a process restart to take effect. See the configuration documentation
+(available in the docs directory of the tarball and in <<configuration>>) for details.
+
+==== Deploy Configuration
+
+Copy the masters, slaves, accumulo-env.sh, and if necessary, accumulo-site.xml
+from the +$ACCUMULO_HOME/conf/+ directory on the master to all the machines
+specified in the slaves file.
+
+==== Sensitive Configuration Values
+
+Accumulo has a number of properties that can be specified via the accumulo-site.xml
+file which are sensitive in nature, instance.secret and trace.token.property.password
+are two common examples. Both of these properties, if compromised, have the ability
+to result in data being leaked to users who should not have access to that data.
+
+In Hadoop-2.6.0, a new CredentialProvider class was introduced which serves as a common
+implementation to abstract away the storage and retrieval of passwords from plaintext
+storage in configuration files. Any Property marked with the +Sensitive+ annotation
+is a candidate for use with these CredentialProviders. For version of Hadoop which lack
+these classes, the feature will just be unavailable for use.
+
+A comma separated list of CredentialProviders can be configured using the Accumulo Property
++general.security.credential.provider.paths+. Each configured URL will be consulted
+when the Configuration object for accumulo-site.xml is accessed.
+
+==== Using a JavaKeyStoreCredentialProvider for storage
+
+One of the implementations provided in Hadoop-2.6.0 is a Java KeyStore CredentialProvider.
+Each entry in the KeyStore is the Accumulo Property key name. For example, to store the
+\texttt{instance.secret}, the following command can be used:
+
+ hadoop credential create instance.secret --provider jceks://file/etc/accumulo/conf/accumulo.jceks
+
+The command will then prompt you to enter the secret to use and create a keystore in:
+
+ /etc/accumulo/conf/accumulo.jceks
+
+Then, accumulo-site.xml must be configured to use this KeyStore as a CredentialProvider:
+
+[source,xml]
+<property>
+ <name>general.security.credential.provider.paths</name>
+ <value>jceks://file/etc/accumulo/conf/accumulo.jceks</value>
+</property>
+
+This configuration will then transparently extract the +instance.secret+ from
+the configured KeyStore and alleviates a human readable storage of the sensitive
+property.
+
+A KeyStore can also be stored in HDFS, which will make the KeyStore readily available to
+all Accumulo servers. If the local filesystem is used, be aware that each Accumulo server
+will expect the KeyStore in the same location.
+
+==== Custom Table Tags
+
+Accumulo has the ability for users to add custom tags to tables. This allows
+applications to set application-level metadata about a table. These tags can be
+anything from a table description, administrator notes, date created, etc.
+This is done by naming and setting a property with a prefix +table.custom.*+.
+
+=== Initialization
+
+Accumulo must be initialized to create the structures it uses internally to locate
+data across the cluster. HDFS is required to be configured and running before
+Accumulo can be initialized.
+
+Once HDFS is started, initialization can be performed by executing
++$ACCUMULO_HOME/bin/accumulo init+ . This script will prompt for a name
+for this instance of Accumulo. The instance name is used to identify a set of tables
+and instance-specific settings. The script will then write some information into
+HDFS so Accumulo can start properly.
+
+The initialization script will prompt you to set a root password. Once Accumulo is
+initialized it can be started.
+
+=== Running
+
+==== Starting Accumulo
+
+Make sure Hadoop is configured on all of the machines in the cluster, including
+access to a shared HDFS instance. Make sure HDFS and ZooKeeper are running.
+Make sure ZooKeeper is configured and running on at least one machine in the
+cluster.
+Start Accumulo using the +bin/start-all.sh+ script.
+
+To verify that Accumulo is running, check the Status page as described in
+<<monitoring>>. In addition, the Shell can provide some information about the status of
+tables via reading the metadata tables.
+
+==== Stopping Accumulo
+
+To shutdown cleanly, run +bin/stop-all.sh+ and the master will orchestrate the
+shutdown of all the tablet servers. Shutdown waits for all minor compactions to finish, so it may
+take some time for particular configurations.
+
+==== Adding a Node
+
+Update your +$ACCUMULO_HOME/conf/slaves+ (or +$ACCUMULO_CONF_DIR/slaves+) file to account for the addition.
+
+ $ACCUMULO_HOME/bin/accumulo admin start <host(s)> {<host> ...}
+
+Alternatively, you can ssh to each of the hosts you want to add and run:
+
+ $ACCUMULO_HOME/bin/start-here.sh
+
+Make sure the host in question has the new configuration, or else the tablet
+server won't start; at a minimum this needs to be on the host(s) being added,
+but in practice it's good to ensure consistent configuration across all nodes.
+
+==== Decomissioning a Node
+
+If you need to take a node out of operation, you can trigger a graceful shutdown of a tablet
+server. Accumulo will automatically rebalance the tablets across the available tablet servers.
+
+ $ACCUMULO_HOME/bin/accumulo admin stop <host(s)> {<host> ...}
+
+Alternatively, you can ssh to each of the hosts you want to remove and run:
+
+ $ACCUMULO_HOME/bin/stop-here.sh
+
+Be sure to update your +$ACCUMULO_HOME/conf/slaves+ (or +$ACCUMULO_CONF_DIR/slaves+) file to
+account for the removal of these hosts. Bear in mind that the monitor will not re-read the
+slaves file automatically, so it will report the decomissioned servers as down; it's
+recommended that you restart the monitor so that the node list is up to date.
+
++==== Restarting process on a node
++
++Occasionally, it might be necessary to restart the processes on a specific node. In addition
++to the +start-all.sh+ and +stop-all.sh+ scripts, Accumulo contains scripts to start/stop all processes
++on a node and start/stop a given process on a node.
++
+++start-here.sh+ and +stop-here.sh+ will start/stop all Accumulo processes on the current node. The
++necessary processes to start/stop are determined via the "hosts" files (e.g. slaves, masters, etc).
++These scripts expect no arguments.
++
+++start-server.sh+ can also be useful in starting a given process on a host.
++The first argument to the process is the hostname of the machine. Use the same host that
++you specified in hosts file (if you specified FQDN in the masters file, use the FQDN, not
++the shortname). The second argument is the name of the process to start (e.g. master, tserver).
++
++The steps described to decomission a node can also be used (without removal of the host
++from the +$ACCUMUL_HOME/conf/slaves+ file) to gracefully stop a node. This will
++ensure that the tabletserver is cleanly stopped and recovery will not need to be performed
++when the tablets are re-hosted.
++
+[[monitoring]]
+=== Monitoring
+
+==== Accumulo Monitor
+The Accumulo Monitor provides an interface for monitoring the status and health of
+Accumulo components. The Accumulo Monitor provides a web UI for accessing this information at
++http://_monitorhost_:50095/+.
+
+Things highlighted in yellow may be in need of attention.
+If anything is highlighted in red on the monitor page, it is something that definitely needs attention.
+
+The Overview page contains some summary information about the Accumulo instance, including the version, instance name, and instance ID.
+There is a table labeled Accumulo Master with current status, a table listing the active Zookeeper servers, and graphs displaying various metrics over time.
+These include ingest and scan performance and other useful measurements.
+
+The Master Server, Tablet Servers, and Tables pages display metrics grouped in different ways (e.g. by tablet server or by table).
+Metrics typically include number of entries (key/value pairs), ingest and query rates.
+The number of running scans, major and minor compactions are in the form _number_running_ (_number_queued_).
+Another important metric is hold time, which is the amount of time a tablet has been waiting but unable to flush its memory in a minor compaction.
+
+The Server Activity page graphically displays tablet server status, with each server represented as a circle or square.
+Different metrics may be assigned to the nodes' color and speed of oscillation.
+The Overall Avg metric is only used on the Server Activity page, and represents the average of all the other metrics (after normalization).
+Similarly, the Overall Max metric picks the metric with the maximum normalized value.
+
+The Garbage Collector page displays a list of garbage collection cycles, the number of files found of each type (including deletion candidates in use and files actually deleted), and the length of the deletion cycle.
+The Traces page displays data for recent traces performed (see the following section for information on <<tracing>>).
+The Recent Logs page displays warning and error logs forwarded to the monitor from all Accumulo processes.
+Also, the XML and JSON links provide metrics in XML and JSON formats, respectively.
+
+==== SSL
+SSL may be enabled for the monitor page by setting the following properties in the +accumulo-site.xml+ file:
+
+ monitor.ssl.keyStore
+ monitor.ssl.keyStorePassword
+ monitor.ssl.trustStore
+ monitor.ssl.trustStorePassword
+
+If the Accumulo conf directory has been configured (in particular the +accumulo-env.sh+ file must be set up), the +generate_monitor_certificate.sh+ script in the Accumulo +bin+ directory can be used to create the keystore and truststore files with random passwords.
+The script will print out the properties that need to be added to the +accumulo-site.xml+ file.
+The stores can also be generated manually with the Java +keytool+ command, whose usage can be seen in the +generate_monitor_certificate.sh+ script.
+
+If desired, the SSL ciphers allowed for connections can be controlled via the following properties in +accumulo-site.xml+:
+
+ monitor.ssl.include.ciphers
+ monitor.ssl.exclude.ciphers
+
+If SSL is enabled, the monitor URL can only be accessed via https.
+This also allows you to access the Accumulo shell through the monitor page.
+The left navigation bar will have a new link to Shell.
+An Accumulo user name and password must be entered for access to the shell.
+
+=== Metrics
+Accumulo is capable of using the Hadoop Metrics2 library and is configured by default to use it. Metrics2 is a library
+which allows for routing of metrics generated by registered MetricsSources to configured MetricsSinks. Examples of sinks
+that are implemented by Hadoop include file-based logging, Graphite and Ganglia. All metric sources are exposed via JMX
+when using Metrics2.
+
+Previous to Accumulo 1.7.0, JMX endpoints could be exposed in addition to file-based logging of those metrics configured via
+the +accumulo-metrics.xml+ file. This mechanism can still be used by setting +general.legacy.metrics+ to +true+ in +accumulo-site.xml+.
+
+==== Metrics2 Configuration
+
+Metrics2 is configured by examining the classpath for a file that matches +hadoop-metrics2*.properties+. The example configuration
+files that Accumulo provides for use include +hadoop-metrics2-accumulo.properties+ as a template which can be used to enable
+file, Graphite or Ganglia sinks (some minimal configuration required for Graphite and Ganglia). Because the Hadoop configuration is
+also on the Accumulo classpath, be sure that you do not have multiple Metrics2 configuration files. It is recommended to consolidate
+metrics in a single properties file in a central location to remove ambiguity. The contents of +hadoop-metrics2-accumulo.properties+
+can be added to a central +hadoop-metrics2.properties+ in +$HADOOP_CONF_DIR+.
+
+As a note for configuring the file sink, the provided path should be absolute. A relative path or file name will be created relative
+to the directory in which the Accumulo process was started. External tools, such as logrotate, can be used to prevent these files
+from growing without bound.
+
+Each server process should have log messages from the Metrics2 library about the sinks that were created. Be sure to check
+the Accumulo processes log files when debugging missing metrics output.
+
+For additional information on configuring Metrics2, visit the
+https://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html[Javadoc page for Metrics2].
+
+[[tracing]]
+=== Tracing
+It can be difficult to determine why some operations are taking longer
+than expected. For example, you may be looking up items with very low
+latency, but sometimes the lookups take much longer. Determining the
+cause of the delay is difficult because the system is distributed, and
+the typical lookup is fast.
+
+Accumulo has been instrumented to record the time that various
+operations take when tracing is turned on. The fact that tracing is
+enabled follows all the requests made on behalf of the user throughout
+the distributed infrastructure of accumulo, and across all threads of
+execution.
+
+These time spans will be inserted into the +trace+ table in
+Accumulo. You can browse recent traces from the Accumulo monitor
+page. You can also read the +trace+ table directly like any
+other table.
+
+The design of Accumulo's distributed tracing follows that of
+http://research.google.com/pubs/pub36356.html[Google's Dapper].
+
+==== Tracers
+To collect traces, Accumulo needs at least one server listed in
+ +$ACCUMULO_HOME/conf/tracers+. The server collects traces
+from clients and writes them to the +trace+ table. The Accumulo
+user that the tracer connects to Accumulo with can be configured with
+the following properties
+
+ trace.user
+ trace.token.property.password
+
+Other tracer configuration properties include
+
+ trace.port.client
+ trace.table
+
+==== Configuring Tracing
+Traces are collected via SpanReceivers. The default SpanReceiver
+configured is org.apache.accumulo.core.trace.ZooTraceClient, which
+sends spans to an Accumulo Tracer process, as discussed in the
+previous section. This default can be changed to a different span
+receiver, or additional span receivers can be added in a
+comma-separated list, by modifying the property
+
+ trace.span.receivers
+
+Individual span receivers may require their own configuration
+parameters, which are grouped under the trace.span.receiver.*
+prefix. The ZooTraceClient requires the following property that
+indicates where the tracer servers will register themselves in
+ZooKeeper.
+
+ trace.span.receiver.zookeeper.path
+
+This is configured to /tracers by default. If multiple Accumulo
+instances are sharing the same ZooKeeper quorum, take care to
+configure Accumulo with unique values for this property.
+
+Hadoop can also be configured to send traces to Accumulo, as of
+Hadoop 2.6.0, by setting the following properties in Hadoop's
+core-site.xml file (the path property is optional if left as the
+default).
+
+ <property>
+ <name>hadoop.htrace.spanreceiver.classes</name>
+ <value>org.apache.accumulo.core.trace.ZooTraceClient</value>
+ </property>
+ <property>
+ <name>hadoop.tracer.zookeeper.host</name>
+ <value>zookeeperHost:2181</value>
+ </property>
+ <property>
+ <name>hadoop.tracer.zookeeper.path</name>
+ <value>/tracers</value>
+ </property>
+
+The accumulo-core, accumulo-trace, and libthrift jars must also
+be placed on Hadoop's classpath.
+
+==== Instrumenting a Client
+Tracing can be used to measure a client operation, such as a scan, as
+the operation traverses the distributed system. To enable tracing for
+your application call
+
+[source,java]
+import org.apache.accumulo.core.trace.DistributedTrace;
+...
+DistributedTrace.enable(hostname, "myApplication");
+// do some tracing
+...
+DistributedTrace.disable();
+
+Once tracing has been enabled, a client can wrap an operation in a trace.
+
+[source,java]
+import org.htrace.Sampler;
+import org.htrace.Trace;
+import org.htrace.TraceScope;
+...
+TraceScope scope = Trace.startSpan("Client Scan", Sampler.ALWAYS);
+BatchScanner scanner = conn.createBatchScanner(...);
+// Configure your scanner
+for (Entry entry : scanner) {
+}
+scope.close();
+
+Additionally, the user can create additional Spans within a Trace.
+The sampler for the trace should only be specified with the first span, and subsequent spans will be collected depending on whether that first span was sampled.
+
+[source,java]
+TraceScope scope = Trace.startSpan("Client Update", Sampler.ALWAYS);
+...
+TraceScope readScope = Trace.startSpan("Read");
+...
+readScope.close();
+...
+TraceScope writeScope = Trace.startSpan("Write");
+...
+writeScope.close();
+scope.close();
+
+Like Dapper, Accumulo tracing supports user defined annotations to associate additional data with a Trace.
+Checking whether currently tracing is necessary when using a sampler other than Sampler.ALWAYS.
+
+[source,java]
+...
+int numberOfEntriesRead = 0;
+TraceScope readScope = Trace.startSpan("Read");
+// Do the read, update the counter
+...
+if (Trace.isTracing)
+ readScope.getSpan().addKVAnnotation("Number of Entries Read".getBytes(StandardCharsets.UTF_8),
+ String.valueOf(numberOfEntriesRead).getBytes(StandardCharsets.UTF_8));
+
+It is also possible to add timeline annotations to your spans.
+This associates a string with a given timestamp between the start and stop times for a span.
+
+[source,java]
+...
+writeScope.getSpan().addTimelineAnnotation("Initiating Flush");
+
+Some client operations may have a high volume within your
+application. As such, you may wish to only sample a percentage of
+operations for tracing. As seen below, the CountSampler can be used to
+help enable tracing for 1-in-1000 operations
+
+[source,java]
+import org.htrace.impl.CountSampler;
+...
+Sampler sampler = new CountSampler(1000);
+...
+TraceScope readScope = Trace.startSpan("Read", sampler);
+...
+readScope.close();
+
+Remember to close all spans and disable tracing when finished.
+
+[source,java]
+DistributedTrace.disable();
+
+==== Viewing Collected Traces
+To view collected traces, use the "Recent Traces" link on the Monitor
+UI. You can also programmatically access and print traces using the
++TraceDump+ class.
+
+==== Tracing from the Shell
+You can enable tracing for operations run from the shell by using the
++trace on+ and +trace off+ commands.
+
+----
+root@test test> trace on
+
+root@test test> scan
+a b:c [] d
+
+root@test test> trace off
+Waiting for trace information
+Waiting for trace information
+Trace started at 2013/08/26 13:24:08.332
+Time Start Service@Location Name
+ 3628+0 shell@localhost shell:root
+ 8+1690 shell@localhost scan
+ 7+1691 shell@localhost scan:location
+ 6+1692 tserver@localhost startScan
+ 5+1692 tserver@localhost tablet read ahead 6
+----
+
+=== Logging
+Accumulo processes each write to a set of log files. By default these are found under
++$ACCUMULO/logs/+.
+
+=== Recovery
+
+In the event of TabletServer failure or error on shutting Accumulo down, some
+mutations may not have been minor compacted to HDFS properly. In this case,
+Accumulo will automatically reapply such mutations from the write-ahead log
+either when the tablets from the failed server are reassigned by the Master (in the
+case of a single TabletServer failure) or the next time Accumulo starts (in the event of
+failure during shutdown).
+
+Recovery is performed by asking a tablet server to sort the logs so that tablets can easily find their missing
+updates. The sort status of each file is displayed on
+Accumulo monitor status page. Once the recovery is complete any
+tablets involved should return to an ``online'' state. Until then those tablets will be
+unavailable to clients.
+
+The Accumulo client library is configured to retry failed mutations and in many
+cases clients will be able to continue processing after the recovery process without
+throwing an exception.
http://git-wip-us.apache.org/repos/asf/accumulo/blob/06946905/docs/src/main/asciidoc/chapters/implementation.txt
----------------------------------------------------------------------
diff --cc docs/src/main/asciidoc/chapters/implementation.txt
index 0000000,0000000..9ec66ff
new file mode 100644
--- /dev/null
+++ b/docs/src/main/asciidoc/chapters/implementation.txt
@@@ -1,0 -1,0 +1,75 @@@
++// Licensed to the Apache Software Foundation (ASF) under one or more
++// contributor license agreements. See the NOTICE file distributed with
++// this work for additional information regarding copyright ownership.
++// The ASF licenses this file to You under the Apache License, Version 2.0
++// (the "License"); you may not use this file except in compliance with
++// the License. You may obtain a copy of the License at
++//
++// http://www.apache.org/licenses/LICENSE-2.0
++//
++// Unless required by applicable law or agreed to in writing, software
++// distributed under the License is distributed on an "AS IS" BASIS,
++// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
++// See the License for the specific language governing permissions and
++// limitations under the License.
++
++== Implementation Details
++
++=== Fault-Tolerant Executor (FATE)
++
++Accumulo must implement a number of distributed, multi-step operations to support
++the client API. Creating a new table is a simple example of an atomic client call
++which requires multiple steps in the implementation: get a unique table ID, configure
++default table permissions, populate information in ZooKeeper to record the table's
++existence, create directories in HDFS for the table's data, etc. Implementing these
++steps in a way that is tolerant to node failure and other concurrent operations is
++very difficult to achieve. Accumulo includes a Fault-Tolerant Executor (FATE) which
++is widely used server-side to implement the client API safely and correctly.
++
++FATE is the implementation detail which ensures that tables in creation when the
++Master dies will be successfully created when another Master process is started.
++This alleviates the need for any external tools to correct some bad state -- Accumulo can
++undo the failure and self-heal without any external intervention.
++
++=== Overview
++
++FATE consists of two primary components: a repeatable, persisted operation (REPO), a storage
++layer for REPOs and an execution system to run REPOs. Accumulo uses ZooKeeper as the storage
++layer for FATE and the Accumulo Master acts as the execution system to run REPOs.
++
++The important characteristic of REPOs are that they implemented in a way that is idempotent:
++every operation must be able to undo or replay a partial execution of itself. Requiring the
++implementation of the operation to support this functional greatly simplifies the execution
++of these operations. This property is also what guarantees safety in light of failure conditions.
++
++=== Administration
++
++Sometimes, it is useful to inspect the current FATE operations, both pending and executing.
++For example, a command that is not completing could be blocked on the execution of another
++operation. Accumulo provides an Accumulo shell command to interact with fate.
++
++The +fate+ shell command accepts a number of arguments for different functionality:
+++list+/+print+, +fail+, +delete+.
++
++==== List/Print
++
++Without any additional arguments, this command will print all operations that still exist in
++the FATE store (ZooKeeper). This will include active, pending, and completed operations (completed
++operations are lazily removed from the store). Each operation includes a unique "transaction ID", the
++state of the operation (e.g. +NEW+, +IN_PROGRESS+, +FAILED+), any locks the
++transaction actively holds and any locks it is waiting to acquire.
++
++This option can also accept transaction IDs which will restrict the list of transactions shown.
++
++==== Fail
++
++This command can be used to manually fail a FATE transaction and requires a transaction ID
++as an argument. Failing an operation is not a normal procedure and should only be performed
++by an administrator who understands the implications of why they are failing the operation.
++
++==== Delete
++
++This command requires a transaction ID and will delete any locks that the transaction
++holds. Like the fail command, this command should only be used in extreme circumstances
++by an administrator that understands the implications of the command they are about to
++invoke. It is not normal to invoke this command.