You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by mw...@apache.org on 2017/05/22 20:10:50 UTC
[01/10] accumulo-website git commit: Split up long administration and
troubleshooting overview pages
Repository: accumulo-website
Updated Branches:
refs/heads/asf-site 652e2a53d -> 3e4443c11
refs/heads/master 7e6a7b9f2 -> 036341f26
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/036341f2/_docs-unreleased/troubleshooting/advanced.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/troubleshooting/advanced.md b/_docs-unreleased/troubleshooting/advanced.md
new file mode 100644
index 0000000..95d9ce6
--- /dev/null
+++ b/_docs-unreleased/troubleshooting/advanced.md
@@ -0,0 +1,365 @@
+---
+title: Advanced Troubleshooting
+category: troubleshooting
+order: 2
+---
+
+## Tablet server locks
+
+**My tablet server lost its lock. Why?**
+
+The primary reason a tablet server loses its lock is that it has been pushed into swap.
+
+A large java program (like the tablet server) may have a large portion
+of its memory image unused. The operation system will favor pushing
+this allocated, but unused memory into swap so that the memory can be
+re-used as a disk buffer. When the java virtual machine decides to
+access this memory, the OS will begin flushing disk buffers to return that
+memory to the VM. This can cause the entire process to block long
+enough for the zookeeper lock to be lost.
+
+Configure your system to reduce the kernel parameter _swappiness_ from the default (60) to zero.
+
+**My tablet server lost its lock, and I have already set swappiness to zero. Why?**
+
+Be careful not to over-subscribe memory. This can be easy to do if
+your accumulo processes run on the same nodes as hadoop's map-reduce
+framework. Remember to add up:
+
+* size of the JVM for the tablet server
+* size of the in-memory map, if using the native map implementation
+* size of the JVM for the data node
+* size of the JVM for the task tracker
+* size of the JVM times the maximum number of mappers and reducers
+* size of the kernel and any support processes
+
+If a 16G node can run 2 mappers and 2 reducers, and each can be 2G,
+then there is only 8G for the data node, tserver, task tracker and OS.
+
+Reduce the memory footprint of each component until it fits comfortably.
+
+**My tablet server lost its lock, swappiness is zero, and my node has lots of unused memory!**
+
+The JVM memory garbage collector may fall behind and cause a
+"stop-the-world" garbage collection. On a large memory virtual
+machine, this collection can take a long time. This happens more
+frequently when the JVM is getting low on free memory. Check the logs
+of the tablet server. You will see lines like this:
+
+ 2013-06-20 13:43:20,607 [tabletserver.TabletServer] DEBUG: gc ParNew=0.00(+0.00) secs
+ ConcurrentMarkSweep=0.00(+0.00) secs freemem=1,868,325,952(+1,868,325,952) totalmem=2,040,135,680
+
+When `freemem` becomes small relative to the amount of memory
+needed, the JVM will spend more time finding free memory than
+performing work. This can cause long delays in sending keep-alive
+messages to zookeeper.
+
+Ensure the tablet server JVM is not running low on memory.
+
+**I'm seeing errors in tablet server logs that include the words "MutationsRejectedException" and "# constraint violations: 1". Moments after that the server died.**
+
+The error you are seeing is part of a failing tablet server scenario.
+This is a bit complicated, so name two of your tablet servers A and B.
+
+Tablet server A is hosting a tablet, let's call it a-tablet.
+
+Tablet server B is hosting a metadata tablet, let's call it m-tablet.
+
+m-tablet records the information about a-tablet, for example, the names of the files it is using to store data.
+
+When A ingests some data, it eventually flushes the updates from memory to a file.
+
+Tablet server A then writes this new information to m-tablet, on Tablet server B.
+
+Here's a likely failure scenario:
+
+Tablet server A does not have enough memory for all the processes running on it.
+The operating system sees a large chunk of the tablet server being unused, and swaps it out to disk to make room for other processes.
+Tablet server A does a java memory garbage collection, which causes it to start using all the memory allocated to it.
+As the server starts pulling data from swap, it runs very slowly.
+It fails to send the keep-alive messages to zookeeper in a timely fashion, and it looses its zookeeper session.
+
+But, it's running so slowly, that it takes a moment to realize it should no longer be hosting tablets.
+
+The thread that is flushing a-tablet memory attempts to update m-tablet with the new file information.
+
+Fortunately there's a constraint on m-tablet.
+Mutations to the metadata table must contain a valid zookeeper session.
+This prevents tablet server A from making updates to m-tablet when it no long has the right to host the tablet.
+
+The "MutationsRejectedException" error is from tablet server A making an update to tablet server B's m-tablet.
+It's getting a constraint violation: tablet server A has lost its zookeeper session, and will fail momentarily.
+
+Ensure that memory is not over-allocated. Monitor swap usage, or turn swap off.
+
+**My accumulo client is getting a MutationsRejectedException. The monitor is displaying "No Such SessionID" errors.**
+
+When your client starts sending mutations to accumulo, it creates a session. Once the session is created,
+mutations are streamed to accumulo, without acknowledgement, against this session. Once the client is done,
+it will close the session, and get an acknowledgement.
+
+If the client fails to communicate with accumulo, it will release the session, assuming that the client has died.
+If the client then attempts to send more mutations against the session, you will see "No Such SessionID" errors on
+the server, and MutationRejectedExceptions in the client.
+
+The client library should be either actively using the connection to the tablet servers,
+or closing the connection and sessions. If the session times out, something is causing your client
+to pause.
+
+The most frequent source of these pauses are java garbage collection pauses
+due to the JVM running out of memory, or being swapped out to disk.
+
+Ensure your client has adequate memory and is not being swapped out to disk.
+
+## HDFS Failures
+
+**I had disasterous HDFS failure. After bringing everything back up, several tablets refuse to go online.**
+
+Data written to tablets is written into memory before being written into indexed files. In case the server
+is lost before the data is saved into a an indexed file, all data stored in memory is first written into a
+write-ahead log (WAL). When a tablet is re-assigned to a new tablet server, the write-ahead logs are read to
+recover any mutations that were in memory when the tablet was last hosted.
+
+If a write-ahead log cannot be read, then the tablet is not re-assigned. All it takes is for one of
+the blocks in the write-ahead log to be missing. This is unlikely unless multiple data nodes in HDFS have been
+lost.
+
+Get the WAL files online and healthy. Restore any data nodes that may be down.
+
+**How do find out which tablets are offline?**
+
+Use `accumulo admin checkTablets`
+
+ $ accumulo admin checkTablets
+
+**I lost three data nodes, and I'm missing blocks in a WAL. I don't care about data loss, how
+can I get those tablets online?**
+
+See the [system metadata table page][metadata] which shows a typical metadata table listing.
+The entries with a column family of `log` are references to the WAL for that tablet.
+If you know what WAL is bad, you can find all the references with a grep in the shell:
+
+ shell> grep 0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995
+ 3< log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995|6
+
+You can remove the WAL references in the metadata table.
+
+ shell> grant -u root Table.WRITE -t accumulo.metadata
+ shell> delete 3< log 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995
+
+Note: the colon (`:`) is omitted when specifying the _row cf cq_ for the delete command.
+
+The master will automatically discover the tablet no longer has a bad WAL reference and will
+assign the tablet. You will need to remove the reference from all the tablets to get them
+online.
+
+**The metadata (or root) table has references to a corrupt WAL.**
+
+This is a much more serious state, since losing updates to the metadata table will result
+in references to old files which may not exist, or lost references to new files, resulting
+in tablets that cannot be read, or large amounts of data loss.
+
+The best hope is to restore the WAL by fixing HDFS data nodes and bringing the data back online.
+If this is not possible, the best approach is to re-create the instance and bulk import all files from
+the old instance into a new tables.
+
+A complete set of instructions for doing this is outside the scope of this guide,
+but the basic approach is:
+
+* Use `tables -l` in the shell to discover the table name to table id mapping
+* Stop all accumulo processes on all nodes
+* Move the accumulo directory in HDFS out of the way:
+ $ hadoop fs -mv /accumulo /corrupt
+* Re-initalize accumulo
+* Recreate tables, users and permissions
+* Import the directories under `/corrupt/tables/<id>` into the new instance
+
+**One or more HDFS Files under /accumulo/tables are corrupt**
+
+Accumulo maintains multiple references into the tablet files in the metadata
+tables and within the tablet server hosting the file, this makes it difficult to
+reliably just remove those references.
+
+The directory structure in HDFS for tables will follow the general structure:
+
+ /accumulo
+ /accumulo/tables/
+ /accumulo/tables/!0
+ /accumulo/tables/!0/default_tablet/A000001.rf
+ /accumulo/tables/!0/t-00001/A000002.rf
+ /accumulo/tables/1
+ /accumulo/tables/1/default_tablet/A000003.rf
+ /accumulo/tables/1/t-00001/A000004.rf
+ /accumulo/tables/1/t-00001/A000005.rf
+ /accumulo/tables/2/default_tablet/A000006.rf
+ /accumulo/tables/2/t-00001/A000007.rf
+
+If files under `/accumulo/tables` are corrupt, the best course of action is to
+recover those files in hdsf see the section on HDFS. Once these recovery efforts
+have been exhausted, the next step depends on where the missing file(s) are
+located. Different actions are required when the bad files are in Accumulo data
+table files or if they are metadata table files.
+
+*Data File Corruption*
+
+When an Accumulo data file is corrupt, the most reliable way to restore Accumulo
+operations is to replace the missing file with an ``empty'' file so that
+references to the file in the METADATA table and within the tablet server
+hosting the file can be resolved by Accumulo. An empty file can be created using
+the CreateEmpty utiity:
+
+ $ accumulo org.apache.accumulo.core.file.rfile.CreateEmpty /path/to/empty/file/empty.rf
+
+The process is to delete the corrupt file and then move the empty file into its
+place (The generated empty file can be copied and used multiple times if necessary and does not need
+to be regenerated each time)
+
+ $ hadoop fs –rm /accumulo/tables/corrupt/file/thename.rf; \
+ hadoop fs -mv /path/to/empty/file/empty.rf /accumulo/tables/corrupt/file/thename.rf
+
+*Metadata File Corruption*
+
+If the corrupt files are metadata files, read the [system metadata tables][metadata]
+(under the path `/accumulo/tables/!0`). Then, you will need to rebuild
+the metadata table by initializing a new instance of Accumulo and then importing
+all of the existing data into the new instance. This is the same procedure as
+recovering from a zookeeper failure (see next section), except that
+you will have the benefit of having the existing user and table authorizations
+that are maintained in zookeeper.
+
+You can use the DumpZookeeper utility to save this information for reference
+before creating the new instance. You will not be able to use RestoreZookeeper
+because the table names and references are likely to be different between the
+original and the new instances, but it can serve as a reference.
+
+If the files cannot be recovered, replace corrupt data files with a empty
+rfiles to allow references in the metadata table and in the tablet servers to be
+resolved. Rebuild the metadata table if the corrupt files are metadata files.
+
+*Write-Ahead Log(WAL) File Corruption*
+
+In certain versions of Accumulo, a corrupt WAL file (caused by HDFS corruption
+or a bug in Accumulo that created the file) can block the successful recovery
+of one to many Tablets. Accumulo can be stuck in a loop trying to recover the
+WAL file, never being able to succeed.
+
+In the cases where the WAL file's original contents are unrecoverable or some degree
+of data loss is acceptable (beware if the WAL file contains updates to the Accumulo
+metadat table!), the following process can be followed to create an valid, empty
+WAL file. Run the following commands as the Accumulo unix user (to ensure that
+the proper file permissions in HDFS)
+
+ $ echo -n -e '--- Log File Header (v2) ---\x00\x00\x00\x00' > empty.wal
+
+The above creates a file with the text "--- Log File Header (v2) ---" and then
+four bytes. You should verify the contents of the file with a hexdump tool.
+
+Then, place this empty WAL in HDFS and then replace the corrupt WAL file in HDFS
+with the empty WAL.
+
+ $ hdfs dfs -moveFromLocal empty.wal /user/accumulo/empty.wal
+ $ hdfs dfs -mv /user/accumulo/empty.wal /accumulo/wal/tserver-4.example.com+10011/26abec5b-63e7-40dd-9fa1-b8ad2436606e
+
+After the corrupt WAL file has been replaced, the system should automatically recover.
+It may be necessary to restart the Accumulo Master process as an exponential
+backup policy is used which could lead to a long wait before Accumulo will
+try to re-load the WAL file.
+
+## Zookeeper Failures
+
+**I lost my ZooKeeper quorum (hardware failure), but HDFS is still intact. How can I recover my Accumulo instance?**
+
+ZooKeeper, in addition to its lock-service capabilities, also serves to bootstrap an Accumulo
+instance from some location in HDFS. It contains the pointers to the root tablet in HDFS which
+is then used to load the Accumulo metadata tablets, which then loads all user tables. ZooKeeper
+also stores all namespace and table configuration, the user database, the mapping of table IDs to
+table names, and more across Accumulo restarts.
+
+Presently, the only way to recover such an instance is to initialize a new instance and import all
+of the old data into the new instance. The easiest way to tackle this problem is to first recreate
+the mapping of table ID to table name and then recreate each of those tables in the new instance.
+Set any necessary configuration on the new tables and add some split points to the tables to close
+the gap between how many splits the old table had and no splits.
+
+The directory structure in HDFS for tables will follow the general structure:
+
+ /accumulo
+ /accumulo/tables/
+ /accumulo/tables/1
+ /accumulo/tables/1/default_tablet/A000001.rf
+ /accumulo/tables/1/t-00001/A000002.rf
+ /accumulo/tables/1/t-00001/A000003.rf
+ /accumulo/tables/2/default_tablet/A000004.rf
+ /accumulo/tables/2/t-00001/A000005.rf
+
+For each table, make a new directory that you can move (or copy if you have the HDFS space to do so)
+all of the rfiles for a given table into. For example, to process the table with an ID of `1`, make a new directory,
+say `/new-table-1` and then copy all files from `/accumulo/tables/1/\*/*.rf` into that directory. Additionally,
+make a directory, `/new-table-1-failures`, for any failures during the import process. Then, issue the import
+command using the Accumulo shell into the new table, telling Accumulo to not re-set the timestamp:
+
+ user@instance new_table> importdirectory /new-table-1 /new-table-1-failures false
+
+Any RFiles which were failed to be loaded will be placed in `/new-table-1-failures`. Rfiles that were successfully
+imported will no longer exist in `/new-table-1`. For failures, move them back to the import directory and retry
+the `importdirectory` command.
+
+It is *extremely* important to note that this approach may introduce stale data back into
+the tables. For a few reasons, RFiles may exist in the table directory which are candidates for deletion but have
+not yet been deleted. Additionally, deleted data which was not compacted away, but still exists in write-ahead logs if
+the original instance was somehow recoverable, will be re-introduced in the new instance. Table splits and merges
+(which also include the deleteRows API call on TableOperations, are also vulnerable to this problem. This process should
+*not* be used if these are unacceptable risks. It is possible to try to re-create a view of the `accumulo.metadata`
+table to prune out files that are candidates for deletion, but this is a difficult task that also may not be entirely accurate.
+
+Likewise, it is also possible that data loss may occur from write-ahead log (WAL) files which existed on the old table but
+were not minor-compacted into an RFile. Again, it may be possible to reconstruct the state of these WAL files to
+replay data not yet in an RFile; however, this is a difficult task and is not implemented in any automated fashion.
+
+The `importdirectory` shell command can be used to import RFiles from the old instance into a newly created instance,
+but extreme care should go into the decision to do this as it may result in reintroduction of stale data or the
+omission of new data.
+
+## Upgrade Issues
+
+**I upgraded from 1.4 to 1.5 to 1.6 but still have some WAL files on local disk. Do I have any way to recover them?**
+
+Yes, you can recover them by running the LocalWALRecovery utility (not available in 1.8 and later) on each node that needs recovery performed. The utility
+will default to using the directory specified by `logger.dir.walog` in your configuration, or can be
+overriden by using the `--local-wal-directories` option on the tool. It can be invoked as follows:
+
+ accumulo org.apache.accumulo.tserver.log.LocalWALRecovery
+
+## File Naming Conventions
+
+**Why are files named like they are? Why do some start with `C` and others with `F`?**
+
+The file names give you a basic idea for the source of the file.
+
+The base of the filename is a base-36 unique number. All filenames in accumulo are coordinated
+with a counter in zookeeper, so they are always unique, which is useful for debugging.
+
+The leading letter gives you an idea of how the file was created:
+
+* `F` - Flush: entries in memory were written to a file (Minor Compaction)
+* `M` - Merging compaction: entries in memory were combined with the smallest file to create one new file
+* `C` - Several files, but not all files, were combined to produce this file (Major Compaction)
+* `A` - All files were compacted, delete entries were dropped
+* `I` - Bulk import, complete, sorted index files. Always in a directory starting with `b-`
+
+This simple file naming convention allows you to see the basic structure of the files from just
+their filenames, and reason about what should be happening to them next, just
+by scanning their entries in the metadata tables.
+
+For example, if you see multiple files with `M` prefixes, the tablet is, or was, up against its
+maximum file limit, so it began merging memory updates with files to keep the file count reasonable. This
+slows down ingest performance, so knowing there are many files like this tells you that the system
+is struggling to keep up with ingest vs the compaction strategy which reduces the number of files.
+
+## HDFS Decommissioning Issues
+
+**My Hadoop DataNode is hung for hours trying to decommission.**
+
+Write Ahead Logs stay open until they hit the size threshold, which could be many hours or days in some cases. These open files will prevent a DN from finishing its decommissioning process (HDFS-3599) in some versions of Hadoop 2. If you stop the DN, then the WALog file will not be closed and you could lose data. To work around this issue, we now close WALogs on a time period specified by the property `tserver.walog.max.age` with a default period of 24 hours.
+
+[metadata]: {{ page.docs_baseurl }}/troubleshooting/system-metadata-tables
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/036341f2/_docs-unreleased/troubleshooting/basic.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/troubleshooting/basic.md b/_docs-unreleased/troubleshooting/basic.md
new file mode 100644
index 0000000..7f82df7
--- /dev/null
+++ b/_docs-unreleased/troubleshooting/basic.md
@@ -0,0 +1,227 @@
+---
+title: Basic Troubleshooting
+category: troubleshooting
+order: 1
+---
+
+## General
+
+**The tablet server does not seem to be running!? What happened?**
+
+Accumulo is a distributed system. It is supposed to run on remote
+equipment, across hundreds of computers. Each program that runs on
+these remote computers writes down events as they occur, into a local
+file. By default, this is defined in `conf/accumulo-env.sh` as `ACCUMULO_LOG_DIR`.
+Look in the `$ACCUMULO_LOG_DIR/tserver*.log` file. Specifically, check the end of the file.
+
+**The tablet server did not start and the debug log does not exists! What happened?**
+
+When the individual programs are started, the stdout and stderr output
+of these programs are stored in `.out` and `.err` files in
+`$ACCUMULO_LOG_DIR`. Often, when there are missing configuration
+options, files or permissions, messages will be left in these files.
+Probably a start-up problem. Look in `$ACCUMULO_LOG_DIR/tserver*.err`
+
+**Accumulo is not working, what's wrong?**
+
+There's a small web server that collects information about all the
+components that make up a running Accumulo instance. It will highlight
+unusual or unexpected conditions.
+
+Point your browser to the monitor (typically the master host, on port 9995). Is anything red or yellow?
+
+**My browser is reporting connection refused, and I cannot get to the monitor**
+
+The monitor program's output is also written to .err and .out files in
+the `$ACCUMULO_LOG_DIR`. Look for problems in this file if the
+`$ACCUMULO_LOG_DIR/monitor*.log` file does not exist.
+
+The monitor program is probably not running. Check the log files for errors.
+
+**My browser hangs trying to talk to the monitor.**
+
+Your browser needs to be able to reach the monitor program. Often
+large clusters are firewalled, or use a VPN for internal
+communications. You can use SSH to proxy your browser to the cluster,
+or consult with your system administrator to gain access to the server
+from your browser.
+
+It is sometimes helpful to use a text-only browser to sanity-check the
+monitor while on the machine running the monitor:
+
+ $ links http://localhost:9995
+
+Verify that you are not firewalled from the monitor if it is running on a remote host.
+
+**The monitor responds, but there are no numbers for tservers and tables. The summary page says the master is down.**
+
+The monitor program gathers all the details about the master and the
+tablet servers through the master. It will be mostly blank if the
+master is down. Check for a running master.
+
+## Accumulo Processes
+
+**My tablet server crashed! The logs say that it lost its zookeeper lock.**
+
+Tablet servers reserve a lock in zookeeper to maintain their ownership
+over the tablets that have been assigned to them. Part of their
+responsibility for keeping the lock is to send zookeeper a keep-alive
+message periodically. If the tablet server fails to send a message in
+a timely fashion, zookeeper will remove the lock and notify the tablet
+server. If the tablet server does not receive a message from
+zookeeper, it will assume its lock has been lost, too. If a tablet
+server loses its lock, it kills itself: everything assumes it is dead
+already.
+
+Investigate why the tablet server did not send a timely message to
+zookeeper.
+
+**I need to decommission a node. How do I stop the tablet server on it?**
+
+Use the admin command:
+
+ $ accumulo admin stop hostname:9997
+ 2013-07-16 13:15:38,403 [util.Admin] INFO : Stopping server 12.34.56.78:9997
+
+**I cannot login to a tablet server host, and the tablet server will not shut down. How can I kill the server?**
+
+Sometimes you can kill a "stuck" tablet server by deleting its lock in zookeeper:
+
+ $ accumulo org.apache.accumulo.server.util.TabletServerLocks --list
+ 127.0.0.1:9997 TSERV_CLIENT=127.0.0.1:9997
+ $ accumulo org.apache.accumulo.server.util.TabletServerLocks -delete 127.0.0.1:9997
+ $ accumulo org.apache.accumulo.server.util.TabletServerLocks -list
+ 127.0.0.1:9997 null
+
+You can find the master and instance id for any accumulo instances using the same zookeeper instance:
+
+```
+$ accumulo org.apache.accumulo.server.util.ListInstances
+INFO : Using ZooKeepers localhost:2181
+
+ Instance Name | Instance ID | Master
+---------------------+--------------------------------------+-------------------------------
+ "test" | 6140b72e-edd8-4126-b2f5-e74a8bbe323b | 127.0.0.1:9999
+```
+
+**One of my Accumulo processes died. How do I bring it back?**
+
+The easiest way to bring all services online for an Accumulo instance is to run the `accumulo-cluster` script.
+
+ $ accumulo-cluster start
+
+This process will check the process listing, using `jps` on each host before attempting to restart a service on the given host.
+Typically, this check is sufficient except in the face of a hung/zombie process. For large clusters, it may be
+undesirable to ssh to every node in the cluster to ensure that all hosts are running the appropriate processes and `accumulo-service` may be of use.
+
+ $ ssh host_with_dead_process
+ $ accumulo-service tserver start
+
+**My process died again. Should I restart it via `cron` or tools like `supervisord`?**
+
+A repeatedly dying Accumulo process is a sign of a larger problem. Typically these problems are due to a
+misconfiguration of Accumulo or over-saturation of resources. Blind automation of any service restart inside of Accumulo
+is generally an undesirable situation as it is indicative of a problem that is being masked and ignored. Accumulo
+processes should be stable on the order of months and not require frequent restart.
+
+## Accumulo Clients
+
+**Accumulo is not showing me any data!**
+
+Do you have your auths set so that it matches your visibilities?
+
+**What are my visibilities?**
+
+Use the [PrintInfo][print-info] tool on a representative file to get some idea
+of the visibilities in the underlying data.
+
+Note that the use of `PrintInfo` is an administrative tool and can only
+by used by someone who can access the underlying Accumulo data. It
+does not provide the normal access controls in Accumulo.
+
+## HDFS
+
+Accumulo reads and writes to the Hadoop Distributed File System.
+Accumulo needs this file system available at all times for normal operations.
+
+**Accumulo is having problems "getting a block blk_1234567890123". How do I fix it?**
+
+This troubleshooting guide does not cover HDFS, but in general, you
+want to make sure that all the datanodes are running and an fsck check
+finds the file system clean:
+
+ $ hadoop fsck /accumulo
+
+You can use:
+
+ $ hadoop fsck /accumulo/path/to/corrupt/file -locations -blocks -files
+
+to locate the block references of individual corrupt files and use those
+references to search the name node and individual data node logs to determine which
+servers those blocks have been assigned and then try to fix any underlying file
+system issues on those nodes.
+
+On a larger cluster, you may need to increase the number of Xcievers for HDFS DataNodes:
+
+```xml
+<property>
+ <name>dfs.datanode.max.xcievers</name>
+ <value>4096</value>
+</property>
+```
+
+Verify HDFS is healthy, check the datanode logs.
+
+## Zookeeper
+
+**The `accumulo init` command is hanging. It says something about talking to zookeeper.**
+
+Zookeeper is also a distributed service. You will need to ensure that
+it is up. You can run the zookeeper command line tool to connect to
+any one of the zookeeper servers:
+
+ $ zkCli.sh -server zoohost
+ ...
+ [zk: zoohost:2181(CONNECTED) 0]
+
+It is important to see the word `CONNECTED`! If you only see
+`CONNECTING` you will need to diagnose zookeeper errors.
+
+Check to make sure that zookeeper is up, and that
+`accumulo-site.xml` has been pointed to
+your zookeeper server(s).
+
+**Zookeeper is running, but it does not say CONNECTED**
+
+Zookeeper processes talk to each other to elect a leader. All updates
+go through the leader and propagate to a majority of all the other
+nodes. If a majority of the nodes cannot be reached, zookeeper will
+not allow updates. Zookeeper also limits the number connections to a
+server from any other single host. By default, this limit can be as small as 10
+and can be reached in some everything-on-one-machine test configurations.
+
+You can check the election status and connection status of clients by
+asking the zookeeper nodes for their status. You connect to zookeeper
+and ask it with the four-letter `stat` command:
+
+```
+$ nc zoohost 2181
+stat
+Zookeeper version: 3.4.5-1392090, built on 09/30/2012 17:52 GMT
+Clients:
+ /127.0.0.1:58289[0](queued=0,recved=1,sent=0)
+ /127.0.0.1:60231[1](queued=0,recved=53910,sent=53915)
+
+Latency min/avg/max: 0/5/3008
+Received: 1561459
+Sent: 1561592
+Connections: 2
+Outstanding: 0
+Zxid: 0x621a3b
+Mode: standalone
+Node count: 22524
+```
+
+Check zookeeper status, verify that it has a quorum, and has not exceeded maxClientCnxns.
+
+[print-info]: {{ page.docs_baseurl }}/troubleshooting/tools#PrintInfo
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/036341f2/_docs-unreleased/troubleshooting/overview.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/troubleshooting/overview.md b/_docs-unreleased/troubleshooting/overview.md
deleted file mode 100644
index 0b6946c..0000000
--- a/_docs-unreleased/troubleshooting/overview.md
+++ /dev/null
@@ -1,820 +0,0 @@
----
-title: Overview
-category: troubleshooting
-order: 1
----
-
-## Logs
-
-**The tablet server does not seem to be running!? What happened?**
-
-Accumulo is a distributed system. It is supposed to run on remote
-equipment, across hundreds of computers. Each program that runs on
-these remote computers writes down events as they occur, into a local
-file. By default, this is defined in `conf/accumulo-env.sh` as `ACCUMULO_LOG_DIR`.
-Look in the `$ACCUMULO_LOG_DIR/tserver*.log` file. Specifically, check the end of the file.
-
-**The tablet server did not start and the debug log does not exists! What happened?**
-
-When the individual programs are started, the stdout and stderr output
-of these programs are stored in `.out` and `.err` files in
-`$ACCUMULO_LOG_DIR`. Often, when there are missing configuration
-options, files or permissions, messages will be left in these files.
-Probably a start-up problem. Look in `$ACCUMULO_LOG_DIR/tserver*.err`
-
-## Monitor
-
-**Accumulo is not working, what's wrong?**
-
-There's a small web server that collects information about all the
-components that make up a running Accumulo instance. It will highlight
-unusual or unexpected conditions.
-
-Point your browser to the monitor (typically the master host, on port 9995). Is anything red or yellow?
-
-**My browser is reporting connection refused, and I cannot get to the monitor**
-
-The monitor program's output is also written to .err and .out files in
-the `$ACCUMULO_LOG_DIR`. Look for problems in this file if the
-`$ACCUMULO_LOG_DIR/monitor*.log` file does not exist.
-
-The monitor program is probably not running. Check the log files for errors.
-
-**My browser hangs trying to talk to the monitor.**
-
-Your browser needs to be able to reach the monitor program. Often
-large clusters are firewalled, or use a VPN for internal
-communications. You can use SSH to proxy your browser to the cluster,
-or consult with your system administrator to gain access to the server
-from your browser.
-
-It is sometimes helpful to use a text-only browser to sanity-check the
-monitor while on the machine running the monitor:
-
- $ links http://localhost:9995
-
-Verify that you are not firewalled from the monitor if it is running on a remote host.
-
-**The monitor responds, but there are no numbers for tservers and tables. The summary page says the master is down.**
-
-The monitor program gathers all the details about the master and the
-tablet servers through the master. It will be mostly blank if the
-master is down. Check for a running master.
-
-## HDFS
-
-Accumulo reads and writes to the Hadoop Distributed File System.
-Accumulo needs this file system available at all times for normal operations.
-
-**Accumulo is having problems "getting a block blk_1234567890123". How do I fix it?**
-
-This troubleshooting guide does not cover HDFS, but in general, you
-want to make sure that all the datanodes are running and an fsck check
-finds the file system clean:
-
- $ hadoop fsck /accumulo
-
-You can use:
-
- $ hadoop fsck /accumulo/path/to/corrupt/file -locations -blocks -files
-
-to locate the block references of individual corrupt files and use those
-references to search the name node and individual data node logs to determine which
-servers those blocks have been assigned and then try to fix any underlying file
-system issues on those nodes.
-
-On a larger cluster, you may need to increase the number of Xcievers for HDFS DataNodes:
-
-```xml
-<property>
- <name>dfs.datanode.max.xcievers</name>
- <value>4096</value>
-</property>
-```
-
-Verify HDFS is healthy, check the datanode logs.
-
-## Zookeeper
-
-** The"accumulo init" command is hanging. It says something about talking to zookeeper.**
-
-Zookeeper is also a distributed service. You will need to ensure that
-it is up. You can run the zookeeper command line tool to connect to
-any one of the zookeeper servers:
-
- $ zkCli.sh -server zoohost
- ...
- [zk: zoohost:2181(CONNECTED) 0]
-
-It is important to see the word `CONNECTED`! If you only see
-`CONNECTING` you will need to diagnose zookeeper errors.
-
-Check to make sure that zookeeper is up, and that
-`accumulo-site.xml` has been pointed to
-your zookeeper server(s).
-
-**Zookeeper is running, but it does not say CONNECTED**
-
-Zookeeper processes talk to each other to elect a leader. All updates
-go through the leader and propagate to a majority of all the other
-nodes. If a majority of the nodes cannot be reached, zookeeper will
-not allow updates. Zookeeper also limits the number connections to a
-server from any other single host. By default, this limit can be as small as 10
-and can be reached in some everything-on-one-machine test configurations.
-
-You can check the election status and connection status of clients by
-asking the zookeeper nodes for their status. You connect to zookeeper
-and ask it with the four-letter `stat` command:
-
-```
-$ nc zoohost 2181
-stat
-Zookeeper version: 3.4.5-1392090, built on 09/30/2012 17:52 GMT
-Clients:
- /127.0.0.1:58289[0](queued=0,recved=1,sent=0)
- /127.0.0.1:60231[1](queued=0,recved=53910,sent=53915)
-
-Latency min/avg/max: 0/5/3008
-Received: 1561459
-Sent: 1561592
-Connections: 2
-Outstanding: 0
-Zxid: 0x621a3b
-Mode: standalone
-Node count: 22524
-```
-
-Check zookeeper status, verify that it has a quorum, and has not exceeded maxClientCnxns.
-
-**My tablet server crashed! The logs say that it lost its zookeeper lock.**
-
-Tablet servers reserve a lock in zookeeper to maintain their ownership
-over the tablets that have been assigned to them. Part of their
-responsibility for keeping the lock is to send zookeeper a keep-alive
-message periodically. If the tablet server fails to send a message in
-a timely fashion, zookeeper will remove the lock and notify the tablet
-server. If the tablet server does not receive a message from
-zookeeper, it will assume its lock has been lost, too. If a tablet
-server loses its lock, it kills itself: everything assumes it is dead
-already.
-
-Investigate why the tablet server did not send a timely message to
-zookeeper.
-
-### Keeping the tablet server lock
-
-**My tablet server lost its lock. Why?**
-
-The primary reason a tablet server loses its lock is that it has been pushed into swap.
-
-A large java program (like the tablet server) may have a large portion
-of its memory image unused. The operation system will favor pushing
-this allocated, but unused memory into swap so that the memory can be
-re-used as a disk buffer. When the java virtual machine decides to
-access this memory, the OS will begin flushing disk buffers to return that
-memory to the VM. This can cause the entire process to block long
-enough for the zookeeper lock to be lost.
-
-Configure your system to reduce the kernel parameter _swappiness_ from the default (60) to zero.
-
-**My tablet server lost its lock, and I have already set swappiness to zero. Why?**
-
-Be careful not to over-subscribe memory. This can be easy to do if
-your accumulo processes run on the same nodes as hadoop's map-reduce
-framework. Remember to add up:
-
-* size of the JVM for the tablet server
-* size of the in-memory map, if using the native map implementation
-* size of the JVM for the data node
-* size of the JVM for the task tracker
-* size of the JVM times the maximum number of mappers and reducers
-* size of the kernel and any support processes
-
-If a 16G node can run 2 mappers and 2 reducers, and each can be 2G,
-then there is only 8G for the data node, tserver, task tracker and OS.
-
-Reduce the memory footprint of each component until it fits comfortably.
-
-**My tablet server lost its lock, swappiness is zero, and my node has lots of unused memory!**
-
-The JVM memory garbage collector may fall behind and cause a
-"stop-the-world" garbage collection. On a large memory virtual
-machine, this collection can take a long time. This happens more
-frequently when the JVM is getting low on free memory. Check the logs
-of the tablet server. You will see lines like this:
-
- 2013-06-20 13:43:20,607 [tabletserver.TabletServer] DEBUG: gc ParNew=0.00(+0.00) secs
- ConcurrentMarkSweep=0.00(+0.00) secs freemem=1,868,325,952(+1,868,325,952) totalmem=2,040,135,680
-
-When `freemem` becomes small relative to the amount of memory
-needed, the JVM will spend more time finding free memory than
-performing work. This can cause long delays in sending keep-alive
-messages to zookeeper.
-
-Ensure the tablet server JVM is not running low on memory.
-
-**I'm seeing errors in tablet server logs that include the words "MutationsRejectedException" and "# constraint violations: 1". Moments after that the server died.**
-
-The error you are seeing is part of a failing tablet server scenario.
-This is a bit complicated, so name two of your tablet servers A and B.
-
-Tablet server A is hosting a tablet, let's call it a-tablet.
-
-Tablet server B is hosting a metadata tablet, let's call it m-tablet.
-
-m-tablet records the information about a-tablet, for example, the names of the files it is using to store data.
-
-When A ingests some data, it eventually flushes the updates from memory to a file.
-
-Tablet server A then writes this new information to m-tablet, on Tablet server B.
-
-Here's a likely failure scenario:
-
-Tablet server A does not have enough memory for all the processes running on it.
-The operating system sees a large chunk of the tablet server being unused, and swaps it out to disk to make room for other processes.
-Tablet server A does a java memory garbage collection, which causes it to start using all the memory allocated to it.
-As the server starts pulling data from swap, it runs very slowly.
-It fails to send the keep-alive messages to zookeeper in a timely fashion, and it looses its zookeeper session.
-
-But, it's running so slowly, that it takes a moment to realize it should no longer be hosting tablets.
-
-The thread that is flushing a-tablet memory attempts to update m-tablet with the new file information.
-
-Fortunately there's a constraint on m-tablet.
-Mutations to the metadata table must contain a valid zookeeper session.
-This prevents tablet server A from making updates to m-tablet when it no long has the right to host the tablet.
-
-The "MutationsRejectedException" error is from tablet server A making an update to tablet server B's m-tablet.
-It's getting a constraint violation: tablet server A has lost its zookeeper session, and will fail momentarily.
-
-Ensure that memory is not over-allocated. Monitor swap usage, or turn swap off.
-
-**My accumulo client is getting a MutationsRejectedException. The monitor is displaying "No Such SessionID" errors.**
-
-When your client starts sending mutations to accumulo, it creates a session. Once the session is created,
-mutations are streamed to accumulo, without acknowledgement, against this session. Once the client is done,
-it will close the session, and get an acknowledgement.
-
-If the client fails to communicate with accumulo, it will release the session, assuming that the client has died.
-If the client then attempts to send more mutations against the session, you will see "No Such SessionID" errors on
-the server, and MutationRejectedExceptions in the client.
-
-The client library should be either actively using the connection to the tablet servers,
-or closing the connection and sessions. If the session times out, something is causing your client
-to pause.
-
-The most frequent source of these pauses are java garbage collection pauses
-due to the JVM running out of memory, or being swapped out to disk.
-
-Ensure your client has adequate memory and is not being swapped out to disk.
-
-## Tools
-
-The accumulo script can be used to run classes from the command line.
-This section shows how a few of the utilities work, but there are many
-more.
-
-There's a class that will examine an accumulo storage file and print
-out basic metadata.
-
-```
-$ accumulo org.apache.accumulo.core.file.rfile.PrintInfo /accumulo/tables/1/default_tablet/A000000n.rf
-2013-07-16 08:17:14,778 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
-Locality group : <DEFAULT>
- Start block : 0
- Num blocks : 1
- Index level 0 : 62 bytes 1 blocks
- First key : 288be9ab4052fe9e span:34078a86a723e5d3:3da450f02108ced5 [] 1373373521623 false
- Last key : start:13fc375709e id:615f5ee2dd822d7a [] 1373373821660 false
- Num entries : 466
- Column families : [waitForCommits, start, md major compactor 1, md major compactor 2, md major compactor 3,
- bringOnline, prep, md major compactor 4, md major compactor 5, md root major compactor 3,
- minorCompaction, wal, compactFiles, md root major compactor 4, md root major compactor 1,
- md root major compactor 2, compact, id, client:update, span, update, commit, write,
- majorCompaction]
-
-Meta block : BCFile.index
- Raw size : 4 bytes
- Compressed size : 12 bytes
- Compression type : gz
-
-Meta block : RFile.index
- Raw size : 780 bytes
- Compressed size : 344 bytes
- Compression type : gz
-```
-
-When trying to diagnose problems related to key size, the `PrintInfo` tool can provide a histogram of the individual key sizes:
-
- $ accumulo org.apache.accumulo.core.file.rfile.PrintInfo --histogram /accumulo/tables/1/default_tablet/A000000n.rf
- ...
- Up to size count %-age
- 10 : 222 28.23%
- 100 : 244 71.77%
- 1000 : 0 0.00%
- 10000 : 0 0.00%
- 100000 : 0 0.00%
- 1000000 : 0 0.00%
- 10000000 : 0 0.00%
- 100000000 : 0 0.00%
- 1000000000 : 0 0.00%
- 10000000000 : 0 0.00%
-
-Likewise, `PrintInfo` will dump the key-value pairs and show you the contents of the RFile:
-
- $ accumulo org.apache.accumulo.core.file.rfile.PrintInfo --dump /accumulo/tables/1/default_tablet/A000000n.rf
- row columnFamily:columnQualifier [visibility] timestamp deleteFlag -> Value
- ...
-
-**Accumulo is not showing me any data!**
-
-Do you have your auths set so that it matches your visibilities?
-
-**What are my visibilities?**
-
-Use `PrintInfo` on a representative file to get some idea of the visibilities in the underlying data.
-
-Note that the use of `PrintInfo` is an administrative tool and can only
-by used by someone who can access the underlying Accumulo data. It
-does not provide the normal access controls in Accumulo.
-
-If you would like to backup, or otherwise examine the contents of Zookeeper, there are commands to dump and load to/from XML.
-
- $ accumulo org.apache.accumulo.server.util.DumpZookeeper --root /accumulo >dump.xml
- $ accumulo org.apache.accumulo.server.util.RestoreZookeeper --overwrite < dump.xml
-
-**How can I get the information in the monitor page for my cluster monitoring system?**
-
-Use GetMasterStats:
-
- $ accumulo org.apache.accumulo.test.GetMasterStats | grep Load
- OS Load Average: 0.27
-
-**The monitor page is showing an offline tablet. How can I find out which tablet it is?**
-
-Use FindOfflineTablets:
-
- $ accumulo org.apache.accumulo.server.util.FindOfflineTablets
- 2<<@(null,null,localhost:9997) is UNASSIGNED #walogs:2
-
-Here's what the output means:
-
-* `2<<` -
- This is the tablet from (-inf, pass:[+]inf) for the
- table with id 2. The command `tables -l` in the shell will show table ids for
- tables.
-
-* `@(null, null, localhost:9997)` -
- Location information. The
- format is `@(assigned, hosted, last)`. In this case, the
- tablet has not been assigned, is not hosted anywhere, and was once
- hosted on localhost.
-
-* `#walogs:2` -
- The number of write-ahead logs that this tablet requires for recovery.
-
-An unassigned tablet with write-ahead logs is probably waiting for
-logs to be sorted for efficient recovery.
-
-**How can I be sure that the metadata tables are up and consistent?**
-
-`CheckForMetadataProblems` will verify the start/end of
-every tablet matches, and the start and stop for the table is empty:
-
- $ accumulo org.apache.accumulo.server.util.CheckForMetadataProblems -u root --password
- Enter the connection password:
- All is well for table !0
- All is well for table 1
-
-**My hadoop cluster has lost a file due to a NameNode failure. How can I remove the file?**
-
-There's a utility that will check every file reference and ensure
-that the file exists in HDFS. Optionally, it will remove the
-reference:
-
- $ accumulo org.apache.accumulo.server.util.RemoveEntriesForMissingFiles -u root --password
- Enter the connection password:
- 2013-07-16 13:10:57,293 [util.RemoveEntriesForMissingFiles] INFO : File /accumulo/tables/2/default_tablet/F0000005.rf
- is missing
- 2013-07-16 13:10:57,296 [util.RemoveEntriesForMissingFiles] INFO : 1 files of 3 missing
-
-**I have many entries in zookeeper for old instances I no longer need. How can I remove them?**
-
-Use CleanZookeeper:
-
- $ accumulo org.apache.accumulo.server.util.CleanZookeeper
-
-This command will not delete the instance pointed to by the local `accumulo-site.xml` file.
-
-**I need to decommission a node. How do I stop the tablet server on it?**
-
-Use the admin command:
-
- $ accumulo admin stop hostname:9997
- 2013-07-16 13:15:38,403 [util.Admin] INFO : Stopping server 12.34.56.78:9997
-
-**I cannot login to a tablet server host, and the tablet server will not shut down. How can I kill the server?**
-
-Sometimes you can kill a "stuck" tablet server by deleting its lock in zookeeper:
-
- $ accumulo org.apache.accumulo.server.util.TabletServerLocks --list
- 127.0.0.1:9997 TSERV_CLIENT=127.0.0.1:9997
- $ accumulo org.apache.accumulo.server.util.TabletServerLocks -delete 127.0.0.1:9997
- $ accumulo org.apache.accumulo.server.util.TabletServerLocks -list
- 127.0.0.1:9997 null
-
-You can find the master and instance id for any accumulo instances using the same zookeeper instance:
-
-```
-$ accumulo org.apache.accumulo.server.util.ListInstances
-INFO : Using ZooKeepers localhost:2181
-
- Instance Name | Instance ID | Master
----------------------+--------------------------------------+-------------------------------
- "test" | 6140b72e-edd8-4126-b2f5-e74a8bbe323b | 127.0.0.1:9999
-```
-
-## System Metadata Tables
-
-Accumulo tracks information about tables in metadata tables. The metadata for
-most tables is contained within the metadata table in the accumulo namespace,
-while metadata for that table is contained in the root table in the accumulo
-namespace. The root table is composed of a single tablet, which does not
-split, so it is also called the root tablet. Information about the root
-table, such as its location and write-ahead logs, are stored in ZooKeeper.
-
-Let's create a table and put some data into it:
-
-```
-shell> createtable test
-
-shell> tables -l
-accumulo.metadata => !0
-accumulo.root => `r
-test => 2
-trace => 1
-
-shell> insert a b c d
-
-shell> flush -w
-```
-
-Now let's take a look at the metadata for this table:
-
- shell> table accumulo.metadata
- shell> scan -b 3; -e 3<
- 3< file:/default_tablet/F000009y.rf [] 186,1
- 3< last:13fe86cd27101e5 [] 127.0.0.1:9997
- 3< loc:13fe86cd27101e5 [] 127.0.0.1:9997
- 3< log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995|6
- 3< srv:dir [] /default_tablet
- 3< srv:flush [] 1
- 3< srv:lock [] tservers/127.0.0.1:9997/zlock-0000000001$13fe86cd27101e5
- 3< srv:time [] M1373998392323
- 3< ~tab:~pr [] \x00
-
-Let's decode this little session:
-
-* `scan -b 3; -e 3<` - Every tablet gets its own row. Every row starts with the table id followed by
- `;` or `<`, and followed by the end row split point for that tablet.
-
-* `file:/default_tablet/F000009y.rf [] 186,1` -
- File entry for this tablet. This tablet contains a single file reference. The
- file is `/accumulo/tables/3/default_tablet/F000009y.rf`. It contains 1
- key/value pair, and is 186 bytes long.
-
-* `last:13fe86cd27101e5 [] 127.0.0.1:9997` -
- Last location for this tablet. It was last held on 127.0.0.1:9997, and the
- unique tablet server lock data was `13fe86cd27101e5+. The default balancer
- will tend to put tablets back on their last location.
-
-* `loc:13fe86cd27101e5 [] 127.0.0.1:9997` -
- The current location of this tablet.
-
-* `log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0. ...` -
- This tablet has a reference to a single write-ahead log. This file can be found in
- `/accumulo/wal/127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995`. The value
- of this entry could refer to multiple files. This tablet's data is encoded as
- `6` within the log.
-
-* `srv:dir [] /default_tablet` -
- Files written for this tablet will be placed into
- `/accumulo/tables/3/default_tablet`.
-
-* `srv:flush [] 1` -
- Flush id. This table has successfully completed the flush with the id of +1+.
-
-* `srv:lock [] tservers/127.0.0.1:9997/zlock-0000000001\$13fe86cd27101e5` -
- This is the lock information for the tablet holding the present lock. This
- information is checked against zookeeper whenever this is updated, which
- prevents a metadata update from a tablet server that no longer holds its
- lock.
-
-* `srv:time [] M1373998392323` -
- This indicates the time time type (+M+ for milliseconds or +L+ for logical) and the timestamp of the most recently written key in this tablet. It is used to ensure automatically assigned key timestamps are strictly increasing for the tablet, regardless of the tablet server's system time.
-
-* `~tab:~pr [] \x00` -
- The end-row marker for the previous tablet (prev-row). The first byte
- indicates the presence of a prev-row. This tablet has the range (-inf, `inf),
- so it has no prev-row (or end row).
-
-Besides these columns, you may see:
-
-* `rowId future:zooKeeperID location` -
- Tablet has been assigned to a tablet, but not yet loaded.
-
-* `~del:filename` -
- When a tablet server is done use a file, it will create a delete marker in the appropriate metadata table, unassociated with any tablet. The garbage collector will remove the marker, and the file, when no other reference to the file exists.
-
-* `~blip:txid` -
- Bulk-Load In Progress marker.
-
-* `rowId loaded:filename` -
- A file has been bulk-loaded into this tablet, however the bulk load has not yet completed on other tablets, so this marker prevents the file from being loaded multiple times.
-
-* `rowId !cloned` -
- A marker that indicates that this tablet has been successfully cloned.
-
-* `rowId splitRatio:ratio` -
- A marker that indicates a split is in progress, and the files are being split at the given ratio.
-
-* `rowId chopped` -
- A marker that indicates that the files in the tablet do not contain keys outside the range of the tablet.
-
-* `rowId scan` -
- A marker that prevents a file from being removed while there are still active scans using it.
-
-## Simple System Recovery
-
-**One of my Accumulo processes died. How do I bring it back?**
-
-The easiest way to bring all services online for an Accumulo instance is to run the `accumulo-cluster` script.
-
- $ accumulo-cluster start
-
-This process will check the process listing, using `jps` on each host before attempting to restart a service on the given host.
-Typically, this check is sufficient except in the face of a hung/zombie process. For large clusters, it may be
-undesirable to ssh to every node in the cluster to ensure that all hosts are running the appropriate processes and `accumulo-service` may be of use.
-
- $ ssh host_with_dead_process
- $ accumulo-service tserver start
-
-**My process died again. Should I restart it via `cron` or tools like `supervisord`?**
-
-A repeatedly dying Accumulo process is a sign of a larger problem. Typically these problems are due to a
-misconfiguration of Accumulo or over-saturation of resources. Blind automation of any service restart inside of Accumulo
-is generally an undesirable situation as it is indicative of a problem that is being masked and ignored. Accumulo
-processes should be stable on the order of months and not require frequent restart.
-
-## Advanced System Recovery
-
-### HDFS Failure
-
-**I had disasterous HDFS failure. After bringing everything back up, several tablets refuse to go online.**
-
-Data written to tablets is written into memory before being written into indexed files. In case the server
-is lost before the data is saved into a an indexed file, all data stored in memory is first written into a
-write-ahead log (WAL). When a tablet is re-assigned to a new tablet server, the write-ahead logs are read to
-recover any mutations that were in memory when the tablet was last hosted.
-
-If a write-ahead log cannot be read, then the tablet is not re-assigned. All it takes is for one of
-the blocks in the write-ahead log to be missing. This is unlikely unless multiple data nodes in HDFS have been
-lost.
-
-Get the WAL files online and healthy. Restore any data nodes that may be down.
-
-**How do find out which tablets are offline?**
-
-Use `accumulo admin checkTablets`
-
- $ accumulo admin checkTablets
-
-**I lost three data nodes, and I'm missing blocks in a WAL. I don't care about data loss, how
-can I get those tablets online?**
-
-See the discussion in <<metadata>>, which shows a typical metadata table listing.
-The entries with a column family of `log` are references to the WAL for that tablet.
-If you know what WAL is bad, you can find all the references with a grep in the shell:
-
- shell> grep 0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995
- 3< log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995|6
-
-You can remove the WAL references in the metadata table.
-
- shell> grant -u root Table.WRITE -t accumulo.metadata
- shell> delete 3< log 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995
-
-Note: the colon (`:`) is omitted when specifying the _row cf cq_ for the delete command.
-
-The master will automatically discover the tablet no longer has a bad WAL reference and will
-assign the tablet. You will need to remove the reference from all the tablets to get them
-online.
-
-**The metadata (or root) table has references to a corrupt WAL.**
-
-This is a much more serious state, since losing updates to the metadata table will result
-in references to old files which may not exist, or lost references to new files, resulting
-in tablets that cannot be read, or large amounts of data loss.
-
-The best hope is to restore the WAL by fixing HDFS data nodes and bringing the data back online.
-If this is not possible, the best approach is to re-create the instance and bulk import all files from
-the old instance into a new tables.
-
-A complete set of instructions for doing this is outside the scope of this guide,
-but the basic approach is:
-
-* Use `tables -l` in the shell to discover the table name to table id mapping
-* Stop all accumulo processes on all nodes
-* Move the accumulo directory in HDFS out of the way:
- $ hadoop fs -mv /accumulo /corrupt
-* Re-initalize accumulo
-* Recreate tables, users and permissions
-* Import the directories under `/corrupt/tables/<id>` into the new instance
-
-**One or more HDFS Files under /accumulo/tables are corrupt**
-
-Accumulo maintains multiple references into the tablet files in the metadata
-tables and within the tablet server hosting the file, this makes it difficult to
-reliably just remove those references.
-
-The directory structure in HDFS for tables will follow the general structure:
-
- /accumulo
- /accumulo/tables/
- /accumulo/tables/!0
- /accumulo/tables/!0/default_tablet/A000001.rf
- /accumulo/tables/!0/t-00001/A000002.rf
- /accumulo/tables/1
- /accumulo/tables/1/default_tablet/A000003.rf
- /accumulo/tables/1/t-00001/A000004.rf
- /accumulo/tables/1/t-00001/A000005.rf
- /accumulo/tables/2/default_tablet/A000006.rf
- /accumulo/tables/2/t-00001/A000007.rf
-
-If files under `/accumulo/tables` are corrupt, the best course of action is to
-recover those files in hdsf see the section on HDFS. Once these recovery efforts
-have been exhausted, the next step depends on where the missing file(s) are
-located. Different actions are required when the bad files are in Accumulo data
-table files or if they are metadata table files.
-
-*Data File Corruption*
-
-When an Accumulo data file is corrupt, the most reliable way to restore Accumulo
-operations is to replace the missing file with an ``empty'' file so that
-references to the file in the METADATA table and within the tablet server
-hosting the file can be resolved by Accumulo. An empty file can be created using
-the CreateEmpty utiity:
-
- $ accumulo org.apache.accumulo.core.file.rfile.CreateEmpty /path/to/empty/file/empty.rf
-
-The process is to delete the corrupt file and then move the empty file into its
-place (The generated empty file can be copied and used multiple times if necessary and does not need
-to be regenerated each time)
-
- $ hadoop fs –rm /accumulo/tables/corrupt/file/thename.rf; \
- hadoop fs -mv /path/to/empty/file/empty.rf /accumulo/tables/corrupt/file/thename.rf
-
-*Metadata File Corruption*
-
-If the corrupt files are metadata files, see <<metadata>> (under the path
-`/accumulo/tables/!0`) then you will need to rebuild
-the metadata table by initializing a new instance of Accumulo and then importing
-all of the existing data into the new instance. This is the same procedure as
-recovering from a zookeeper failure (see <<zookeeper_failure>>), except that
-you will have the benefit of having the existing user and table authorizations
-that are maintained in zookeeper.
-
-You can use the DumpZookeeper utility to save this information for reference
-before creating the new instance. You will not be able to use RestoreZookeeper
-because the table names and references are likely to be different between the
-original and the new instances, but it can serve as a reference.
-
-If the files cannot be recovered, replace corrupt data files with a empty
-rfiles to allow references in the metadata table and in the tablet servers to be
-resolved. Rebuild the metadata table if the corrupt files are metadata files.
-
-*Write-Ahead Log(WAL) File Corruption*
-
-In certain versions of Accumulo, a corrupt WAL file (caused by HDFS corruption
-or a bug in Accumulo that created the file) can block the successful recovery
-of one to many Tablets. Accumulo can be stuck in a loop trying to recover the
-WAL file, never being able to succeed.
-
-In the cases where the WAL file's original contents are unrecoverable or some degree
-of data loss is acceptable (beware if the WAL file contains updates to the Accumulo
-metadat table!), the following process can be followed to create an valid, empty
-WAL file. Run the following commands as the Accumulo unix user (to ensure that
-the proper file permissions in HDFS)
-
- $ echo -n -e '--- Log File Header (v2) ---\x00\x00\x00\x00' > empty.wal
-
-The above creates a file with the text "--- Log File Header (v2) ---" and then
-four bytes. You should verify the contents of the file with a hexdump tool.
-
-Then, place this empty WAL in HDFS and then replace the corrupt WAL file in HDFS
-with the empty WAL.
-
- $ hdfs dfs -moveFromLocal empty.wal /user/accumulo/empty.wal
- $ hdfs dfs -mv /user/accumulo/empty.wal /accumulo/wal/tserver-4.example.com+10011/26abec5b-63e7-40dd-9fa1-b8ad2436606e
-
-After the corrupt WAL file has been replaced, the system should automatically recover.
-It may be necessary to restart the Accumulo Master process as an exponential
-backup policy is used which could lead to a long wait before Accumulo will
-try to re-load the WAL file.
-
-### ZooKeeper Failure
-
-**I lost my ZooKeeper quorum (hardware failure), but HDFS is still intact. How can I recover my Accumulo instance?**
-
-ZooKeeper, in addition to its lock-service capabilities, also serves to bootstrap an Accumulo
-instance from some location in HDFS. It contains the pointers to the root tablet in HDFS which
-is then used to load the Accumulo metadata tablets, which then loads all user tables. ZooKeeper
-also stores all namespace and table configuration, the user database, the mapping of table IDs to
-table names, and more across Accumulo restarts.
-
-Presently, the only way to recover such an instance is to initialize a new instance and import all
-of the old data into the new instance. The easiest way to tackle this problem is to first recreate
-the mapping of table ID to table name and then recreate each of those tables in the new instance.
-Set any necessary configuration on the new tables and add some split points to the tables to close
-the gap between how many splits the old table had and no splits.
-
-The directory structure in HDFS for tables will follow the general structure:
-
- /accumulo
- /accumulo/tables/
- /accumulo/tables/1
- /accumulo/tables/1/default_tablet/A000001.rf
- /accumulo/tables/1/t-00001/A000002.rf
- /accumulo/tables/1/t-00001/A000003.rf
- /accumulo/tables/2/default_tablet/A000004.rf
- /accumulo/tables/2/t-00001/A000005.rf
-
-For each table, make a new directory that you can move (or copy if you have the HDFS space to do so)
-all of the rfiles for a given table into. For example, to process the table with an ID of `1`, make a new directory,
-say `/new-table-1` and then copy all files from `/accumulo/tables/1/\*/*.rf` into that directory. Additionally,
-make a directory, `/new-table-1-failures`, for any failures during the import process. Then, issue the import
-command using the Accumulo shell into the new table, telling Accumulo to not re-set the timestamp:
-
- user@instance new_table> importdirectory /new-table-1 /new-table-1-failures false
-
-Any RFiles which were failed to be loaded will be placed in `/new-table-1-failures`. Rfiles that were successfully
-imported will no longer exist in `/new-table-1`. For failures, move them back to the import directory and retry
-the `importdirectory` command.
-
-It is *extremely* important to note that this approach may introduce stale data back into
-the tables. For a few reasons, RFiles may exist in the table directory which are candidates for deletion but have
-not yet been deleted. Additionally, deleted data which was not compacted away, but still exists in write-ahead logs if
-the original instance was somehow recoverable, will be re-introduced in the new instance. Table splits and merges
-(which also include the deleteRows API call on TableOperations, are also vulnerable to this problem. This process should
-*not* be used if these are unacceptable risks. It is possible to try to re-create a view of the `accumulo.metadata`
-table to prune out files that are candidates for deletion, but this is a difficult task that also may not be entirely accurate.
-
-Likewise, it is also possible that data loss may occur from write-ahead log (WAL) files which existed on the old table but
-were not minor-compacted into an RFile. Again, it may be possible to reconstruct the state of these WAL files to
-replay data not yet in an RFile; however, this is a difficult task and is not implemented in any automated fashion.
-
-The `importdirectory` shell command can be used to import RFiles from the old instance into a newly created instance,
-but extreme care should go into the decision to do this as it may result in reintroduction of stale data or the
-omission of new data.
-
-## Upgrade Issues
-
-**I upgraded from 1.4 to 1.5 to 1.6 but still have some WAL files on local disk. Do I have any way to recover them?**
-
-Yes, you can recover them by running the LocalWALRecovery utility (not available in 1.8 and later) on each node that needs recovery performed. The utility
-will default to using the directory specified by `logger.dir.walog` in your configuration, or can be
-overriden by using the `--local-wal-directories` option on the tool. It can be invoked as follows:
-
- accumulo org.apache.accumulo.tserver.log.LocalWALRecovery
-
-## File Naming Conventions
-
-**Why are files named like they are? Why do some start with `C` and others with `F`?**
-
-The file names give you a basic idea for the source of the file.
-
-The base of the filename is a base-36 unique number. All filenames in accumulo are coordinated
-with a counter in zookeeper, so they are always unique, which is useful for debugging.
-
-The leading letter gives you an idea of how the file was created:
-
-* `F` - Flush: entries in memory were written to a file (Minor Compaction)
-* `M` - Merging compaction: entries in memory were combined with the smallest file to create one new file
-* `C` - Several files, but not all files, were combined to produce this file (Major Compaction)
-* `A` - All files were compacted, delete entries were dropped
-* `I` - Bulk import, complete, sorted index files. Always in a directory starting with `b-`
-
-This simple file naming convention allows you to see the basic structure of the files from just
-their filenames, and reason about what should be happening to them next, just
-by scanning their entries in the metadata tables.
-
-For example, if you see multiple files with `M` prefixes, the tablet is, or was, up against its
-maximum file limit, so it began merging memory updates with files to keep the file count reasonable. This
-slows down ingest performance, so knowing there are many files like this tells you that the system
-is struggling to keep up with ingest vs the compaction strategy which reduces the number of files.
-
-## HDFS Decommissioning Issues
-
-**My Hadoop DataNode is hung for hours trying to decommission.**
-
-Write Ahead Logs stay open until they hit the size threshold, which could be many hours or days in some cases. These open files will prevent a DN from finishing its decommissioning process (HDFS-3599) in some versions of Hadoop 2. If you stop the DN, then the WALog file will not be closed and you could lose data. To work around this issue, we now close WALogs on a time period specified by the property `tserver.walog.max.age` with a default period of 24 hours.
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/036341f2/_docs-unreleased/troubleshooting/system-metadata-tables.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/troubleshooting/system-metadata-tables.md b/_docs-unreleased/troubleshooting/system-metadata-tables.md
new file mode 100644
index 0000000..35b0db6
--- /dev/null
+++ b/_docs-unreleased/troubleshooting/system-metadata-tables.md
@@ -0,0 +1,114 @@
+---
+title: System Metadata Tables
+category: troubleshooting
+order: 4
+---
+
+Accumulo tracks information about tables in metadata tables. The metadata for
+most tables is contained within the metadata table in the accumulo namespace,
+while metadata for that table is contained in the root table in the accumulo
+namespace. The root table is composed of a single tablet, which does not
+split, so it is also called the root tablet. Information about the root
+table, such as its location and write-ahead logs, are stored in ZooKeeper.
+
+Let's create a table and put some data into it:
+
+```
+shell> createtable test
+
+shell> tables -l
+accumulo.metadata => !0
+accumulo.root => `r
+test => 2
+trace => 1
+
+shell> insert a b c d
+
+shell> flush -w
+```
+
+Now let's take a look at the metadata for this table:
+
+ shell> table accumulo.metadata
+ shell> scan -b 3; -e 3<
+ 3< file:/default_tablet/F000009y.rf [] 186,1
+ 3< last:13fe86cd27101e5 [] 127.0.0.1:9997
+ 3< loc:13fe86cd27101e5 [] 127.0.0.1:9997
+ 3< log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995|6
+ 3< srv:dir [] /default_tablet
+ 3< srv:flush [] 1
+ 3< srv:lock [] tservers/127.0.0.1:9997/zlock-0000000001$13fe86cd27101e5
+ 3< srv:time [] M1373998392323
+ 3< ~tab:~pr [] \x00
+
+Let's decode this little session:
+
+* `scan -b 3; -e 3<` - Every tablet gets its own row. Every row starts with the table id followed by
+ `;` or `<`, and followed by the end row split point for that tablet.
+
+* `file:/default_tablet/F000009y.rf [] 186,1` -
+ File entry for this tablet. This tablet contains a single file reference. The
+ file is `/accumulo/tables/3/default_tablet/F000009y.rf`. It contains 1
+ key/value pair, and is 186 bytes long.
+
+* `last:13fe86cd27101e5 [] 127.0.0.1:9997` -
+ Last location for this tablet. It was last held on 127.0.0.1:9997, and the
+ unique tablet server lock data was `13fe86cd27101e5+. The default balancer
+ will tend to put tablets back on their last location.
+
+* `loc:13fe86cd27101e5 [] 127.0.0.1:9997` -
+ The current location of this tablet.
+
+* `log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0. ...` -
+ This tablet has a reference to a single write-ahead log. This file can be found in
+ `/accumulo/wal/127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995`. The value
+ of this entry could refer to multiple files. This tablet's data is encoded as
+ `6` within the log.
+
+* `srv:dir [] /default_tablet` -
+ Files written for this tablet will be placed into
+ `/accumulo/tables/3/default_tablet`.
+
+* `srv:flush [] 1` -
+ Flush id. This table has successfully completed the flush with the id of +1+.
+
+* `srv:lock [] tservers/127.0.0.1:9997/zlock-0000000001\$13fe86cd27101e5` -
+ This is the lock information for the tablet holding the present lock. This
+ information is checked against zookeeper whenever this is updated, which
+ prevents a metadata update from a tablet server that no longer holds its
+ lock.
+
+* `srv:time [] M1373998392323` -
+ This indicates the time time type (+M+ for milliseconds or +L+ for logical) and the timestamp of the most recently written key in this tablet. It is used to ensure automatically assigned key timestamps are strictly increasing for the tablet, regardless of the tablet server's system time.
+
+* `~tab:~pr [] \x00` -
+ The end-row marker for the previous tablet (prev-row). The first byte
+ indicates the presence of a prev-row. This tablet has the range (-inf, `inf),
+ so it has no prev-row (or end row).
+
+Besides these columns, you may see:
+
+* `rowId future:zooKeeperID location` -
+ Tablet has been assigned to a tablet, but not yet loaded.
+
+* `~del:filename` -
+ When a tablet server is done use a file, it will create a delete marker in the appropriate metadata table, unassociated with any tablet. The garbage collector will remove the marker, and the file, when no other reference to the file exists.
+
+* `~blip:txid` -
+ Bulk-Load In Progress marker.
+
+* `rowId loaded:filename` -
+ A file has been bulk-loaded into this tablet, however the bulk load has not yet completed on other tablets, so this marker prevents the file from being loaded multiple times.
+
+* `rowId !cloned` -
+ A marker that indicates that this tablet has been successfully cloned.
+
+* `rowId splitRatio:ratio` -
+ A marker that indicates a split is in progress, and the files are being split at the given ratio.
+
+* `rowId chopped` -
+ A marker that indicates that the files in the tablet do not contain keys outside the range of the tablet.
+
+* `rowId scan` -
+ A marker that prevents a file from being removed while there are still active scans using it.
+
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/036341f2/_docs-unreleased/troubleshooting/tools.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/troubleshooting/tools.md b/_docs-unreleased/troubleshooting/tools.md
new file mode 100644
index 0000000..8a0d8ed
--- /dev/null
+++ b/_docs-unreleased/troubleshooting/tools.md
@@ -0,0 +1,134 @@
+---
+title: Troubleshooting Tools
+category: troubleshooting
+order: 3
+---
+
+The `accumulo` command can be used to run classes from the command line.
+
+## PrintInfo
+
+The `PrintInfo` tool will examine an Accumulo storage file and print out basic metadata.
+
+```
+$ accumulo org.apache.accumulo.core.file.rfile.PrintInfo /accumulo/tables/1/default_tablet/A000000n.rf
+2013-07-16 08:17:14,778 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
+Locality group : <DEFAULT>
+ Start block : 0
+ Num blocks : 1
+ Index level 0 : 62 bytes 1 blocks
+ First key : 288be9ab4052fe9e span:34078a86a723e5d3:3da450f02108ced5 [] 1373373521623 false
+ Last key : start:13fc375709e id:615f5ee2dd822d7a [] 1373373821660 false
+ Num entries : 466
+ Column families : [waitForCommits, start, md major compactor 1, md major compactor 2, md major compactor 3,
+ bringOnline, prep, md major compactor 4, md major compactor 5, md root major compactor 3,
+ minorCompaction, wal, compactFiles, md root major compactor 4, md root major compactor 1,
+ md root major compactor 2, compact, id, client:update, span, update, commit, write,
+ majorCompaction]
+
+Meta block : BCFile.index
+ Raw size : 4 bytes
+ Compressed size : 12 bytes
+ Compression type : gz
+
+Meta block : RFile.index
+ Raw size : 780 bytes
+ Compressed size : 344 bytes
+ Compression type : gz
+```
+
+When trying to diagnose problems related to key size, the `PrintInfo` tool can provide a histogram of the individual key sizes:
+
+ $ accumulo org.apache.accumulo.core.file.rfile.PrintInfo --histogram /accumulo/tables/1/default_tablet/A000000n.rf
+ ...
+ Up to size count %-age
+ 10 : 222 28.23%
+ 100 : 244 71.77%
+ 1000 : 0 0.00%
+ 10000 : 0 0.00%
+ 100000 : 0 0.00%
+ 1000000 : 0 0.00%
+ 10000000 : 0 0.00%
+ 100000000 : 0 0.00%
+ 1000000000 : 0 0.00%
+ 10000000000 : 0 0.00%
+
+Likewise, `PrintInfo` will dump the key-value pairs and show you the contents of the RFile:
+
+ $ accumulo org.apache.accumulo.core.file.rfile.PrintInfo --dump /accumulo/tables/1/default_tablet/A000000n.rf
+ row columnFamily:columnQualifier [visibility] timestamp deleteFlag -> Value
+ ...
+
+## GetMasterStats
+
+The `GetMasterStats` tool can be used to retrieve Accumulo state and statistics:
+
+
+ $ accumulo org.apache.accumulo.test.GetMasterStats | grep Load
+ OS Load Average: 0.27
+
+## FindOfflineTablets
+
+If the Accumulo monitor shows an offline tablet, use `FindOfflineTablets` to find out which
+tablet it is.
+
+ $ accumulo org.apache.accumulo.server.util.FindOfflineTablets
+ 2<<@(null,null,localhost:9997) is UNASSIGNED #walogs:2
+
+Here's what the output means:
+
+* `2<<` -
+ This is the tablet from (-inf, pass:[+]inf) for the
+ table with id 2. The command `tables -l` in the shell will show table ids for
+ tables.
+
+* `@(null, null, localhost:9997)` -
+ Location information. The
+ format is `@(assigned, hosted, last)`. In this case, the
+ tablet has not been assigned, is not hosted anywhere, and was once
+ hosted on localhost.
+
+* `#walogs:2` -
+ The number of write-ahead logs that this tablet requires for recovery.
+
+An unassigned tablet with write-ahead logs is probably waiting for
+logs to be sorted for efficient recovery.
+
+## CheckForMetadataProblems
+
+The `CheckForMetadataProblems` tool can be used to make sure metadata
+tables are up and consistent. It will verify the start/end of
+every tablet matches, and the start and stop for the table is empty:
+
+ $ accumulo org.apache.accumulo.server.util.CheckForMetadataProblems -u root --password
+ Enter the connection password:
+ All is well for table !0
+ All is well for table 1
+
+## RemoveEntriesForMissingFiles
+
+If your Hadoop cluster has a lost a file due to a NameNode failure, you can remove the
+the file reference using `RemoveEntriesForMissingFiles`. It will check every file reference
+and ensure that the file exists in HDFS. Optionally, it will remove the reference:
+
+ $ accumulo org.apache.accumulo.server.util.RemoveEntriesForMissingFiles -u root --password
+ Enter the connection password:
+ 2013-07-16 13:10:57,293 [util.RemoveEntriesForMissingFiles] INFO : File /accumulo/tables/2/default_tablet/F0000005.rf
+ is missing
+ 2013-07-16 13:10:57,296 [util.RemoveEntriesForMissingFiles] INFO : 1 files of 3 missing
+
+## CleanZookeeper
+
+If you have entries in zookeeper for old instances that you no longer need, remove them using CleanZookeeper:
+
+ $ accumulo org.apache.accumulo.server.util.CleanZookeeper
+
+This command will not delete the instance pointed to by the local `accumulo-site.xml` file.
+
+## DumpZookeeper & RestoreZookeeper
+
+If you would like to backup, or otherwise examine the contents of Zookeeper, there are commands to dump and load to/from XML.
+
+ $ accumulo org.apache.accumulo.server.util.DumpZookeeper --root /accumulo >dump.xml
+ $ accumulo org.apache.accumulo.server.util.RestoreZookeeper --overwrite < dump.xml
+
[03/10] accumulo-website git commit: Split up long administration and
troubleshooting overview pages
Posted by mw...@apache.org.
Split up long administration and troubleshooting overview pages
Project: http://git-wip-us.apache.org/repos/asf/accumulo-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo-website/commit/036341f2
Tree: http://git-wip-us.apache.org/repos/asf/accumulo-website/tree/036341f2
Diff: http://git-wip-us.apache.org/repos/asf/accumulo-website/diff/036341f2
Branch: refs/heads/master
Commit: 036341f268ca9363b617c182073533db56cd3e1e
Parents: 7e6a7b9
Author: Mike Walch <mw...@apache.org>
Authored: Mon May 22 16:02:36 2017 -0400
Committer: Mike Walch <mw...@apache.org>
Committed: Mon May 22 16:02:36 2017 -0400
----------------------------------------------------------------------
_docs-unreleased/administration/installation.md | 719 +++++++++++
_docs-unreleased/administration/kerberos.md | 2 +-
.../administration/monitoring-metrics.md | 95 ++
_docs-unreleased/administration/overview.md | 1151 ------------------
_docs-unreleased/administration/replication.md | 2 +-
_docs-unreleased/administration/tracing.md | 347 ++++++
_docs-unreleased/troubleshooting/advanced.md | 365 ++++++
_docs-unreleased/troubleshooting/basic.md | 227 ++++
_docs-unreleased/troubleshooting/overview.md | 820 -------------
.../troubleshooting/system-metadata-tables.md | 114 ++
_docs-unreleased/troubleshooting/tools.md | 134 ++
11 files changed, 2003 insertions(+), 1973 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/036341f2/_docs-unreleased/administration/installation.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/administration/installation.md b/_docs-unreleased/administration/installation.md
new file mode 100644
index 0000000..32724ef
--- /dev/null
+++ b/_docs-unreleased/administration/installation.md
@@ -0,0 +1,719 @@
+---
+title: Installation
+category: administration
+order: 1
+---
+
+## Hardware
+
+Because we are running essentially two or three systems simultaneously layered
+across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to
+consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have
+at least one core and 2 - 4 GB each.
+
+One core running HDFS can typically keep 2 to 4 disks busy, so each machine may
+typically have as little as 2 x 300GB disks and as much as 4 x 1TB or 2TB disks.
+
+It is possible to do with less than this, such as with 1u servers with 2 cores and 4GB
+each, but in this case it is recommended to only run up to two processes per
+machine -- i.e. DataNode and TabletServer or DataNode and MapReduce worker but
+not all three. The constraint here is having enough available heap space for all the
+processes on a machine.
+
+## Network
+
+Accumulo communicates via remote procedure calls over TCP/IP for both passing
+data and control messages. In addition, Accumulo uses HDFS clients to
+communicate with HDFS. To achieve good ingest and query performance, sufficient
+network bandwidth must be available between any two machines.
+
+In addition to needing access to ports associated with HDFS and ZooKeeper, Accumulo will
+use the following default ports. Please make sure that they are open, or change
+their value in accumulo-site.xml.
+
+|Port | Description | Property Name
+|-----|-------------|--------------
+|4445 | Shutdown Port (Accumulo MiniCluster) | n/a
+|4560 | Accumulo monitor (for centralized log display) | monitor.port.log4j
+|9995 | Accumulo HTTP monitor | monitor.port.client
+|9997 | Tablet Server | tserver.port.client
+|9998 | Accumulo GC | gc.port.client
+|9999 | Master Server | master.port.client
+|12234 | Accumulo Tracer | trace.port.client
+|42424 | Accumulo Proxy Server | n/a
+|10001 | Master Replication service | master.replication.coordinator.port
+|10002 | TabletServer Replication service | replication.receipt.service.port
+
+In addition, the user can provide `0` and an ephemeral port will be chosen instead. This
+ephemeral port is likely to be unique and not already bound. Thus, configuring ports to
+use `0` instead of an explicit value, should, in most cases, work around any issues of
+running multiple distinct Accumulo instances (or any other process which tries to use the
+same default ports) on the same hardware. Finally, the *.port.client properties will work
+with the port range syntax (M-N) allowing the user to specify a range of ports for the
+service to attempt to bind. The ports in the range will be tried in a 1-up manner starting
+at the low end of the range to, and including, the high end of the range.
+
+## Download Tarball
+
+Download a binary distribution of Accumulo and install it to a directory on a disk with
+sufficient space:
+
+ cd <install directory>
+ tar xzf accumulo-X.Y.Z-bin.tar.gz # Replace 'X.Y.Z' with your Accumulo version
+ cd accumulo-X.Y.Z
+
+Repeat this step on each machine in your cluster. Typically, the same `<install directory>`
+is chosen for all machines in the cluster.
+
+There are four scripts in the `bin/` directory that are used to manage Accumulo:
+
+1. `accumulo` - Runs Accumulo command-line tools and starts Accumulo processes
+2. `accumulo-service` - Runs Accumulo processes as services
+3. `accumulo-cluster` - Manages Accumulo cluster on a single node or several nodes
+4. `accumulo-util` - Accumulo utilities for creating configuration, native libraries, etc.
+
+These scripts will be used in the remaining instructions to configure and run Accumulo.
+
+## Dependencies
+
+Accumulo requires HDFS and ZooKeeper to be configured and running
+before starting. Password-less SSH should be configured between at least the
+Accumulo master and TabletServer machines. It is also a good idea to run Network
+Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of
+sync, which can cause problems with automatically timestamped data.
+
+## Configuration
+
+The Accumulo tarball contains a `conf/` directory where Accumulo looks for configuration. If you
+installed Accumulo using downstream packaging, the `conf/` could be something else like
+`/etc/accumulo/`.
+
+Before starting Accumulo, the configuration files `accumulo-env.sh` and `accumulo-site.xml` must
+exist in `conf/` and be properly configured. If you are using `accumulo-cluster` to launch
+a cluster, the `conf/` directory must also contain hosts file for Accumulo services (i.e `gc`,
+`masters`, `monitor`, `tservers`, `tracers`). You can either create these files manually or run
+`accumulo-cluster create-config`.
+
+Logging is configured in `accumulo-env.sh` to use three log4j configuration files in `conf/`. The
+file used depends on the Accumulo command or service being run. Logging for most Accumulo services
+(i.e Master, TabletServer, Garbage Collector) is configured by `log4j-service.properties` except for
+the Monitor which is configured by `log4j-monitor.properties`. All Accumulo commands (i.e `init`,
+`shell`, etc) are configured by `log4j.properties`.
+
+### Configure accumulo-env.sh
+
+Accumulo needs to know where to find the software it depends on. Edit accumulo-env.sh
+and specify the following:
+
+1. Enter the location of Hadoop for `$HADOOP_PREFIX`
+2. Enter the location of ZooKeeper for `$ZOOKEEPER_HOME`
+3. Optionally, choose a different location for Accumulo logs using `$ACCUMULO_LOG_DIR`
+
+Accumulo uses `HADOOP_PREFIX` and `ZOOKEEPER_HOME` to locate Hadoop and Zookeeper jars
+and add them the `CLASSPATH` variable. If you are running a vendor-specific release of Hadoop
+or Zookeeper, you may need to change how your `CLASSPATH` is built in `accumulo-env.sh`. If
+Accumulo has problems later on finding jars, run `accumulo classpath -d` to debug and print
+Accumulo's classpath.
+
+You may want to change the default memory settings for Accumulo's TabletServer which are
+by set in the `JAVA_OPTS` settings for 'tservers' in `accumulo-env.sh`. Note the
+syntax is that of the Java JVM command line options. This value should be less than the
+physical memory of the machines running TabletServers.
+
+There are similar options for the master's memory usage and the garbage collector
+process. Reduce these if they exceed the physical RAM of your hardware and
+increase them, within the bounds of the physical RAM, if a process fails because of
+insufficient memory.
+
+Note that you will be specifying the Java heap space in accumulo-env.sh. You should
+make sure that the total heap space used for the Accumulo tserver and the Hadoop
+DataNode and TaskTracker is less than the available memory on each worker node in
+the cluster. On large clusters, it is recommended that the Accumulo master, Hadoop
+NameNode, secondary NameNode, and Hadoop JobTracker all be run on separate
+machines to allow them to use more heap space. If you are running these on the
+same machine on a small cluster, likewise make sure their heap space settings fit
+within the available memory.
+
+### Native Map
+
+The tablet server uses a data structure called a MemTable to store sorted key/value
+pairs in memory when they are first received from the client. When a minor compaction
+occurs, this data structure is written to HDFS. The MemTable will default to using
+memory in the JVM but a JNI version, called the native map, can be used to significantly
+speed up performance by utilizing the memory space of the native operating system. The
+native map also avoids the performance implications brought on by garbage collection
+in the JVM by causing it to pause much less frequently.
+
+#### Building
+
+32-bit and 64-bit Linux and Mac OS X versions of the native map can be built by executing
+`accumulo-util build-native`. If your system's default compiler options are insufficient,
+you can add additional compiler options to the command line, such as options for the
+architecture. These will be passed to the Makefile in the environment variable `USERFLAGS`.
+
+Examples:
+
+ accumulo-util build-native
+ accumulo-util build-native -m32
+
+After building the native map from the source, you will find the artifact in
+`lib/native`. Upon starting up, the tablet server will look
+in this directory for the map library. If the file is renamed or moved from its
+target directory, the tablet server may not be able to find it. The system can
+also locate the native maps shared library by setting `LD_LIBRARY_PATH`
+(or `DYLD_LIBRARY_PATH` on Mac OS X) in `accumulo-env.sh`.
+
+#### Native Maps Configuration
+
+As mentioned, Accumulo will use the native libraries if they are found in the expected
+location and `tserver.memory.maps.native.enabled` is set to `true` (which is the default).
+Using the native maps over JVM Maps nets a noticeable improvement in ingest rates; however,
+certain configuration variables are important to modify when increasing the size of the
+native map.
+
+To adjust the size of the native map, increase the value of `tserver.memory.maps.max`.
+By default, the maximum size of the native map is 1GB. When increasing this value, it is
+also important to adjust the values of `table.compaction.minor.logs.threshold` and
+`tserver.walog.max.size`. `table.compaction.minor.logs.threshold` is the maximum
+number of write-ahead log files that a tablet can reference before they will be automatically
+minor compacted. `tserver.walog.max.size` is the maximum size of a write-ahead log.
+
+The maximum size of the native maps for a server should be less than the product
+of the write-ahead log maximum size and minor compaction threshold for log files:
+
+`$table.compaction.minor.logs.threshold * $tserver.walog.max.size >= $tserver.memory.maps.max`
+
+This formula ensures that minor compactions won't be automatically triggered before the native
+maps can be completely saturated.
+
+Subsequently, when increasing the size of the write-ahead logs, it can also be important
+to increase the HDFS block size that Accumulo uses when creating the files for the write-ahead log.
+This is controlled via `tserver.wal.blocksize`. A basic recommendation is that when
+`tserver.walog.max.size` is larger than 2GB in size, set `tserver.wal.blocksize` to 2GB.
+Increasing the block size to a value larger than 2GB can result in decreased write
+performance to the write-ahead log file which will slow ingest.
+
+### Cluster Specification
+
+If you are using `accumulo-cluster` to start a cluster, configure the following on the
+machine that will serve as the Accumulo master:
+
+1. Write the IP address or domain name of the Accumulo Master to the `conf/masters` file.
+2. Write the IP addresses or domain name of the machines that will be TabletServers in `conf/tservers`, one per line.
+
+Note that if using domain names rather than IP addresses, DNS must be configured
+properly for all machines participating in the cluster. DNS can be a confusing source
+of errors.
+
+### Configure accumulo-site.xml
+
+Specify appropriate values for the following settings in `accumulo-site.xml`:
+
+```xml
+<property>
+ <name>instance.zookeeper.host</name>
+ <value>zooserver-one:2181,zooserver-two:2181</value>
+ <description>list of zookeeper servers</description>
+</property>
+```
+
+This enables Accumulo to find ZooKeeper. Accumulo uses ZooKeeper to coordinate
+settings between processes and helps finalize TabletServer failure.
+
+```xml
+<property>
+ <name>instance.secret</name>
+ <value>DEFAULT</value>
+</property>
+```
+
+The instance needs a secret to enable secure communication between servers. Configure your
+secret and make sure that the `accumulo-site.xml` file is not readable to other users.
+For alternatives to storing the `instance.secret` in plaintext, please read the
+`Sensitive Configuration Values` section.
+
+Some settings can be modified via the Accumulo shell and take effect immediately, but
+some settings require a process restart to take effect. See the [configuration management][config-mgmt]
+documentation for details.
+
+### Hostnames in configuration files
+
+Accumulo has a number of configuration files which can contain references to other hosts in your
+network. All of the "host" configuration files for Accumulo (`gc`, `masters`, `tservers`, `monitor`,
+`tracers`) as well as `instance.volumes` in accumulo-site.xml must contain some host reference.
+
+While IP address, short hostnames, or fully qualified domain names (FQDN) are all technically valid, it
+is good practice to always use FQDNs for both Accumulo and other processes in your Hadoop cluster.
+Failing to consistently use FQDNs can have unexpected consequences in how Accumulo uses the FileSystem.
+
+A common way for this problem can be observed is via applications that use Bulk Ingest. The Accumulo
+Master coordinates moving the input files to Bulk Ingest to an Accumulo-managed directory. However,
+Accumulo cannot safely move files across different Hadoop FileSystems. This is problematic because
+Accumulo also cannot make reliable assertions across what is the same FileSystem which is specified
+with different names. Naively, while 127.0.0.1:8020 might be a valid identifier for an HDFS instance,
+Accumulo identifies `localhost:8020` as a different HDFS instance than `127.0.0.1:8020`.
+
+### Deploy Configuration
+
+Copy accumulo-env.sh and accumulo-site.xml from the `conf/` directory on the master to all Accumulo
+tablet servers. The "host" configuration files files `accumulo-cluster` only need to be on servers
+where that command is run.
+
+### Sensitive Configuration Values
+
+Accumulo has a number of properties that can be specified via the accumulo-site.xml
+file which are sensitive in nature, instance.secret and trace.token.property.password
+are two common examples. Both of these properties, if compromised, have the ability
+to result in data being leaked to users who should not have access to that data.
+
+In Hadoop-2.6.0, a new CredentialProvider class was introduced which serves as a common
+implementation to abstract away the storage and retrieval of passwords from plaintext
+storage in configuration files. Any Property marked with the `Sensitive` annotation
+is a candidate for use with these CredentialProviders. For version of Hadoop which lack
+these classes, the feature will just be unavailable for use.
+
+A comma separated list of CredentialProviders can be configured using the Accumulo Property
+`general.security.credential.provider.paths`. Each configured URL will be consulted
+when the Configuration object for accumulo-site.xml is accessed.
+
+### Using a JavaKeyStoreCredentialProvider for storage
+
+One of the implementations provided in Hadoop-2.6.0 is a Java KeyStore CredentialProvider.
+Each entry in the KeyStore is the Accumulo Property key name. For example, to store the
+`instance.secret`, the following command can be used:
+
+ hadoop credential create instance.secret --provider jceks://file/etc/accumulo/conf/accumulo.jceks
+
+The command will then prompt you to enter the secret to use and create a keystore in:
+
+ /path/to/accumulo/conf/accumulo.jceks
+
+Then, accumulo-site.xml must be configured to use this KeyStore as a CredentialProvider:
+
+```xml
+<property>
+ <name>general.security.credential.provider.paths</name>
+ <value>jceks://file/path/to/accumulo/conf/accumulo.jceks</value>
+</property>
+```
+
+This configuration will then transparently extract the `instance.secret` from
+the configured KeyStore and alleviates a human readable storage of the sensitive
+property.
+
+A KeyStore can also be stored in HDFS, which will make the KeyStore readily available to
+all Accumulo servers. If the local filesystem is used, be aware that each Accumulo server
+will expect the KeyStore in the same location.
+
+### Client Configuration
+
+In version 1.6.0, Accumulo included a new type of configuration file known as a client
+configuration file. One problem with the traditional "site.xml" file that is prevalent
+through Hadoop is that it is a single file used by both clients and servers. This makes
+it very difficult to protect secrets that are only meant for the server processes while
+allowing the clients to connect to the servers.
+
+The client configuration file is a subset of the information stored in accumulo-site.xml
+meant only for consumption by clients of Accumulo. By default, Accumulo checks a number
+of locations for a client configuration by default:
+
+* `/path/to/accumulo/conf/client.conf`
+* `/etc/accumulo/client.conf`
+* `/etc/accumulo/conf/client.conf`
+* `~/.accumulo/config`
+
+These files are [Java Properties files](https://en.wikipedia.org/wiki/.properties). These files
+can currently contain information about ZooKeeper servers, RPC properties (such as SSL or SASL
+connectors), distributed tracing properties. Valid properties are defined by the [ClientProperty](https://github.com/apache/accumulo/blob/f1d0ec93d9f13ff84844b5ac81e4a7b383ced467/core/src/main/java/org/apache/accumulo/core/client/ClientConfiguration.java#L54)
+enum contained in the client API.
+
+#### Custom Table Tags
+
+Accumulo has the ability for users to add custom tags to tables. This allows
+applications to set application-level metadata about a table. These tags can be
+anything from a table description, administrator notes, date created, etc.
+This is done by naming and setting a property with a prefix `table.custom.*`.
+
+Currently, table properties are stored in ZooKeeper. This means that the number
+and size of custom properties should be restricted on the order of 10's of properties
+at most without any properties exceeding 1MB in size. ZooKeeper's performance can be
+very sensitive to an excessive number of nodes and the sizes of the nodes. Applications
+which leverage the user of custom properties should take these warnings into
+consideration. There is no enforcement of these warnings via the API.
+
+#### Configuring the ClassLoader
+
+Accumulo builds its Java classpath in `accumulo-env.sh`. After an Accumulo application has started, it will load classes from the locations
+specified in the deprecated `general.classpaths` property. Additionally, Accumulo will load classes from the locations specified in the
+`general.dynamic.classpaths` property and will monitor and reload them if they change. The reloading feature is useful during the development
+and testing of iterators as new or modified iterator classes can be deployed to Accumulo without having to restart the database.
+/
+Accumulo also has an alternate configuration for the classloader which will allow it to load classes from remote locations. This mechanism
+uses Apache Commons VFS which enables locations such as http and hdfs to be used. This alternate configuration also uses the
+`general.classpaths` property in the same manner described above. It differs in that you need to configure the
+`general.vfs.classpaths` property instead of the `general.dynamic.classpath` property. As in the default configuration, this alternate
+configuration will also monitor the vfs locations for changes and reload if necessary.
+
+The Accumulo classpath can be viewed in human readable format by running `accumulo classpath -d`.
+
+##### ClassLoader Contexts
+
+With the addition of the VFS based classloader, we introduced the notion of classloader contexts. A context is identified
+by a name and references a set of locations from which to load classes and can be specified in the accumulo-site.xml file or added
+using the `config` command in the shell. Below is an example for specify the app1 context in the accumulo-site.xml file:
+
+```xml
+<property>
+ <name>general.vfs.context.classpath.app1</name>
+ <value>hdfs://localhost:8020/applicationA/classpath/.*.jar,file:///opt/applicationA/lib/.*.jar</value>
+ <description>Application A classpath, loads jars from HDFS and local file system</description>
+</property>
+```
+
+The default behavior follows the Java ClassLoader contract in that classes, if they exists, are loaded from the parent classloader first.
+You can override this behavior by delegating to the parent classloader after looking in this classloader first. An example of this
+configuration is:
+
+```xml
+<property>
+ <name>general.vfs.context.classpath.app1.delegation=post</name>
+ <value>hdfs://localhost:8020/applicationA/classpath/.*.jar,file:///opt/applicationA/lib/.*.jar</value>
+ <description>Application A classpath, loads jars from HDFS and local file system</description>
+</property>
+```
+
+To use contexts in your application you can set the `table.classpath.context` on your tables or use the `setClassLoaderContext()` method on Scanner
+and BatchScanner passing in the name of the context, app1 in the example above. Setting the property on the table allows your minc, majc, and scan
+iterators to load classes from the locations defined by the context. Passing the context name to the scanners allows you to override the table setting
+to load only scan time iterators from a different location.
+
+## Initialization
+
+Accumulo must be initialized to create the structures it uses internally to locate
+data across the cluster. HDFS is required to be configured and running before
+Accumulo can be initialized.
+
+Once HDFS is started, initialization can be performed by executing
+`accumulo init` . This script will prompt for a name
+for this instance of Accumulo. The instance name is used to identify a set of tables
+and instance-specific settings. The script will then write some information into
+HDFS so Accumulo can start properly.
+
+The initialization script will prompt you to set a root password. Once Accumulo is
+initialized it can be started.
+
+## Running
+
+### Starting Accumulo
+
+Make sure Hadoop is configured on all of the machines in the cluster, including
+access to a shared HDFS instance. Make sure HDFS and ZooKeeper are running.
+Make sure ZooKeeper is configured and running on at least one machine in the
+cluster.
+Start Accumulo using `accumulo-cluster start`.
+
+To verify that Accumulo is running, check the [Accumulo monitor][monitor].
+In addition, the Shell can provide some information about the status of tables via reading the metadata tables.
+
+### Stopping Accumulo
+
+To shutdown cleanly, run `accumulo-cluster stop` and the master will orchestrate the
+shutdown of all the tablet servers. Shutdown waits for all minor compactions to finish, so it may
+take some time for particular configurations.
+
+### Adding a Tablet Server
+
+Update your `conf/tservers` file to account for the addition.
+
+Next, ssh to each of the hosts you want to add and run:
+
+ accumulo-service tserver start
+
+Make sure the host in question has the new configuration, or else the tablet
+server won't start; at a minimum this needs to be on the host(s) being added,
+but in practice it's good to ensure consistent configuration across all nodes.
+
+### Decomissioning a Tablet Server
+
+If you need to take a node out of operation, you can trigger a graceful shutdown of a tablet
+server. Accumulo will automatically rebalance the tablets across the available tablet servers.
+
+ accumulo admin stop <host(s)> {<host> ...}
+
+Alternatively, you can ssh to each of the hosts you want to remove and run:
+
+ accumulo-service tserver stop
+
+Be sure to update your `conf/tservers` file to
+account for the removal of these hosts. Bear in mind that the monitor will not re-read the
+tservers file automatically, so it will report the decommissioned servers as down; it's
+recommended that you restart the monitor so that the node list is up to date.
+
+The steps described to decommission a node can also be used (without removal of the host
+from the `conf/tservers` file) to gracefully stop a node. This will
+ensure that the tabletserver is cleanly stopped and recovery will not need to be performed
+when the tablets are re-hosted.
+
+### Restarting process on a node
+
+Occasionally, it might be necessary to restart the processes on a specific node. In addition
+to the `accumulo-cluster` script, Accumulo has a `accumulo-service` script that
+can be use to start/stop processes on a node.
+
+#### A note on rolling restarts
+
+For sufficiently large Accumulo clusters, restarting multiple TabletServers within a short window can place significant
+load on the Master server. If slightly lower availability is acceptable, this load can be reduced by globally setting
+`table.suspend.duration` to a positive value.
+
+With `table.suspend.duration` set to, say, `5m`, Accumulo will wait
+for 5 minutes for any dead TabletServer to return before reassigning that TabletServer's responsibilities to other TabletServers.
+If the TabletServer returns to the cluster before the specified timeout has elapsed, Accumulo will assign the TabletServer
+its original responsibilities.
+
+It is important not to choose too large a value for `table.suspend.duration`, as during this time, all scans against the
+data that TabletServer had hosted will block (or time out).
+
+### Running multiple TabletServers on a single node
+
+With very powerful nodes, it may be beneficial to run more than one TabletServer on a given
+node. This decision should be made carefully and with much deliberation as Accumulo is designed
+to be able to scale to using 10's of GB of RAM and 10's of CPU cores.
+
+Accumulo TabletServers bind certain ports on the host to accommodate remote procedure calls to/from
+other nodes. Running more than one TabletServer on a host requires that you set the environment variable
+`ACCUMULO_SERVICE_INSTANCE` to an instance number (i.e 1, 2) for each instance that is started. Also, set
+these properties in `accumulo-site.xml`:
+
+```xml
+ <property>
+ <name>tserver.port.search</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>replication.receipt.service.port</name>
+ <value>0</value>
+ </property>
+```
+
+## Logging
+
+Accumulo processes each write to a set of log files. By default, these logs are found at directory
+set by `ACCUMULO_LOG_DIR` in `accumulo-env.sh`.
+
+## Recovery
+
+In the event of TabletServer failure or error on shutting Accumulo down, some
+mutations may not have been minor compacted to HDFS properly. In this case,
+Accumulo will automatically reapply such mutations from the write-ahead log
+either when the tablets from the failed server are reassigned by the Master (in the
+case of a single TabletServer failure) or the next time Accumulo starts (in the event of
+failure during shutdown).
+
+Recovery is performed by asking a tablet server to sort the logs so that tablets can easily find their missing
+updates. The sort status of each file is displayed on
+Accumulo monitor status page. Once the recovery is complete any
+tablets involved should return to an `online` state. Until then those tablets will be
+unavailable to clients.
+
+The Accumulo client library is configured to retry failed mutations and in many
+cases clients will be able to continue processing after the recovery process without
+throwing an exception.
+
+## Migrating Accumulo from non-HA Namenode to HA Namenode
+
+The following steps will allow a non-HA instance to be migrated to an HA instance. Consider an HDFS URL
+`hdfs://namenode.example.com:8020` which is going to be moved to `hdfs://nameservice1`.
+
+Before moving HDFS over to the HA namenode, use `accumulo admin volumes` to confirm
+that the only volume displayed is the volume from the current namenode's HDFS URL.
+
+ Listing volumes referenced in zookeeper
+ Volume : hdfs://namenode.example.com:8020/accumulo
+
+ Listing volumes referenced in accumulo.root tablets section
+ Volume : hdfs://namenode.example.com:8020/accumulo
+ Listing volumes referenced in accumulo.root deletes section (volume replacement occurrs at deletion time)
+
+ Listing volumes referenced in accumulo.metadata tablets section
+ Volume : hdfs://namenode.example.com:8020/accumulo
+
+ Listing volumes referenced in accumulo.metadata deletes section (volume replacement occurrs at deletion time)
+
+After verifying the current volume is correct, shut down the cluster and transition HDFS to the HA nameservice.
+
+Edit `accumulo-site.xml` to notify accumulo that a volume is being replaced. First,
+add the new nameservice volume to the `instance.volumes` property. Next, add the
+`instance.volumes.replacements` property in the form of `old new`. It's important to not include
+the volume that's being replaced in `instance.volumes`, otherwise it's possible accumulo could continue
+to write to the volume.
+
+```xml
+<!-- instance.dfs.uri and instance.dfs.dir should not be set-->
+<property>
+ <name>instance.volumes</name>
+ <value>hdfs://nameservice1/accumulo</value>
+</property>
+<property>
+ <name>instance.volumes.replacements</name>
+ <value>hdfs://namenode.example.com:8020/accumulo hdfs://nameservice1/accumulo</value>
+</property>
+```
+
+Run `accumulo init --add-volumes` and start up the accumulo cluster. Verify that the
+new nameservice volume shows up with `accumulo admin volumes`.
+
+ Listing volumes referenced in zookeeper
+ Volume : hdfs://namenode.example.com:8020/accumulo
+ Volume : hdfs://nameservice1/accumulo
+
+ Listing volumes referenced in accumulo.root tablets section
+ Volume : hdfs://namenode.example.com:8020/accumulo
+ Volume : hdfs://nameservice1/accumulo
+ Listing volumes referenced in accumulo.root deletes section (volume replacement occurrs at deletion time)
+
+ Listing volumes referenced in accumulo.metadata tablets section
+ Volume : hdfs://namenode.example.com:8020/accumulo
+ Volume : hdfs://nameservice1/accumulo
+ Listing volumes referenced in accumulo.metadata deletes section (volume replacement occurrs at deletion time)
+
+Some erroneous GarbageCollector messages may still be seen for a small period while data is transitioning to
+the new volumes. This is expected and can usually be ignored.
+
+## Achieving Stability in a VM Environment
+
+For testing, demonstration, and even operation uses, Accumulo is often
+installed and run in a virtual machine (VM) environment. The majority of
+long-term operational uses of Accumulo are on bare-metal cluster. However, the
+core design of Accumulo and its dependencies do not preclude running stably for
+long periods within a VM. Many of Accumulo’s operational robustness features to
+handle failures like periodic network partitioning in a large cluster carry
+over well to VM environments. This guide covers general recommendations for
+maximizing stability in a VM environment, including some of the common failure
+modes that are more common when running in VMs.
+
+### Known failure modes: Setup and Troubleshooting
+
+In addition to the general failure modes of running Accumulo, VMs can introduce a
+couple of environmental challenges that can affect process stability. Clock
+drift is something that is more common in VMs, especially when VMs are
+suspended and resumed. Clock drift can cause Accumulo servers to assume that
+they have lost connectivity to the other Accumulo processes and/or lose their
+locks in Zookeeper. VM environments also frequently have constrained resources,
+such as CPU, RAM, network, and disk throughput and capacity. Accumulo generally
+deals well with constrained resources from a stability perspective (optimizing
+performance will require additional tuning, which is not covered in this
+section), however there are some limits.
+
+#### Physical Memory
+
+One of those limits has to do with the Linux out of memory killer. A common
+failure mode in VM environments (and in some bare metal installations) is when
+the Linux out of memory killer decides to kill processes in order to avoid a
+kernel panic when provisioning a memory page. This often happens in VMs due to
+the large number of processes that must run in a small memory footprint. In
+addition to the Linux core processes, a single-node Accumulo setup requires a
+Hadoop Namenode, a Hadoop Secondary Namenode a Hadoop Datanode, a Zookeeper
+server, an Accumulo Master, an Accumulo GC and an Accumulo TabletServer.
+Typical setups also include an Accumulo Monitor, an Accumulo Tracer, a Hadoop
+ResourceManager, a Hadoop NodeManager, provisioning software, and client
+applications. Between all of these processes, it is not uncommon to
+over-subscribe the available RAM in a VM. We recommend setting up VMs without
+swap enabled, so rather than performance grinding to a halt when physical
+memory is exhausted the kernel will randomly* select processes to kill in order
+to free up memory.
+
+Calculating the maximum possible memory usage is essential in creating a stable
+Accumulo VM setup. Safely engineering memory allocation for stability is a
+matter of then bringing the calculated maximum memory usage under the physical
+memory by a healthy margin. The margin is to account for operating system-level
+operations, such as managing process, maintaining virtual memory pages, and
+file system caching. When the java out-of-memory killer finds your process, you
+will probably only see evidence of that in /var/log/messages. Out-of-memory
+process kills do not show up in Accumulo or Hadoop logs.
+
+To calculate the max memory usage of all java virtual machine (JVM) processes
+add the maximum heap size (often limited by a -Xmx... argument, such as in
+accumulo-site.xml) and the off-heap memory usage. Off-heap memory usage
+includes the following:
+
+* "Permanent Space", where the JVM stores Classes, Methods, and other code elements. This can be limited by a JVM flag such as `-XX:MaxPermSize:100m`, and is typically tens of megabytes.
+* Code generation space, where the JVM stores just-in-time compiled code. This is typically small enough to ignore
+* Socket buffers, where the JVM stores send and receive buffers for each socket.
+* Thread stacks, where the JVM allocates memory to manage each thread.
+* Direct memory space and JNI code, where applications can allocate memory outside of the JVM-managed space. For Accumulo, this includes the native in-memory maps that are allocated with the memory.maps.max parameter in accumulo-site.xml.
+* Garbage collection space, where the JVM stores information used for garbage collection.
+
+You can assume that each Hadoop and Accumulo process will use ~100-150MB for
+Off-heap memory, plus the in-memory map of the Accumulo TServer process. A
+simple calculation for physical memory requirements follows:
+
+```
+ Physical memory needed
+ = (per-process off-heap memory) + (heap memory) + (other processes) + (margin)
+ = (number of java processes * 150M + native map) + (sum of -Xmx settings for java process) + (total applications memory, provisioning memory, etc.) + (1G)
+ = (11*150M +500M) + (1G +1G +1G +256M +1G +256M +512M +512M +512M +512M +512M) + (2G) + (1G)
+ = (2150M) + (7G) + (2G) + (1G)
+ = ~12GB
+```
+
+These calculations can add up quickly with the large number of processes,
+especially in constrained VM environments. To reduce the physical memory
+requirements, it is a good idea to reduce maximum heap limits and turn off
+unnecessary processes. If you're not using YARN in your application, you can
+turn off the ResourceManager and NodeManager. If you're not expecting to
+re-provision the cluster frequently you can turn off or reduce provisioning
+processes such as Salt Stack minions and masters.
+
+#### Disk Space
+
+Disk space is primarily used for two operations: storing data and storing logs.
+While Accumulo generally stores all of its key/value data in HDFS, Accumulo,
+Hadoop, and Zookeeper all store a significant amount of logs in a directory on
+a local file system. Care should be taken to make sure that (a) limitations to
+the amount of logs generated are in place, and (b) enough space is available to
+host the generated logs on the partitions that they are assigned. When space is
+not available to log, processes will hang. This can cause interruptions in
+availability of Accumulo, as well as cascade into failures of various
+processes.
+
+Hadoop, Accumulo, and Zookeeper use log4j as a logging mechanism, and each of
+them has a way of limiting the logs and directing them to a particular
+directory. Logs are generated independently for each process, so when
+considering the total space you need to add up the maximum logs generated by
+each process. Typically, a rolling log setup in which each process can generate
+something like 10 100MB files is instituted, resulting in a maximum file system
+usage of 1GB per process. Default setups for Hadoop and Zookeeper are often
+unbounded, so it is important to set these limits in the logging configuration
+files for each subsystem. Consult the user manual for each system for
+instructions on how to limit generated logs.
+
+#### Zookeeper Interaction
+
+Accumulo is designed to scale up to thousands of nodes. At that scale,
+intermittent interruptions in network service and other rare failures of
+compute nodes become more common. To limit the impact of node failures on
+overall service availability, Accumulo uses a heartbeat monitoring system that
+leverages Zookeeper's ephemeral locks. There are several conditions that can
+occur that cause Accumulo process to lose their Zookeeper locks, some of which
+are true interruptions to availability and some of which are false positives.
+Several of these conditions become more common in VM environments, where they
+can be exacerbated by resource constraints and clock drift.
+
+#### Tested Versions
+
+Each release of Accumulo is built with a specific version of Apache
+Hadoop, Apache ZooKeeper and Apache Thrift. We expect Accumulo to
+work with versions that are API compatible with those versions.
+However this compatibility is not guaranteed because Hadoop, ZooKeeper
+and Thift may not provide guarantees between their own versions. We
+have also found that certain versions of Accumulo and Hadoop included
+bugs that greatly affected overall stability. Thrift is particularly
+prone to compatibility changes between versions and you must use the
+same version your Accumulo is built with.
+
+Please check the release notes for your Accumulo version or use the
+mailing lists at https://accumulo.apache.org for more info.
+
+[monitor]: {{page.docs_baseurl}}/administration/monitoring-metrics#monitor
+[config-mgmt]: {{page.docs_baseurl}}/administration/configuration-management
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/036341f2/_docs-unreleased/administration/kerberos.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/administration/kerberos.md b/_docs-unreleased/administration/kerberos.md
index 45cd4f9..d3a041b 100644
--- a/_docs-unreleased/administration/kerberos.md
+++ b/_docs-unreleased/administration/kerberos.md
@@ -1,7 +1,7 @@
---
title: Kerberos
category: administration
-order: 4
+order: 9
---
## Overview
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/036341f2/_docs-unreleased/administration/monitoring-metrics.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/administration/monitoring-metrics.md b/_docs-unreleased/administration/monitoring-metrics.md
new file mode 100644
index 0000000..afb8df0
--- /dev/null
+++ b/_docs-unreleased/administration/monitoring-metrics.md
@@ -0,0 +1,95 @@
+---
+title: Monitoring & Metrics
+category: administration
+order: 4
+---
+
+## Monitoring
+
+### Accumulo Monitor
+
+The Accumulo Monitor provides an interface for monitoring the status and health of
+Accumulo components. The Accumulo Monitor provides a web UI for accessing this information at
+`http://_monitorhost_:9995/`.
+
+Things highlighted in yellow may be in need of attention.
+If anything is highlighted in red on the monitor page, it is something that definitely needs attention.
+
+The Overview page contains some summary information about the Accumulo instance, including the version, instance name, and instance ID.
+There is a table labeled Accumulo Master with current status, a table listing the active Zookeeper servers, and graphs displaying various metrics over time.
+These include ingest and scan performance and other useful measurements.
+
+The Master Server, Tablet Servers, and Tables pages display metrics grouped in different ways (e.g. by tablet server or by table).
+Metrics typically include number of entries (key/value pairs), ingest and query rates.
+The number of running scans, major and minor compactions are in the form _number_running_ (_number_queued_).
+Another important metric is hold time, which is the amount of time a tablet has been waiting but unable to flush its memory in a minor compaction.
+
+The Server Activity page graphically displays tablet server status, with each server represented as a circle or square.
+Different metrics may be assigned to the nodes' color and speed of oscillation.
+The Overall Avg metric is only used on the Server Activity page, and represents the average of all the other metrics (after normalization).
+Similarly, the Overall Max metric picks the metric with the maximum normalized value.
+
+The Garbage Collector page displays a list of garbage collection cycles, the number of files found of each type (including deletion candidates in use and files actually deleted), and the length of the deletion cycle.
+The Traces page displays data for recent traces performed (see the following section for information on [tracing][tracing]).
+The Recent Logs page displays warning and error logs forwarded to the monitor from all Accumulo processes.
+Also, the XML and JSON links provide metrics in XML and JSON formats, respectively.
+
+### SSL
+
+SSL may be enabled for the monitor page by setting the following properties in the `accumulo-site.xml` file:
+
+ monitor.ssl.keyStore
+ monitor.ssl.keyStorePassword
+ monitor.ssl.trustStore
+ monitor.ssl.trustStorePassword
+
+If the Accumulo conf directory has been configured (in particular the `accumulo-env.sh` file must be set up), the
+`accumulo-util gen-monitor-cert` command can be used to create the keystore and truststore files with random passwords. The command
+will print out the properties that need to be added to the `accumulo-site.xml` file. The stores can also be generated manually with the
+Java `keytool` command, whose usage can be seen in the `accumulo-util` script.
+
+If desired, the SSL ciphers allowed for connections can be controlled via the following properties in `accumulo-site.xml`:
+
+ monitor.ssl.include.ciphers
+ monitor.ssl.exclude.ciphers
+
+If SSL is enabled, the monitor URL can only be accessed via https.
+This also allows you to access the Accumulo shell through the monitor page.
+The left navigation bar will have a new link to Shell.
+An Accumulo user name and password must be entered for access to the shell.
+
+## Metrics
+
+Accumulo can expose metrics through a legacy metrics library and using the Hadoop Metrics2 library.
+
+### Legacy Metrics
+
+Accumulo has a legacy metrics library that can be exposes metrics using JMX endpoints or file-based logging. These metrics can
+be enabled by setting `general.legacy.metrics` to `true` in `accumulo-site.xml` and placing the `accumulo-metrics.xml`
+configuration file on the classpath (which is typically done by placing the file in the `conf/` directory). A template for
+`accumulo-metrics.xml` can be found in `conf/templates` of the Accumulo tarball.
+
+### Hadoop Metrics2
+
+Hadoop Metrics2 is a library which allows for routing of metrics generated by registered MetricsSources to
+configured MetricsSinks. Examples of sinks that are implemented by Hadoop include file-based logging, Graphite and Ganglia.
+All metric sources are exposed via JMX when using Metrics2.
+
+Metrics2 is configured by examining the classpath for a file that matches `hadoop-metrics2*.properties`. The Accumulo tarball
+contains an example `hadoop-metrics2-accumulo.properties` file in `conf/templates` which can be copied to `conf/` to place
+on classpath. This file is used to enable file, Graphite or Ganglia sinks (some minimal configuration required for Graphite
+and Ganglia). Because the Hadoop configuration is also on the Accumulo classpath, be sure that you do not have multiple
+Metrics2 configuration files. It is recommended to consolidate metrics in a single properties file in a central location to
+remove ambiguity. The contents of `hadoop-metrics2-accumulo.properties` can be added to a central `hadoop-metrics2.properties`
+in `$HADOOP_CONF_DIR`.
+
+As a note for configuring the file sink, the provided path should be absolute. A relative path or file name will be created relative
+to the directory in which the Accumulo process was started. External tools, such as logrotate, can be used to prevent these files
+from growing without bound.
+
+Each server process should have log messages from the Metrics2 library about the sinks that were created. Be sure to check
+the Accumulo processes log files when debugging missing metrics output.
+
+For additional information on configuring Metrics2, visit the [Javadoc page for Metrics2](https://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html).
+
+[tracing]: {{page.docs_baseurl}}/administration/tracing
[06/10] accumulo-website git commit: Jekyll build from master:036341f
Posted by mw...@apache.org.
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/troubleshooting/advanced.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/troubleshooting/advanced.html b/docs/unreleased/troubleshooting/advanced.html
new file mode 100644
index 0000000..987bdb8
--- /dev/null
+++ b/docs/unreleased/troubleshooting/advanced.html
@@ -0,0 +1,728 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<meta charset="utf-8">
+<meta http-equiv="X-UA-Compatible" content="IE=edge">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<link href="https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/paper/bootstrap.min.css" rel="stylesheet" integrity="sha384-awusxf8AUojygHf2+joICySzB780jVvQaVCAt1clU3QsyAitLGul28Qxb2r1e5g+" crossorigin="anonymous">
+<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
+<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.css">
+<link href="/css/accumulo.css" rel="stylesheet" type="text/css">
+
+<title>Accumulo Documentation - Advanced Troubleshooting</title>
+
+<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
+<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
+<script type="text/javascript" src="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.js"></script>
+<script>
+ // show location of canonical site if not currently on the canonical site
+ $(function() {
+ var host = window.location.host;
+ if (typeof host !== 'undefined' && host !== 'accumulo.apache.org') {
+ $('#non-canonical').show();
+ }
+ });
+
+ $(function() {
+ // decorate section headers with anchors
+ return $("h2, h3, h4, h5, h6").each(function(i, el) {
+ var $el, icon, id;
+ $el = $(el);
+ id = $el.attr('id');
+ icon = '<i class="fa fa-link"></i>';
+ if (id) {
+ return $el.append($("<a />").addClass("header-link").attr("href", "#" + id).html(icon));
+ }
+ });
+ });
+
+ // configure Google Analytics
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ if (ga.hasOwnProperty('loaded') && ga.loaded === true) {
+ ga('create', 'UA-50934829-1', 'apache.org');
+ ga('send', 'pageview');
+ }
+</script>
+
+</head>
+<body style="padding-top: 100px">
+
+ <nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="/"><img id="nav-logo" alt="Apache Accumulo" class="img-responsive" src="/images/accumulo-logo.png" width="200"
+ /></a>
+ </div>
+ <div class="collapse navbar-collapse" id="navbar-items">
+ <ul class="nav navbar-nav">
+ <li class="nav-link"><a href="/downloads">Download</a></li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Releases<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/release/accumulo-1.8.1/">1.8.1 (Latest)</a></li>
+ <li><a href="/release/accumulo-1.7.3/">1.7.3</a></li>
+ <li><a href="/release/accumulo-1.6.6/">1.6.6</a></li>
+ <li><a href="/release/">Archive</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/1.8/accumulo_user_manual.html">User Manual (1.8)</a></li>
+ <li><a href="/1.8/apidocs">Javadocs (1.8)</a></li>
+ <li><a href="/1.8/examples">Examples (1.8)</a></li>
+ <li><a href="/features">Features</a></li>
+ <li><a href="/glossary">Glossary</a></li>
+ <li><a href="/external-docs">External Docs</a></li>
+ <li><a href="/docs-archive/">Archive</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/get_involved">Get Involved</a></li>
+ <li><a href="/mailing_list">Mailing Lists</a></li>
+ <li><a href="/people">People</a></li>
+ <li><a href="/related-projects">Related Projects</a></li>
+ <li><a href="/contributor/">Contributor Guide</a></li>
+ </ul>
+ </li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://www.apache.org">Apache Homepage <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/licenses/LICENSE-2.0">License <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/sponsorship">Sponsorship <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/security">Security <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/thanks">Thanks <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct <i class="fa fa-external-link"></i></a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12">
+
+ <div id="non-canonical" style="display: none; background-color: #F0E68C; padding-left: 1em;">
+ Visit the official site at: <a href="https://accumulo.apache.org">https://accumulo.apache.org</a>
+ </div>
+ <div id="content">
+
+ <div class="alert alert-danger" role="alert">This documentation is for an unreleased version of Apache Accumulo that is currently under development! Check out the <a href="/1.8/accumulo_user_manual.html">documentation for the latest release</a>.</div>
+
+<div class="row">
+ <div class="col-md-3">
+ <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true">
+ <div class="panel panel-default">
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsegetting-started" aria-expanded="false" aria-controls="collapsegetting-started">
+ Getting started
+ </a>
+ </h4>
+ </div>
+ <div id="collapsegetting-started" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/design">Accumulo Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/clients">Accumulo Clients</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/shell">Accumulo Shell</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_design">Table Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_configuration">Table Configuration</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsedevelopment" aria-expanded="false" aria-controls="collapsedevelopment">
+ Development
+ </a>
+ </h4>
+ </div>
+ <div id="collapsedevelopment" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/sampling">Sampling</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/summaries">Summary Statistics</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/security">Security</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapseadministration" aria-expanded="false" aria-controls="collapseadministration">
+ Administration
+ </a>
+ </h4>
+ </div>
+ <div id="collapseadministration" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/multivolume">Multi-Volume Installations</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsetroubleshooting" aria-expanded="true" aria-controls="collapsetroubleshooting">
+ Troubleshooting
+ </a>
+ </h4>
+ </div>
+ <div id="collapsetroubleshooting" class="panel-collapse collapse in" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
+
+ </div>
+ </div>
+
+
+
+ </div>
+ </div>
+ </div>
+ <div class="col-md-9">
+
+ <p><a href="/docs/unreleased/">Accumulo unreleased docs</a> >> Troubleshooting >> Advanced Troubleshooting</p>
+
+
+ <h1>Advanced Troubleshooting</h1>
+
+ <h2 id="tablet-server-locks">Tablet server locks</h2>
+
+<p><strong>My tablet server lost its lock. Why?</strong></p>
+
+<p>The primary reason a tablet server loses its lock is that it has been pushed into swap.</p>
+
+<p>A large java program (like the tablet server) may have a large portion
+of its memory image unused. The operation system will favor pushing
+this allocated, but unused memory into swap so that the memory can be
+re-used as a disk buffer. When the java virtual machine decides to
+access this memory, the OS will begin flushing disk buffers to return that
+memory to the VM. This can cause the entire process to block long
+enough for the zookeeper lock to be lost.</p>
+
+<p>Configure your system to reduce the kernel parameter <em>swappiness</em> from the default (60) to zero.</p>
+
+<p><strong>My tablet server lost its lock, and I have already set swappiness to zero. Why?</strong></p>
+
+<p>Be careful not to over-subscribe memory. This can be easy to do if
+your accumulo processes run on the same nodes as hadoop’s map-reduce
+framework. Remember to add up:</p>
+
+<ul>
+ <li>size of the JVM for the tablet server</li>
+ <li>size of the in-memory map, if using the native map implementation</li>
+ <li>size of the JVM for the data node</li>
+ <li>size of the JVM for the task tracker</li>
+ <li>size of the JVM times the maximum number of mappers and reducers</li>
+ <li>size of the kernel and any support processes</li>
+</ul>
+
+<p>If a 16G node can run 2 mappers and 2 reducers, and each can be 2G,
+then there is only 8G for the data node, tserver, task tracker and OS.</p>
+
+<p>Reduce the memory footprint of each component until it fits comfortably.</p>
+
+<p><strong>My tablet server lost its lock, swappiness is zero, and my node has lots of unused memory!</strong></p>
+
+<p>The JVM memory garbage collector may fall behind and cause a
+“stop-the-world” garbage collection. On a large memory virtual
+machine, this collection can take a long time. This happens more
+frequently when the JVM is getting low on free memory. Check the logs
+of the tablet server. You will see lines like this:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>2013-06-20 13:43:20,607 [tabletserver.TabletServer] DEBUG: gc ParNew=0.00(+0.00) secs
+ ConcurrentMarkSweep=0.00(+0.00) secs freemem=1,868,325,952(+1,868,325,952) totalmem=2,040,135,680
+</code></pre>
+</div>
+
+<p>When <code class="highlighter-rouge">freemem</code> becomes small relative to the amount of memory
+needed, the JVM will spend more time finding free memory than
+performing work. This can cause long delays in sending keep-alive
+messages to zookeeper.</p>
+
+<p>Ensure the tablet server JVM is not running low on memory.</p>
+
+<p><strong>I’m seeing errors in tablet server logs that include the words “MutationsRejectedException” and “# constraint violations: 1”. Moments after that the server died.</strong></p>
+
+<p>The error you are seeing is part of a failing tablet server scenario.
+This is a bit complicated, so name two of your tablet servers A and B.</p>
+
+<p>Tablet server A is hosting a tablet, let’s call it a-tablet.</p>
+
+<p>Tablet server B is hosting a metadata tablet, let’s call it m-tablet.</p>
+
+<p>m-tablet records the information about a-tablet, for example, the names of the files it is using to store data.</p>
+
+<p>When A ingests some data, it eventually flushes the updates from memory to a file.</p>
+
+<p>Tablet server A then writes this new information to m-tablet, on Tablet server B.</p>
+
+<p>Here’s a likely failure scenario:</p>
+
+<p>Tablet server A does not have enough memory for all the processes running on it.
+The operating system sees a large chunk of the tablet server being unused, and swaps it out to disk to make room for other processes.
+Tablet server A does a java memory garbage collection, which causes it to start using all the memory allocated to it.
+As the server starts pulling data from swap, it runs very slowly.
+It fails to send the keep-alive messages to zookeeper in a timely fashion, and it looses its zookeeper session.</p>
+
+<p>But, it’s running so slowly, that it takes a moment to realize it should no longer be hosting tablets.</p>
+
+<p>The thread that is flushing a-tablet memory attempts to update m-tablet with the new file information.</p>
+
+<p>Fortunately there’s a constraint on m-tablet.
+Mutations to the metadata table must contain a valid zookeeper session.
+This prevents tablet server A from making updates to m-tablet when it no long has the right to host the tablet.</p>
+
+<p>The “MutationsRejectedException” error is from tablet server A making an update to tablet server B’s m-tablet.
+It’s getting a constraint violation: tablet server A has lost its zookeeper session, and will fail momentarily.</p>
+
+<p>Ensure that memory is not over-allocated. Monitor swap usage, or turn swap off.</p>
+
+<p><strong>My accumulo client is getting a MutationsRejectedException. The monitor is displaying “No Such SessionID” errors.</strong></p>
+
+<p>When your client starts sending mutations to accumulo, it creates a session. Once the session is created,
+mutations are streamed to accumulo, without acknowledgement, against this session. Once the client is done,
+it will close the session, and get an acknowledgement.</p>
+
+<p>If the client fails to communicate with accumulo, it will release the session, assuming that the client has died.
+If the client then attempts to send more mutations against the session, you will see “No Such SessionID” errors on
+the server, and MutationRejectedExceptions in the client.</p>
+
+<p>The client library should be either actively using the connection to the tablet servers,
+or closing the connection and sessions. If the session times out, something is causing your client
+to pause.</p>
+
+<p>The most frequent source of these pauses are java garbage collection pauses
+due to the JVM running out of memory, or being swapped out to disk.</p>
+
+<p>Ensure your client has adequate memory and is not being swapped out to disk.</p>
+
+<h2 id="hdfs-failures">HDFS Failures</h2>
+
+<p><strong>I had disasterous HDFS failure. After bringing everything back up, several tablets refuse to go online.</strong></p>
+
+<p>Data written to tablets is written into memory before being written into indexed files. In case the server
+is lost before the data is saved into a an indexed file, all data stored in memory is first written into a
+write-ahead log (WAL). When a tablet is re-assigned to a new tablet server, the write-ahead logs are read to
+recover any mutations that were in memory when the tablet was last hosted.</p>
+
+<p>If a write-ahead log cannot be read, then the tablet is not re-assigned. All it takes is for one of
+the blocks in the write-ahead log to be missing. This is unlikely unless multiple data nodes in HDFS have been
+lost.</p>
+
+<p>Get the WAL files online and healthy. Restore any data nodes that may be down.</p>
+
+<p><strong>How do find out which tablets are offline?</strong></p>
+
+<p>Use <code class="highlighter-rouge">accumulo admin checkTablets</code></p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo admin checkTablets
+</code></pre>
+</div>
+
+<p><strong>I lost three data nodes, and I’m missing blocks in a WAL. I don’t care about data loss, how
+can I get those tablets online?</strong></p>
+
+<p>See the <a href="/docs/unreleased/troubleshooting/system-metadata-tables">system metadata table page</a> which shows a typical metadata table listing.
+The entries with a column family of <code class="highlighter-rouge">log</code> are references to the WAL for that tablet.
+If you know what WAL is bad, you can find all the references with a grep in the shell:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>shell> grep 0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995
+3< log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995|6
+</code></pre>
+</div>
+
+<p>You can remove the WAL references in the metadata table.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>shell> grant -u root Table.WRITE -t accumulo.metadata
+shell> delete 3< log 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995
+</code></pre>
+</div>
+
+<p>Note: the colon (<code class="highlighter-rouge">:</code>) is omitted when specifying the <em>row cf cq</em> for the delete command.</p>
+
+<p>The master will automatically discover the tablet no longer has a bad WAL reference and will
+assign the tablet. You will need to remove the reference from all the tablets to get them
+online.</p>
+
+<p><strong>The metadata (or root) table has references to a corrupt WAL.</strong></p>
+
+<p>This is a much more serious state, since losing updates to the metadata table will result
+in references to old files which may not exist, or lost references to new files, resulting
+in tablets that cannot be read, or large amounts of data loss.</p>
+
+<p>The best hope is to restore the WAL by fixing HDFS data nodes and bringing the data back online.
+If this is not possible, the best approach is to re-create the instance and bulk import all files from
+the old instance into a new tables.</p>
+
+<p>A complete set of instructions for doing this is outside the scope of this guide,
+but the basic approach is:</p>
+
+<ul>
+ <li>Use <code class="highlighter-rouge">tables -l</code> in the shell to discover the table name to table id mapping</li>
+ <li>Stop all accumulo processes on all nodes</li>
+ <li>Move the accumulo directory in HDFS out of the way:
+ $ hadoop fs -mv /accumulo /corrupt</li>
+ <li>Re-initalize accumulo</li>
+ <li>Recreate tables, users and permissions</li>
+ <li>Import the directories under <code class="highlighter-rouge">/corrupt/tables/<id></code> into the new instance</li>
+</ul>
+
+<p><strong>One or more HDFS Files under /accumulo/tables are corrupt</strong></p>
+
+<p>Accumulo maintains multiple references into the tablet files in the metadata
+tables and within the tablet server hosting the file, this makes it difficult to
+reliably just remove those references.</p>
+
+<p>The directory structure in HDFS for tables will follow the general structure:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>/accumulo
+/accumulo/tables/
+/accumulo/tables/!0
+/accumulo/tables/!0/default_tablet/A000001.rf
+/accumulo/tables/!0/t-00001/A000002.rf
+/accumulo/tables/1
+/accumulo/tables/1/default_tablet/A000003.rf
+/accumulo/tables/1/t-00001/A000004.rf
+/accumulo/tables/1/t-00001/A000005.rf
+/accumulo/tables/2/default_tablet/A000006.rf
+/accumulo/tables/2/t-00001/A000007.rf
+</code></pre>
+</div>
+
+<p>If files under <code class="highlighter-rouge">/accumulo/tables</code> are corrupt, the best course of action is to
+recover those files in hdsf see the section on HDFS. Once these recovery efforts
+have been exhausted, the next step depends on where the missing file(s) are
+located. Different actions are required when the bad files are in Accumulo data
+table files or if they are metadata table files.</p>
+
+<p><em>Data File Corruption</em></p>
+
+<p>When an Accumulo data file is corrupt, the most reliable way to restore Accumulo
+operations is to replace the missing file with an ``empty’’ file so that
+references to the file in the METADATA table and within the tablet server
+hosting the file can be resolved by Accumulo. An empty file can be created using
+the CreateEmpty utiity:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.core.file.rfile.CreateEmpty /path/to/empty/file/empty.rf
+</code></pre>
+</div>
+
+<p>The process is to delete the corrupt file and then move the empty file into its
+place (The generated empty file can be copied and used multiple times if necessary and does not need
+to be regenerated each time)</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ hadoop fs –rm /accumulo/tables/corrupt/file/thename.rf; \
+hadoop fs -mv /path/to/empty/file/empty.rf /accumulo/tables/corrupt/file/thename.rf
+</code></pre>
+</div>
+
+<p><em>Metadata File Corruption</em></p>
+
+<p>If the corrupt files are metadata files, read the <a href="/docs/unreleased/troubleshooting/system-metadata-tables">system metadata tables</a>
+(under the path <code class="highlighter-rouge">/accumulo/tables/!0</code>). Then, you will need to rebuild
+the metadata table by initializing a new instance of Accumulo and then importing
+all of the existing data into the new instance. This is the same procedure as
+recovering from a zookeeper failure (see next section), except that
+you will have the benefit of having the existing user and table authorizations
+that are maintained in zookeeper.</p>
+
+<p>You can use the DumpZookeeper utility to save this information for reference
+before creating the new instance. You will not be able to use RestoreZookeeper
+because the table names and references are likely to be different between the
+original and the new instances, but it can serve as a reference.</p>
+
+<p>If the files cannot be recovered, replace corrupt data files with a empty
+rfiles to allow references in the metadata table and in the tablet servers to be
+resolved. Rebuild the metadata table if the corrupt files are metadata files.</p>
+
+<p><em>Write-Ahead Log(WAL) File Corruption</em></p>
+
+<p>In certain versions of Accumulo, a corrupt WAL file (caused by HDFS corruption
+or a bug in Accumulo that created the file) can block the successful recovery
+of one to many Tablets. Accumulo can be stuck in a loop trying to recover the
+WAL file, never being able to succeed.</p>
+
+<p>In the cases where the WAL file’s original contents are unrecoverable or some degree
+of data loss is acceptable (beware if the WAL file contains updates to the Accumulo
+metadat table!), the following process can be followed to create an valid, empty
+WAL file. Run the following commands as the Accumulo unix user (to ensure that
+the proper file permissions in HDFS)</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ echo -n -e '--- Log File Header (v2) ---\x00\x00\x00\x00' > empty.wal
+</code></pre>
+</div>
+
+<p>The above creates a file with the text “— Log File Header (v2) —” and then
+four bytes. You should verify the contents of the file with a hexdump tool.</p>
+
+<p>Then, place this empty WAL in HDFS and then replace the corrupt WAL file in HDFS
+with the empty WAL.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ hdfs dfs -moveFromLocal empty.wal /user/accumulo/empty.wal
+$ hdfs dfs -mv /user/accumulo/empty.wal /accumulo/wal/tserver-4.example.com+10011/26abec5b-63e7-40dd-9fa1-b8ad2436606e
+</code></pre>
+</div>
+
+<p>After the corrupt WAL file has been replaced, the system should automatically recover.
+It may be necessary to restart the Accumulo Master process as an exponential
+backup policy is used which could lead to a long wait before Accumulo will
+try to re-load the WAL file.</p>
+
+<h2 id="zookeeper-failures">Zookeeper Failures</h2>
+
+<p><strong>I lost my ZooKeeper quorum (hardware failure), but HDFS is still intact. How can I recover my Accumulo instance?</strong></p>
+
+<p>ZooKeeper, in addition to its lock-service capabilities, also serves to bootstrap an Accumulo
+instance from some location in HDFS. It contains the pointers to the root tablet in HDFS which
+is then used to load the Accumulo metadata tablets, which then loads all user tables. ZooKeeper
+also stores all namespace and table configuration, the user database, the mapping of table IDs to
+table names, and more across Accumulo restarts.</p>
+
+<p>Presently, the only way to recover such an instance is to initialize a new instance and import all
+of the old data into the new instance. The easiest way to tackle this problem is to first recreate
+the mapping of table ID to table name and then recreate each of those tables in the new instance.
+Set any necessary configuration on the new tables and add some split points to the tables to close
+the gap between how many splits the old table had and no splits.</p>
+
+<p>The directory structure in HDFS for tables will follow the general structure:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>/accumulo
+/accumulo/tables/
+/accumulo/tables/1
+/accumulo/tables/1/default_tablet/A000001.rf
+/accumulo/tables/1/t-00001/A000002.rf
+/accumulo/tables/1/t-00001/A000003.rf
+/accumulo/tables/2/default_tablet/A000004.rf
+/accumulo/tables/2/t-00001/A000005.rf
+</code></pre>
+</div>
+
+<p>For each table, make a new directory that you can move (or copy if you have the HDFS space to do so)
+all of the rfiles for a given table into. For example, to process the table with an ID of <code class="highlighter-rouge">1</code>, make a new directory,
+say <code class="highlighter-rouge">/new-table-1</code> and then copy all files from <code class="highlighter-rouge">/accumulo/tables/1/\*/*.rf</code> into that directory. Additionally,
+make a directory, <code class="highlighter-rouge">/new-table-1-failures</code>, for any failures during the import process. Then, issue the import
+command using the Accumulo shell into the new table, telling Accumulo to not re-set the timestamp:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>user@instance new_table> importdirectory /new-table-1 /new-table-1-failures false
+</code></pre>
+</div>
+
+<p>Any RFiles which were failed to be loaded will be placed in <code class="highlighter-rouge">/new-table-1-failures</code>. Rfiles that were successfully
+imported will no longer exist in <code class="highlighter-rouge">/new-table-1</code>. For failures, move them back to the import directory and retry
+the <code class="highlighter-rouge">importdirectory</code> command.</p>
+
+<p>It is <em>extremely</em> important to note that this approach may introduce stale data back into
+the tables. For a few reasons, RFiles may exist in the table directory which are candidates for deletion but have
+not yet been deleted. Additionally, deleted data which was not compacted away, but still exists in write-ahead logs if
+the original instance was somehow recoverable, will be re-introduced in the new instance. Table splits and merges
+(which also include the deleteRows API call on TableOperations, are also vulnerable to this problem. This process should
+<em>not</em> be used if these are unacceptable risks. It is possible to try to re-create a view of the <code class="highlighter-rouge">accumulo.metadata</code>
+table to prune out files that are candidates for deletion, but this is a difficult task that also may not be entirely accurate.</p>
+
+<p>Likewise, it is also possible that data loss may occur from write-ahead log (WAL) files which existed on the old table but
+were not minor-compacted into an RFile. Again, it may be possible to reconstruct the state of these WAL files to
+replay data not yet in an RFile; however, this is a difficult task and is not implemented in any automated fashion.</p>
+
+<p>The <code class="highlighter-rouge">importdirectory</code> shell command can be used to import RFiles from the old instance into a newly created instance,
+but extreme care should go into the decision to do this as it may result in reintroduction of stale data or the
+omission of new data.</p>
+
+<h2 id="upgrade-issues">Upgrade Issues</h2>
+
+<p><strong>I upgraded from 1.4 to 1.5 to 1.6 but still have some WAL files on local disk. Do I have any way to recover them?</strong></p>
+
+<p>Yes, you can recover them by running the LocalWALRecovery utility (not available in 1.8 and later) on each node that needs recovery performed. The utility
+will default to using the directory specified by <code class="highlighter-rouge">logger.dir.walog</code> in your configuration, or can be
+overriden by using the <code class="highlighter-rouge">--local-wal-directories</code> option on the tool. It can be invoked as follows:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>accumulo org.apache.accumulo.tserver.log.LocalWALRecovery
+</code></pre>
+</div>
+
+<h2 id="file-naming-conventions">File Naming Conventions</h2>
+
+<p><strong>Why are files named like they are? Why do some start with <code class="highlighter-rouge">C</code> and others with <code class="highlighter-rouge">F</code>?</strong></p>
+
+<p>The file names give you a basic idea for the source of the file.</p>
+
+<p>The base of the filename is a base-36 unique number. All filenames in accumulo are coordinated
+with a counter in zookeeper, so they are always unique, which is useful for debugging.</p>
+
+<p>The leading letter gives you an idea of how the file was created:</p>
+
+<ul>
+ <li><code class="highlighter-rouge">F</code> - Flush: entries in memory were written to a file (Minor Compaction)</li>
+ <li><code class="highlighter-rouge">M</code> - Merging compaction: entries in memory were combined with the smallest file to create one new file</li>
+ <li><code class="highlighter-rouge">C</code> - Several files, but not all files, were combined to produce this file (Major Compaction)</li>
+ <li><code class="highlighter-rouge">A</code> - All files were compacted, delete entries were dropped</li>
+ <li><code class="highlighter-rouge">I</code> - Bulk import, complete, sorted index files. Always in a directory starting with <code class="highlighter-rouge">b-</code></li>
+</ul>
+
+<p>This simple file naming convention allows you to see the basic structure of the files from just
+their filenames, and reason about what should be happening to them next, just
+by scanning their entries in the metadata tables.</p>
+
+<p>For example, if you see multiple files with <code class="highlighter-rouge">M</code> prefixes, the tablet is, or was, up against its
+maximum file limit, so it began merging memory updates with files to keep the file count reasonable. This
+slows down ingest performance, so knowing there are many files like this tells you that the system
+is struggling to keep up with ingest vs the compaction strategy which reduces the number of files.</p>
+
+<h2 id="hdfs-decommissioning-issues">HDFS Decommissioning Issues</h2>
+
+<p><strong>My Hadoop DataNode is hung for hours trying to decommission.</strong></p>
+
+<p>Write Ahead Logs stay open until they hit the size threshold, which could be many hours or days in some cases. These open files will prevent a DN from finishing its decommissioning process (HDFS-3599) in some versions of Hadoop 2. If you stop the DN, then the WALog file will not be closed and you could lose data. To work around this issue, we now close WALogs on a time period specified by the property <code class="highlighter-rouge">tserver.walog.max.age</code> with a default period of 24 hours.</p>
+
+
+ </div>
+</div>
+
+ </div>
+
+
+<footer>
+
+ <p><a href="https://www.apache.org/foundation/contributing"><img src="https://www.apache.org/images/SupportApache-small.png" alt="Support the ASF" id="asf-logo" height="100" /></a></p>
+
+ <p>Copyright © 2011-2017 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.</p>
+
+</footer>
+
+
+ </div>
+ </div>
+ </div>
+</body>
+</html>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/troubleshooting/basic.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/troubleshooting/basic.html b/docs/unreleased/troubleshooting/basic.html
new file mode 100644
index 0000000..18865e1
--- /dev/null
+++ b/docs/unreleased/troubleshooting/basic.html
@@ -0,0 +1,576 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<meta charset="utf-8">
+<meta http-equiv="X-UA-Compatible" content="IE=edge">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<link href="https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/paper/bootstrap.min.css" rel="stylesheet" integrity="sha384-awusxf8AUojygHf2+joICySzB780jVvQaVCAt1clU3QsyAitLGul28Qxb2r1e5g+" crossorigin="anonymous">
+<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
+<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.css">
+<link href="/css/accumulo.css" rel="stylesheet" type="text/css">
+
+<title>Accumulo Documentation - Basic Troubleshooting</title>
+
+<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
+<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
+<script type="text/javascript" src="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.js"></script>
+<script>
+ // show location of canonical site if not currently on the canonical site
+ $(function() {
+ var host = window.location.host;
+ if (typeof host !== 'undefined' && host !== 'accumulo.apache.org') {
+ $('#non-canonical').show();
+ }
+ });
+
+ $(function() {
+ // decorate section headers with anchors
+ return $("h2, h3, h4, h5, h6").each(function(i, el) {
+ var $el, icon, id;
+ $el = $(el);
+ id = $el.attr('id');
+ icon = '<i class="fa fa-link"></i>';
+ if (id) {
+ return $el.append($("<a />").addClass("header-link").attr("href", "#" + id).html(icon));
+ }
+ });
+ });
+
+ // configure Google Analytics
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ if (ga.hasOwnProperty('loaded') && ga.loaded === true) {
+ ga('create', 'UA-50934829-1', 'apache.org');
+ ga('send', 'pageview');
+ }
+</script>
+
+</head>
+<body style="padding-top: 100px">
+
+ <nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="/"><img id="nav-logo" alt="Apache Accumulo" class="img-responsive" src="/images/accumulo-logo.png" width="200"
+ /></a>
+ </div>
+ <div class="collapse navbar-collapse" id="navbar-items">
+ <ul class="nav navbar-nav">
+ <li class="nav-link"><a href="/downloads">Download</a></li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Releases<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/release/accumulo-1.8.1/">1.8.1 (Latest)</a></li>
+ <li><a href="/release/accumulo-1.7.3/">1.7.3</a></li>
+ <li><a href="/release/accumulo-1.6.6/">1.6.6</a></li>
+ <li><a href="/release/">Archive</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/1.8/accumulo_user_manual.html">User Manual (1.8)</a></li>
+ <li><a href="/1.8/apidocs">Javadocs (1.8)</a></li>
+ <li><a href="/1.8/examples">Examples (1.8)</a></li>
+ <li><a href="/features">Features</a></li>
+ <li><a href="/glossary">Glossary</a></li>
+ <li><a href="/external-docs">External Docs</a></li>
+ <li><a href="/docs-archive/">Archive</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/get_involved">Get Involved</a></li>
+ <li><a href="/mailing_list">Mailing Lists</a></li>
+ <li><a href="/people">People</a></li>
+ <li><a href="/related-projects">Related Projects</a></li>
+ <li><a href="/contributor/">Contributor Guide</a></li>
+ </ul>
+ </li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://www.apache.org">Apache Homepage <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/licenses/LICENSE-2.0">License <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/sponsorship">Sponsorship <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/security">Security <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/thanks">Thanks <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct <i class="fa fa-external-link"></i></a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12">
+
+ <div id="non-canonical" style="display: none; background-color: #F0E68C; padding-left: 1em;">
+ Visit the official site at: <a href="https://accumulo.apache.org">https://accumulo.apache.org</a>
+ </div>
+ <div id="content">
+
+ <div class="alert alert-danger" role="alert">This documentation is for an unreleased version of Apache Accumulo that is currently under development! Check out the <a href="/1.8/accumulo_user_manual.html">documentation for the latest release</a>.</div>
+
+<div class="row">
+ <div class="col-md-3">
+ <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true">
+ <div class="panel panel-default">
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsegetting-started" aria-expanded="false" aria-controls="collapsegetting-started">
+ Getting started
+ </a>
+ </h4>
+ </div>
+ <div id="collapsegetting-started" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/design">Accumulo Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/clients">Accumulo Clients</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/shell">Accumulo Shell</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_design">Table Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_configuration">Table Configuration</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsedevelopment" aria-expanded="false" aria-controls="collapsedevelopment">
+ Development
+ </a>
+ </h4>
+ </div>
+ <div id="collapsedevelopment" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/sampling">Sampling</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/summaries">Summary Statistics</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/security">Security</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapseadministration" aria-expanded="false" aria-controls="collapseadministration">
+ Administration
+ </a>
+ </h4>
+ </div>
+ <div id="collapseadministration" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/multivolume">Multi-Volume Installations</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsetroubleshooting" aria-expanded="true" aria-controls="collapsetroubleshooting">
+ Troubleshooting
+ </a>
+ </h4>
+ </div>
+ <div id="collapsetroubleshooting" class="panel-collapse collapse in" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
+
+ </div>
+ </div>
+
+
+
+ </div>
+ </div>
+ </div>
+ <div class="col-md-9">
+
+ <p><a href="/docs/unreleased/">Accumulo unreleased docs</a> >> Troubleshooting >> Basic Troubleshooting</p>
+
+
+ <h1>Basic Troubleshooting</h1>
+
+ <h2 id="general">General</h2>
+
+<p><strong>The tablet server does not seem to be running!? What happened?</strong></p>
+
+<p>Accumulo is a distributed system. It is supposed to run on remote
+equipment, across hundreds of computers. Each program that runs on
+these remote computers writes down events as they occur, into a local
+file. By default, this is defined in <code class="highlighter-rouge">conf/accumulo-env.sh</code> as <code class="highlighter-rouge">ACCUMULO_LOG_DIR</code>.
+Look in the <code class="highlighter-rouge">$ACCUMULO_LOG_DIR/tserver*.log</code> file. Specifically, check the end of the file.</p>
+
+<p><strong>The tablet server did not start and the debug log does not exists! What happened?</strong></p>
+
+<p>When the individual programs are started, the stdout and stderr output
+of these programs are stored in <code class="highlighter-rouge">.out</code> and <code class="highlighter-rouge">.err</code> files in
+<code class="highlighter-rouge">$ACCUMULO_LOG_DIR</code>. Often, when there are missing configuration
+options, files or permissions, messages will be left in these files.
+Probably a start-up problem. Look in <code class="highlighter-rouge">$ACCUMULO_LOG_DIR/tserver*.err</code></p>
+
+<p><strong>Accumulo is not working, what’s wrong?</strong></p>
+
+<p>There’s a small web server that collects information about all the
+components that make up a running Accumulo instance. It will highlight
+unusual or unexpected conditions.</p>
+
+<p>Point your browser to the monitor (typically the master host, on port 9995). Is anything red or yellow?</p>
+
+<p><strong>My browser is reporting connection refused, and I cannot get to the monitor</strong></p>
+
+<p>The monitor program’s output is also written to .err and .out files in
+the <code class="highlighter-rouge">$ACCUMULO_LOG_DIR</code>. Look for problems in this file if the
+<code class="highlighter-rouge">$ACCUMULO_LOG_DIR/monitor*.log</code> file does not exist.</p>
+
+<p>The monitor program is probably not running. Check the log files for errors.</p>
+
+<p><strong>My browser hangs trying to talk to the monitor.</strong></p>
+
+<p>Your browser needs to be able to reach the monitor program. Often
+large clusters are firewalled, or use a VPN for internal
+communications. You can use SSH to proxy your browser to the cluster,
+or consult with your system administrator to gain access to the server
+from your browser.</p>
+
+<p>It is sometimes helpful to use a text-only browser to sanity-check the
+monitor while on the machine running the monitor:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ links http://localhost:9995
+</code></pre>
+</div>
+
+<p>Verify that you are not firewalled from the monitor if it is running on a remote host.</p>
+
+<p><strong>The monitor responds, but there are no numbers for tservers and tables. The summary page says the master is down.</strong></p>
+
+<p>The monitor program gathers all the details about the master and the
+tablet servers through the master. It will be mostly blank if the
+master is down. Check for a running master.</p>
+
+<h2 id="accumulo-processes">Accumulo Processes</h2>
+
+<p><strong>My tablet server crashed! The logs say that it lost its zookeeper lock.</strong></p>
+
+<p>Tablet servers reserve a lock in zookeeper to maintain their ownership
+over the tablets that have been assigned to them. Part of their
+responsibility for keeping the lock is to send zookeeper a keep-alive
+message periodically. If the tablet server fails to send a message in
+a timely fashion, zookeeper will remove the lock and notify the tablet
+server. If the tablet server does not receive a message from
+zookeeper, it will assume its lock has been lost, too. If a tablet
+server loses its lock, it kills itself: everything assumes it is dead
+already.</p>
+
+<p>Investigate why the tablet server did not send a timely message to
+zookeeper.</p>
+
+<p><strong>I need to decommission a node. How do I stop the tablet server on it?</strong></p>
+
+<p>Use the admin command:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo admin stop hostname:9997
+2013-07-16 13:15:38,403 [util.Admin] INFO : Stopping server 12.34.56.78:9997
+</code></pre>
+</div>
+
+<p><strong>I cannot login to a tablet server host, and the tablet server will not shut down. How can I kill the server?</strong></p>
+
+<p>Sometimes you can kill a “stuck” tablet server by deleting its lock in zookeeper:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.server.util.TabletServerLocks --list
+ 127.0.0.1:9997 TSERV_CLIENT=127.0.0.1:9997
+$ accumulo org.apache.accumulo.server.util.TabletServerLocks -delete 127.0.0.1:9997
+$ accumulo org.apache.accumulo.server.util.TabletServerLocks -list
+ 127.0.0.1:9997 null
+</code></pre>
+</div>
+
+<p>You can find the master and instance id for any accumulo instances using the same zookeeper instance:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.server.util.ListInstances
+INFO : Using ZooKeepers localhost:2181
+
+ Instance Name | Instance ID | Master
+---------------------+--------------------------------------+-------------------------------
+ "test" | 6140b72e-edd8-4126-b2f5-e74a8bbe323b | 127.0.0.1:9999
+</code></pre>
+</div>
+
+<p><strong>One of my Accumulo processes died. How do I bring it back?</strong></p>
+
+<p>The easiest way to bring all services online for an Accumulo instance is to run the <code class="highlighter-rouge">accumulo-cluster</code> script.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo-cluster start
+</code></pre>
+</div>
+
+<p>This process will check the process listing, using <code class="highlighter-rouge">jps</code> on each host before attempting to restart a service on the given host.
+Typically, this check is sufficient except in the face of a hung/zombie process. For large clusters, it may be
+undesirable to ssh to every node in the cluster to ensure that all hosts are running the appropriate processes and <code class="highlighter-rouge">accumulo-service</code> may be of use.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ ssh host_with_dead_process
+$ accumulo-service tserver start
+</code></pre>
+</div>
+
+<p><strong>My process died again. Should I restart it via <code class="highlighter-rouge">cron</code> or tools like <code class="highlighter-rouge">supervisord</code>?</strong></p>
+
+<p>A repeatedly dying Accumulo process is a sign of a larger problem. Typically these problems are due to a
+misconfiguration of Accumulo or over-saturation of resources. Blind automation of any service restart inside of Accumulo
+is generally an undesirable situation as it is indicative of a problem that is being masked and ignored. Accumulo
+processes should be stable on the order of months and not require frequent restart.</p>
+
+<h2 id="accumulo-clients">Accumulo Clients</h2>
+
+<p><strong>Accumulo is not showing me any data!</strong></p>
+
+<p>Do you have your auths set so that it matches your visibilities?</p>
+
+<p><strong>What are my visibilities?</strong></p>
+
+<p>Use the <a href="/docs/unreleased/troubleshooting/tools#PrintInfo">PrintInfo</a> tool on a representative file to get some idea
+of the visibilities in the underlying data.</p>
+
+<p>Note that the use of <code class="highlighter-rouge">PrintInfo</code> is an administrative tool and can only
+by used by someone who can access the underlying Accumulo data. It
+does not provide the normal access controls in Accumulo.</p>
+
+<h2 id="hdfs">HDFS</h2>
+
+<p>Accumulo reads and writes to the Hadoop Distributed File System.
+Accumulo needs this file system available at all times for normal operations.</p>
+
+<p><strong>Accumulo is having problems “getting a block blk_1234567890123”. How do I fix it?</strong></p>
+
+<p>This troubleshooting guide does not cover HDFS, but in general, you
+want to make sure that all the datanodes are running and an fsck check
+finds the file system clean:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ hadoop fsck /accumulo
+</code></pre>
+</div>
+
+<p>You can use:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ hadoop fsck /accumulo/path/to/corrupt/file -locations -blocks -files
+</code></pre>
+</div>
+
+<p>to locate the block references of individual corrupt files and use those
+references to search the name node and individual data node logs to determine which
+servers those blocks have been assigned and then try to fix any underlying file
+system issues on those nodes.</p>
+
+<p>On a larger cluster, you may need to increase the number of Xcievers for HDFS DataNodes:</p>
+
+<div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="nt"><property></span>
+ <span class="nt"><name></span>dfs.datanode.max.xcievers<span class="nt"></name></span>
+ <span class="nt"><value></span>4096<span class="nt"></value></span>
+<span class="nt"></property></span>
+</code></pre>
+</div>
+
+<p>Verify HDFS is healthy, check the datanode logs.</p>
+
+<h2 id="zookeeper">Zookeeper</h2>
+
+<p><strong>The <code class="highlighter-rouge">accumulo init</code> command is hanging. It says something about talking to zookeeper.</strong></p>
+
+<p>Zookeeper is also a distributed service. You will need to ensure that
+it is up. You can run the zookeeper command line tool to connect to
+any one of the zookeeper servers:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ zkCli.sh -server zoohost
+...
+[zk: zoohost:2181(CONNECTED) 0]
+</code></pre>
+</div>
+
+<p>It is important to see the word <code class="highlighter-rouge">CONNECTED</code>! If you only see
+<code class="highlighter-rouge">CONNECTING</code> you will need to diagnose zookeeper errors.</p>
+
+<p>Check to make sure that zookeeper is up, and that
+<code class="highlighter-rouge">accumulo-site.xml</code> has been pointed to
+your zookeeper server(s).</p>
+
+<p><strong>Zookeeper is running, but it does not say CONNECTED</strong></p>
+
+<p>Zookeeper processes talk to each other to elect a leader. All updates
+go through the leader and propagate to a majority of all the other
+nodes. If a majority of the nodes cannot be reached, zookeeper will
+not allow updates. Zookeeper also limits the number connections to a
+server from any other single host. By default, this limit can be as small as 10
+and can be reached in some everything-on-one-machine test configurations.</p>
+
+<p>You can check the election status and connection status of clients by
+asking the zookeeper nodes for their status. You connect to zookeeper
+and ask it with the four-letter <code class="highlighter-rouge">stat</code> command:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ nc zoohost 2181
+stat
+Zookeeper version: 3.4.5-1392090, built on 09/30/2012 17:52 GMT
+Clients:
+ /127.0.0.1:58289[0](queued=0,recved=1,sent=0)
+ /127.0.0.1:60231[1](queued=0,recved=53910,sent=53915)
+
+Latency min/avg/max: 0/5/3008
+Received: 1561459
+Sent: 1561592
+Connections: 2
+Outstanding: 0
+Zxid: 0x621a3b
+Mode: standalone
+Node count: 22524
+</code></pre>
+</div>
+
+<p>Check zookeeper status, verify that it has a quorum, and has not exceeded maxClientCnxns.</p>
+
+
+ </div>
+</div>
+
+ </div>
+
+
+<footer>
+
+ <p><a href="https://www.apache.org/foundation/contributing"><img src="https://www.apache.org/images/SupportApache-small.png" alt="Support the ASF" id="asf-logo" height="100" /></a></p>
+
+ <p>Copyright © 2011-2017 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.</p>
+
+</footer>
+
+
+ </div>
+ </div>
+ </div>
+</body>
+</html>
[07/10] accumulo-website git commit: Jekyll build from master:036341f
Posted by mw...@apache.org.
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/administration/replication.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/administration/replication.html b/docs/unreleased/administration/replication.html
index 483f85a..82b2610 100644
--- a/docs/unreleased/administration/replication.html
+++ b/docs/unreleased/administration/replication.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/administration/ssl.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/administration/ssl.html b/docs/unreleased/administration/ssl.html
index 3629405..2a9e708 100644
--- a/docs/unreleased/administration/ssl.html
+++ b/docs/unreleased/administration/ssl.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/administration/tracing.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/administration/tracing.html b/docs/unreleased/administration/tracing.html
new file mode 100644
index 0000000..1a9d0e7
--- /dev/null
+++ b/docs/unreleased/administration/tracing.html
@@ -0,0 +1,714 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<meta charset="utf-8">
+<meta http-equiv="X-UA-Compatible" content="IE=edge">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<link href="https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/paper/bootstrap.min.css" rel="stylesheet" integrity="sha384-awusxf8AUojygHf2+joICySzB780jVvQaVCAt1clU3QsyAitLGul28Qxb2r1e5g+" crossorigin="anonymous">
+<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
+<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.css">
+<link href="/css/accumulo.css" rel="stylesheet" type="text/css">
+
+<title>Accumulo Documentation - Tracing</title>
+
+<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
+<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
+<script type="text/javascript" src="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.js"></script>
+<script>
+ // show location of canonical site if not currently on the canonical site
+ $(function() {
+ var host = window.location.host;
+ if (typeof host !== 'undefined' && host !== 'accumulo.apache.org') {
+ $('#non-canonical').show();
+ }
+ });
+
+ $(function() {
+ // decorate section headers with anchors
+ return $("h2, h3, h4, h5, h6").each(function(i, el) {
+ var $el, icon, id;
+ $el = $(el);
+ id = $el.attr('id');
+ icon = '<i class="fa fa-link"></i>';
+ if (id) {
+ return $el.append($("<a />").addClass("header-link").attr("href", "#" + id).html(icon));
+ }
+ });
+ });
+
+ // configure Google Analytics
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ if (ga.hasOwnProperty('loaded') && ga.loaded === true) {
+ ga('create', 'UA-50934829-1', 'apache.org');
+ ga('send', 'pageview');
+ }
+</script>
+
+</head>
+<body style="padding-top: 100px">
+
+ <nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="/"><img id="nav-logo" alt="Apache Accumulo" class="img-responsive" src="/images/accumulo-logo.png" width="200"
+ /></a>
+ </div>
+ <div class="collapse navbar-collapse" id="navbar-items">
+ <ul class="nav navbar-nav">
+ <li class="nav-link"><a href="/downloads">Download</a></li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Releases<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/release/accumulo-1.8.1/">1.8.1 (Latest)</a></li>
+ <li><a href="/release/accumulo-1.7.3/">1.7.3</a></li>
+ <li><a href="/release/accumulo-1.6.6/">1.6.6</a></li>
+ <li><a href="/release/">Archive</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/1.8/accumulo_user_manual.html">User Manual (1.8)</a></li>
+ <li><a href="/1.8/apidocs">Javadocs (1.8)</a></li>
+ <li><a href="/1.8/examples">Examples (1.8)</a></li>
+ <li><a href="/features">Features</a></li>
+ <li><a href="/glossary">Glossary</a></li>
+ <li><a href="/external-docs">External Docs</a></li>
+ <li><a href="/docs-archive/">Archive</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/get_involved">Get Involved</a></li>
+ <li><a href="/mailing_list">Mailing Lists</a></li>
+ <li><a href="/people">People</a></li>
+ <li><a href="/related-projects">Related Projects</a></li>
+ <li><a href="/contributor/">Contributor Guide</a></li>
+ </ul>
+ </li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://www.apache.org">Apache Homepage <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/licenses/LICENSE-2.0">License <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/sponsorship">Sponsorship <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/security">Security <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/thanks">Thanks <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct <i class="fa fa-external-link"></i></a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12">
+
+ <div id="non-canonical" style="display: none; background-color: #F0E68C; padding-left: 1em;">
+ Visit the official site at: <a href="https://accumulo.apache.org">https://accumulo.apache.org</a>
+ </div>
+ <div id="content">
+
+ <div class="alert alert-danger" role="alert">This documentation is for an unreleased version of Apache Accumulo that is currently under development! Check out the <a href="/1.8/accumulo_user_manual.html">documentation for the latest release</a>.</div>
+
+<div class="row">
+ <div class="col-md-3">
+ <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true">
+ <div class="panel panel-default">
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsegetting-started" aria-expanded="false" aria-controls="collapsegetting-started">
+ Getting started
+ </a>
+ </h4>
+ </div>
+ <div id="collapsegetting-started" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/design">Accumulo Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/clients">Accumulo Clients</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/shell">Accumulo Shell</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_design">Table Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_configuration">Table Configuration</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsedevelopment" aria-expanded="false" aria-controls="collapsedevelopment">
+ Development
+ </a>
+ </h4>
+ </div>
+ <div id="collapsedevelopment" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/sampling">Sampling</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/summaries">Summary Statistics</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/security">Security</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapseadministration" aria-expanded="true" aria-controls="collapseadministration">
+ Administration
+ </a>
+ </h4>
+ </div>
+ <div id="collapseadministration" class="panel-collapse collapse in" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/multivolume">Multi-Volume Installations</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsetroubleshooting" aria-expanded="false" aria-controls="collapsetroubleshooting">
+ Troubleshooting
+ </a>
+ </h4>
+ </div>
+ <div id="collapsetroubleshooting" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
+
+ </div>
+ </div>
+
+
+
+ </div>
+ </div>
+ </div>
+ <div class="col-md-9">
+
+ <p><a href="/docs/unreleased/">Accumulo unreleased docs</a> >> Administration >> Tracing</p>
+
+
+ <h1>Tracing</h1>
+
+ <p>It can be difficult to determine why some operations are taking longer
+than expected. For example, you may be looking up items with very low
+latency, but sometimes the lookups take much longer. Determining the
+cause of the delay is difficult because the system is distributed, and
+the typical lookup is fast.</p>
+
+<p>Accumulo has been instrumented to record the time that various
+operations take when tracing is turned on. The fact that tracing is
+enabled follows all the requests made on behalf of the user throughout
+the distributed infrastructure of accumulo, and across all threads of
+execution.</p>
+
+<p>These time spans will be inserted into the <code class="highlighter-rouge">trace</code> table in
+Accumulo. You can browse recent traces from the Accumulo monitor
+page. You can also read the <code class="highlighter-rouge">trace</code> table directly like any
+other table.</p>
+
+<p>The design of Accumulo’s distributed tracing follows that of <a href="http://research.google.com/pubs/pub36356.html">Google’s Dapper</a>.</p>
+
+<h2 id="tracers">Tracers</h2>
+
+<p>To collect traces, Accumulo needs at least one tracer server running. If you are using <code class="highlighter-rouge">accumulo-cluster</code> to start your cluster,
+configure your server in <code class="highlighter-rouge">conf/tracers</code>. The server collects traces from clients and writes them to the <code class="highlighter-rouge">trace</code> table. The Accumulo
+user that the tracer connects to Accumulo with can be configured with the following properties (see the <a href="/docs/unreleased/administration/configuration-management">configuration management</a>
+page for setting Accumulo server properties)</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>trace.user
+trace.token.property.password
+</code></pre>
+</div>
+
+<p>Other tracer configuration properties include</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>trace.port.client - port tracer listens on
+trace.table - table tracer writes to
+trace.zookeeper.path - zookeeper path where tracers register
+</code></pre>
+</div>
+
+<p>The zookeeper path is configured to /tracers by default. If
+multiple Accumulo instances are sharing the same ZooKeeper
+quorum, take care to configure Accumulo with unique values for
+this property.</p>
+
+<h2 id="configuring-tracing">Configuring Tracing</h2>
+
+<p>Traces are collected via SpanReceivers. The default SpanReceiver
+configured is org.apache.accumulo.core.trace.ZooTraceClient, which
+sends spans to an Accumulo Tracer process, as discussed in the
+previous section. This default can be changed to a different span
+receiver, or additional span receivers can be added in a
+comma-separated list, by modifying the property</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>trace.span.receivers
+</code></pre>
+</div>
+
+<p>Individual span receivers may require their own configuration
+parameters, which are grouped under the trace.span.receiver.*
+prefix. ZooTraceClient uses the following properties. The first
+three properties are populated from other Accumulo properties,
+while the remaining ones should be prefixed with
+trace.span.receiver. when set in the Accumulo configuration.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>tracer.zookeeper.host - populated from instance.zookeepers
+tracer.zookeeper.timeout - populated from instance.zookeeper.timeout
+tracer.zookeeper.path - populated from trace.zookeeper.path
+tracer.send.timer.millis - timer for flushing send queue (in ms, default 1000)
+tracer.queue.size - max queue size (default 5000)
+tracer.span.min.ms - minimum span length to store (in ms, default 1)
+</code></pre>
+</div>
+
+<p>Note that to configure an Accumulo client for tracing, including
+the Accumulo shell, the client configuration must be given the same
+trace.span.receivers, trace.span.receiver.*, and trace.zookeeper.path
+properties as the servers have.</p>
+
+<p>Hadoop can also be configured to send traces to Accumulo, as of
+Hadoop 2.6.0, by setting properties in Hadoop’s core-site.xml
+file. Instead of using the trace.span.receiver.* prefix, Hadoop
+uses hadoop.htrace.*. The Hadoop configuration does not have
+access to Accumulo’s properties, so the
+hadoop.htrace.tracer.zookeeper.host property must be specified.
+The zookeeper timeout defaults to 30000 (30 seconds), and the
+zookeeper path defaults to /tracers. An example of configuring
+Hadoop to send traces to ZooTraceClient is</p>
+
+<div class="language-xml highlighter-rouge"><pre class="highlight"><code> <span class="nt"><property></span>
+ <span class="nt"><name></span>hadoop.htrace.spanreceiver.classes<span class="nt"></name></span>
+ <span class="nt"><value></span>org.apache.accumulo.core.trace.ZooTraceClient<span class="nt"></value></span>
+ <span class="nt"></property></span>
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>hadoop.htrace.tracer.zookeeper.host<span class="nt"></name></span>
+ <span class="nt"><value></span>zookeeperHost:2181<span class="nt"></value></span>
+ <span class="nt"></property></span>
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>hadoop.htrace.tracer.zookeeper.path<span class="nt"></name></span>
+ <span class="nt"><value></span>/tracers<span class="nt"></value></span>
+ <span class="nt"></property></span>
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>hadoop.htrace.tracer.span.min.ms<span class="nt"></name></span>
+ <span class="nt"><value></span>1<span class="nt"></value></span>
+ <span class="nt"></property></span>
+</code></pre>
+</div>
+
+<p>The accumulo-core, accumulo-tracer, accumulo-fate and libthrift
+jars must also be placed on Hadoop’s classpath.</p>
+
+<h3 id="adding-additional-spanreceivers">Adding additional SpanReceivers</h3>
+
+<p><a href="https://github.com/openzipkin/zipkin">Zipkin</a> has a SpanReceiver supported by HTrace and popularized by Twitter
+that users looking for a more graphical trace display may opt to use.
+The following steps configure Accumulo to use <code class="highlighter-rouge">org.apache.htrace.impl.ZipkinSpanReceiver</code>
+in addition to the Accumulo’s default ZooTraceClient, and they serve as a template
+for adding any SpanReceiver to Accumulo:</p>
+
+<ol>
+ <li>
+ <p>Add the Jar containing the ZipkinSpanReceiver class file to the
+<code class="highlighter-rouge">lib/</code> directory. It is critical that the Jar is placed in
+<code class="highlighter-rouge">lib/</code> and NOT in <code class="highlighter-rouge">lib/ext/</code> so that the new SpanReceiver class
+is visible to the same class loader of htrace-core.</p>
+ </li>
+ <li>
+ <p>Add the following to <code class="highlighter-rouge">accumulo-site.xml</code>:</p>
+
+ <div class="highlighter-rouge"><pre class="highlight"><code> <property>
+ <name>trace.span.receivers</name>
+ <value>org.apache.accumulo.tracer.ZooTraceClient,org.apache.htrace.impl.ZipkinSpanReceiver</value>
+ </property>
+</code></pre>
+ </div>
+ </li>
+ <li>
+ <p>Restart your Accumulo tablet servers.</p>
+ </li>
+</ol>
+
+<p>In order to use ZipkinSpanReceiver from a client as well as the Accumulo server,</p>
+
+<ol>
+ <li>
+ <p>Ensure your client can see the ZipkinSpanReceiver class at runtime. For Maven projects,
+this is easily done by adding to your client’s pom.xml (taking care to specify a good version)</p>
+
+ <div class="highlighter-rouge"><pre class="highlight"><code> <dependency>
+ <groupId>org.apache.htrace</groupId>
+ <artifactId>htrace-zipkin</artifactId>
+ <version>3.1.0-incubating</version>
+ <scope>runtime</scope>
+ </dependency>
+</code></pre>
+ </div>
+ </li>
+ <li>
+ <p>Add the following to your client configuration.</p>
+
+ <div class="highlighter-rouge"><pre class="highlight"><code> trace.span.receivers=org.apache.accumulo.tracer.ZooTraceClient,org.apache.htrace.impl.ZipkinSpanReceiver
+</code></pre>
+ </div>
+ </li>
+ <li>
+ <p>Instrument your client as in the next section.</p>
+ </li>
+</ol>
+
+<p>Your SpanReceiver may require additional properties, and if so these should likewise
+be placed in the ClientConfiguration (if applicable) and Accumulo’s <code class="highlighter-rouge">accumulo-site.xml</code>.
+Two such properties for ZipkinSpanReceiver, listed with their default values, are</p>
+
+<div class="language-xml highlighter-rouge"><pre class="highlight"><code> <span class="nt"><property></span>
+ <span class="nt"><name></span>trace.span.receiver.zipkin.collector-hostname<span class="nt"></name></span>
+ <span class="nt"><value></span>localhost<span class="nt"></value></span>
+ <span class="nt"></property></span>
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>trace.span.receiver.zipkin.collector-port<span class="nt"></name></span>
+ <span class="nt"><value></span>9410<span class="nt"></value></span>
+ <span class="nt"></property></span>
+</code></pre>
+</div>
+
+<h3 id="instrumenting-a-client">Instrumenting a Client</h3>
+
+<p>Tracing can be used to measure a client operation, such as a scan, as
+the operation traverses the distributed system. To enable tracing for
+your application call</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.accumulo.core.trace.DistributedTrace</span><span class="o">;</span>
+<span class="o">...</span>
+<span class="n">DistributedTrace</span><span class="o">.</span><span class="na">enable</span><span class="o">(</span><span class="n">hostname</span><span class="o">,</span> <span class="s">"myApplication"</span><span class="o">);</span>
+<span class="c1">// do some tracing</span>
+<span class="o">...</span>
+<span class="n">DistributedTrace</span><span class="o">.</span><span class="na">disable</span><span class="o">();</span>
+</code></pre>
+</div>
+
+<p>Once tracing has been enabled, a client can wrap an operation in a trace.</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.htrace.Sampler</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.htrace.Trace</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.htrace.TraceScope</span><span class="o">;</span>
+<span class="o">...</span>
+<span class="n">TraceScope</span> <span class="n">scope</span> <span class="o">=</span> <span class="n">Trace</span><span class="o">.</span><span class="na">startSpan</span><span class="o">(</span><span class="s">"Client Scan"</span><span class="o">,</span> <span class="n">Sampler</span><span class="o">.</span><span class="na">ALWAYS</span><span class="o">);</span>
+<span class="n">BatchScanner</span> <span class="n">scanner</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="na">createBatchScanner</span><span class="o">(...);</span>
+<span class="c1">// Configure your scanner</span>
+<span class="k">for</span> <span class="o">(</span><span class="n">Entry</span> <span class="n">entry</span> <span class="o">:</span> <span class="n">scanner</span><span class="o">)</span> <span class="o">{</span>
+<span class="o">}</span>
+<span class="n">scope</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
+</code></pre>
+</div>
+
+<p>The user can create additional Spans within a Trace.</p>
+
+<p>The sampler (such as <code class="highlighter-rouge">Sampler.ALWAYS</code>) for the trace should only be specified with a top-level span,
+and subsequent spans will be collected depending on whether that first span was sampled.
+Don’t forget to specify a Sampler at the top-level span
+because the default Sampler only samples when part of a pre-existing trace,
+which will never occur in a client that never specifies a Sampler.</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">TraceScope</span> <span class="n">scope</span> <span class="o">=</span> <span class="n">Trace</span><span class="o">.</span><span class="na">startSpan</span><span class="o">(</span><span class="s">"Client Update"</span><span class="o">,</span> <span class="n">Sampler</span><span class="o">.</span><span class="na">ALWAYS</span><span class="o">);</span>
+<span class="o">...</span>
+<span class="n">TraceScope</span> <span class="n">readScope</span> <span class="o">=</span> <span class="n">Trace</span><span class="o">.</span><span class="na">startSpan</span><span class="o">(</span><span class="s">"Read"</span><span class="o">);</span>
+<span class="o">...</span>
+<span class="n">readScope</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
+<span class="o">...</span>
+<span class="n">TraceScope</span> <span class="n">writeScope</span> <span class="o">=</span> <span class="n">Trace</span><span class="o">.</span><span class="na">startSpan</span><span class="o">(</span><span class="s">"Write"</span><span class="o">);</span>
+<span class="o">...</span>
+<span class="n">writeScope</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
+<span class="n">scope</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
+</code></pre>
+</div>
+
+<p>Like Dapper, Accumulo tracing supports user defined annotations to associate additional data with a Trace.
+Checking whether currently tracing is necessary when using a sampler other than Sampler.ALWAYS.</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="o">...</span>
+<span class="kt">int</span> <span class="n">numberOfEntriesRead</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
+<span class="n">TraceScope</span> <span class="n">readScope</span> <span class="o">=</span> <span class="n">Trace</span><span class="o">.</span><span class="na">startSpan</span><span class="o">(</span><span class="s">"Read"</span><span class="o">);</span>
+<span class="c1">// Do the read, update the counter</span>
+<span class="o">...</span>
+<span class="k">if</span> <span class="o">(</span><span class="n">Trace</span><span class="o">.</span><span class="na">isTracing</span><span class="o">)</span>
+ <span class="n">readScope</span><span class="o">.</span><span class="na">getSpan</span><span class="o">().</span><span class="na">addKVAnnotation</span><span class="o">(</span><span class="s">"Number of Entries Read"</span><span class="o">.</span><span class="na">getBytes</span><span class="o">(</span><span class="n">StandardCharsets</span><span class="o">.</span><span class="na">UTF_8</span><span class="o">),</span>
+ <span class="n">String</span><span class="o">.</span><span class="na">valueOf</span><span class="o">(</span><span class="n">numberOfEntriesRead</span><span class="o">).</span><span class="na">getBytes</span><span class="o">(</span><span class="n">StandardCharsets</span><span class="o">.</span><span class="na">UTF_8</span><span class="o">));</span>
+</code></pre>
+</div>
+
+<p>It is also possible to add timeline annotations to your spans.
+This associates a string with a given timestamp between the start and stop times for a span.</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="o">...</span>
+<span class="n">writeScope</span><span class="o">.</span><span class="na">getSpan</span><span class="o">().</span><span class="na">addTimelineAnnotation</span><span class="o">(</span><span class="s">"Initiating Flush"</span><span class="o">);</span>
+</code></pre>
+</div>
+
+<p>Some client operations may have a high volume within your
+application. As such, you may wish to only sample a percentage of
+operations for tracing. As seen below, the CountSampler can be used to
+help enable tracing for 1-in-1000 operations</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.htrace.impl.CountSampler</span><span class="o">;</span>
+<span class="o">...</span>
+<span class="n">Sampler</span> <span class="n">sampler</span> <span class="o">=</span> <span class="k">new</span> <span class="n">CountSampler</span><span class="o">(</span><span class="n">HTraceConfiguration</span><span class="o">.</span><span class="na">fromMap</span><span class="o">(</span>
+ <span class="n">Collections</span><span class="o">.</span><span class="na">singletonMap</span><span class="o">(</span><span class="n">CountSampler</span><span class="o">.</span><span class="na">SAMPLER_FREQUENCY_CONF_KEY</span><span class="o">,</span> <span class="s">"1000"</span><span class="o">)));</span>
+<span class="o">...</span>
+<span class="n">TraceScope</span> <span class="n">readScope</span> <span class="o">=</span> <span class="n">Trace</span><span class="o">.</span><span class="na">startSpan</span><span class="o">(</span><span class="s">"Read"</span><span class="o">,</span> <span class="n">sampler</span><span class="o">);</span>
+<span class="o">...</span>
+<span class="n">readScope</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
+</code></pre>
+</div>
+
+<p>Remember to close all spans and disable tracing when finished.</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">DistributedTrace</span><span class="o">.</span><span class="na">disable</span><span class="o">();</span>
+</code></pre>
+</div>
+
+<h2 id="viewing-collected-traces">Viewing Collected Traces</h2>
+
+<p>To view collected traces, use the “Recent Traces” link on the Monitor
+UI. You can also programmatically access and print traces using the
+<code class="highlighter-rouge">TraceDump</code> class.</p>
+
+<h3 id="trace-table-format">Trace Table Format</h3>
+
+<p>This section is for developers looking to use data recorded in the trace table
+directly, above and beyond the default services of the Accumulo monitor.
+Please note the trace table format and its supporting classes
+are not in the public API and may be subject to change in future versions.</p>
+
+<p>Each span received by a tracer’s ZooTraceClient is recorded in the trace table
+in the form of three entries: span entries, index entries, and start time entries.
+Span and start time entries record full span information,
+whereas index entries provide indexing into span information
+useful for quickly finding spans by type or start time.</p>
+
+<p>Each entry is illustrated by a description and sample of data.
+In the description, a token in quotes is a String literal,
+whereas other other tokens are span variables.
+Parentheses group parts together, to distinguish colon characters inside the
+column family or qualifier from the colon that separates column family and qualifier.
+We use the format <code class="highlighter-rouge">row columnFamily:columnQualifier columnVisibility value</code>
+(omitting timestamp which records the time an entry is written to the trace table).</p>
+
+<p>Span entries take the following form:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>traceId "span":(parentSpanId:spanId) [] spanBinaryEncoding
+63b318de80de96d1 span:4b8f66077df89de1:3778c6739afe4e1 [] %18;%09;...
+</code></pre>
+</div>
+
+<p>The parentSpanId is “” for the root span of a trace.
+The spanBinaryEncoding is a compact Apache Thrift encoding of the original Span object.
+This allows clients (and the Accumulo monitor) to recover all the details of the original Span
+at a later time, by scanning the trace table and decoding the value of span entries
+via <code class="highlighter-rouge">TraceFormatter.getRemoteSpan(entry)</code>.</p>
+
+<p>The trace table has a formatter class by default (org.apache.accumulo.tracer.TraceFormatter)
+that changes how span entries appear from the Accumulo shell.
+Normal scans to the trace table do not use this formatter representation;
+it exists only to make span entries easier to view inside the Accumulo shell.</p>
+
+<p>Index entries take the following form:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>"idx":service:startTime description:sender [] traceId:elapsedTime
+idx:tserver:14f3828f58b startScan:localhost [] 63b318de80de96d1:1
+</code></pre>
+</div>
+
+<p>The service and sender are set by the first call of each Accumulo process
+(and instrumented client processes) to <code class="highlighter-rouge">DistributedTrace.enable(...)</code>
+(the sender is autodetected if not specified).
+The description is specified in each span.
+Start time and the elapsed time (start - stop, 1 millisecond in the example above)
+are recorded in milliseconds as long values serialized to a string in hex.</p>
+
+<p>Start time entries take the following form:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>"start":startTime "id":traceId [] spanBinaryEncoding
+start:14f3828a351 id:63b318de80de96d1 [] %18;%09;...
+</code></pre>
+</div>
+
+<p>The following classes may be run while Accumulo is running to provide insight into trace statistics. These require
+accumulo-trace-VERSION.jar to be provided on the Accumulo classpath (<code class="highlighter-rouge">lib/ext</code> is fine).</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.tracer.TraceTableStats -u username -p password -i instancename
+$ accumulo org.apache.accumulo.tracer.TraceDump -u username -p password -i instancename -r
+</code></pre>
+</div>
+
+<h3 id="tracing-from-the-shell">Tracing from the Shell</h3>
+<p>You can enable tracing for operations run from the shell by using the
+<code class="highlighter-rouge">trace on</code> and <code class="highlighter-rouge">trace off</code> commands.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>root@test test> trace on
+
+root@test test> scan
+a b:c [] d
+
+root@test test> trace off
+Waiting for trace information
+Waiting for trace information
+Trace started at 2013/08/26 13:24:08.332
+Time Start Service@Location Name
+ 3628+0 shell@localhost shell:root
+ 8+1690 shell@localhost scan
+ 7+1691 shell@localhost scan:location
+ 6+1692 tserver@localhost startScan
+ 5+1692 tserver@localhost tablet read ahead 6
+</code></pre>
+</div>
+
+
+ </div>
+</div>
+
+ </div>
+
+
+<footer>
+
+ <p><a href="https://www.apache.org/foundation/contributing"><img src="https://www.apache.org/images/SupportApache-small.png" alt="Support the ASF" id="asf-logo" height="100" /></a></p>
+
+ <p>Copyright © 2011-2017 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.</p>
+
+</footer>
+
+
+ </div>
+ </div>
+ </div>
+</body>
+</html>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/development/analytics.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/development/analytics.html b/docs/unreleased/development/analytics.html
index 62e5aa9..97f2e27 100644
--- a/docs/unreleased/development/analytics.html
+++ b/docs/unreleased/development/analytics.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/development/development_tools.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/development/development_tools.html b/docs/unreleased/development/development_tools.html
index e815fa2..c8afef8 100644
--- a/docs/unreleased/development/development_tools.html
+++ b/docs/unreleased/development/development_tools.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/development/high_speed_ingest.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/development/high_speed_ingest.html b/docs/unreleased/development/high_speed_ingest.html
index fe83f81..c81e311 100644
--- a/docs/unreleased/development/high_speed_ingest.html
+++ b/docs/unreleased/development/high_speed_ingest.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/development/iterator_design.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/development/iterator_design.html b/docs/unreleased/development/iterator_design.html
index 20fa293..2dfb4c7 100644
--- a/docs/unreleased/development/iterator_design.html
+++ b/docs/unreleased/development/iterator_design.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/development/iterator_testing.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/development/iterator_testing.html b/docs/unreleased/development/iterator_testing.html
index 7decc39..b6a8437 100644
--- a/docs/unreleased/development/iterator_testing.html
+++ b/docs/unreleased/development/iterator_testing.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/development/sampling.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/development/sampling.html b/docs/unreleased/development/sampling.html
index 7d56089..4592c40 100644
--- a/docs/unreleased/development/sampling.html
+++ b/docs/unreleased/development/sampling.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/development/security.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/development/security.html b/docs/unreleased/development/security.html
index da0badf..f6b8038 100644
--- a/docs/unreleased/development/security.html
+++ b/docs/unreleased/development/security.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/development/summaries.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/development/summaries.html b/docs/unreleased/development/summaries.html
index af4e069..ff91054 100644
--- a/docs/unreleased/development/summaries.html
+++ b/docs/unreleased/development/summaries.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/getting-started/clients.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/getting-started/clients.html b/docs/unreleased/getting-started/clients.html
index f50c6dc..4a82259 100644
--- a/docs/unreleased/getting-started/clients.html
+++ b/docs/unreleased/getting-started/clients.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/getting-started/design.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/getting-started/design.html b/docs/unreleased/getting-started/design.html
index 2a3c045..e803da1 100644
--- a/docs/unreleased/getting-started/design.html
+++ b/docs/unreleased/getting-started/design.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/getting-started/shell.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/getting-started/shell.html b/docs/unreleased/getting-started/shell.html
index e245c59..626602b 100644
--- a/docs/unreleased/getting-started/shell.html
+++ b/docs/unreleased/getting-started/shell.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/getting-started/table_configuration.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/getting-started/table_configuration.html b/docs/unreleased/getting-started/table_configuration.html
index 7337139..9c8fc2d 100644
--- a/docs/unreleased/getting-started/table_configuration.html
+++ b/docs/unreleased/getting-started/table_configuration.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/getting-started/table_design.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/getting-started/table_design.html b/docs/unreleased/getting-started/table_design.html
index c4b9d3f..7d0d9b3 100644
--- a/docs/unreleased/getting-started/table_design.html
+++ b/docs/unreleased/getting-started/table_design.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/index.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/index.html b/docs/unreleased/index.html
index 5d756b3..cdf707c 100644
--- a/docs/unreleased/index.html
+++ b/docs/unreleased/index.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
[09/10] accumulo-website git commit: Jekyll build from master:036341f
Posted by mw...@apache.org.
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/administration/monitoring-metrics.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/administration/monitoring-metrics.html b/docs/unreleased/administration/monitoring-metrics.html
new file mode 100644
index 0000000..c407dc7
--- /dev/null
+++ b/docs/unreleased/administration/monitoring-metrics.html
@@ -0,0 +1,432 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<meta charset="utf-8">
+<meta http-equiv="X-UA-Compatible" content="IE=edge">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<link href="https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/paper/bootstrap.min.css" rel="stylesheet" integrity="sha384-awusxf8AUojygHf2+joICySzB780jVvQaVCAt1clU3QsyAitLGul28Qxb2r1e5g+" crossorigin="anonymous">
+<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
+<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.css">
+<link href="/css/accumulo.css" rel="stylesheet" type="text/css">
+
+<title>Accumulo Documentation - Monitoring & Metrics</title>
+
+<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
+<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
+<script type="text/javascript" src="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.js"></script>
+<script>
+ // show location of canonical site if not currently on the canonical site
+ $(function() {
+ var host = window.location.host;
+ if (typeof host !== 'undefined' && host !== 'accumulo.apache.org') {
+ $('#non-canonical').show();
+ }
+ });
+
+ $(function() {
+ // decorate section headers with anchors
+ return $("h2, h3, h4, h5, h6").each(function(i, el) {
+ var $el, icon, id;
+ $el = $(el);
+ id = $el.attr('id');
+ icon = '<i class="fa fa-link"></i>';
+ if (id) {
+ return $el.append($("<a />").addClass("header-link").attr("href", "#" + id).html(icon));
+ }
+ });
+ });
+
+ // configure Google Analytics
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ if (ga.hasOwnProperty('loaded') && ga.loaded === true) {
+ ga('create', 'UA-50934829-1', 'apache.org');
+ ga('send', 'pageview');
+ }
+</script>
+
+</head>
+<body style="padding-top: 100px">
+
+ <nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="/"><img id="nav-logo" alt="Apache Accumulo" class="img-responsive" src="/images/accumulo-logo.png" width="200"
+ /></a>
+ </div>
+ <div class="collapse navbar-collapse" id="navbar-items">
+ <ul class="nav navbar-nav">
+ <li class="nav-link"><a href="/downloads">Download</a></li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Releases<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/release/accumulo-1.8.1/">1.8.1 (Latest)</a></li>
+ <li><a href="/release/accumulo-1.7.3/">1.7.3</a></li>
+ <li><a href="/release/accumulo-1.6.6/">1.6.6</a></li>
+ <li><a href="/release/">Archive</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/1.8/accumulo_user_manual.html">User Manual (1.8)</a></li>
+ <li><a href="/1.8/apidocs">Javadocs (1.8)</a></li>
+ <li><a href="/1.8/examples">Examples (1.8)</a></li>
+ <li><a href="/features">Features</a></li>
+ <li><a href="/glossary">Glossary</a></li>
+ <li><a href="/external-docs">External Docs</a></li>
+ <li><a href="/docs-archive/">Archive</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/get_involved">Get Involved</a></li>
+ <li><a href="/mailing_list">Mailing Lists</a></li>
+ <li><a href="/people">People</a></li>
+ <li><a href="/related-projects">Related Projects</a></li>
+ <li><a href="/contributor/">Contributor Guide</a></li>
+ </ul>
+ </li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://www.apache.org">Apache Homepage <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/licenses/LICENSE-2.0">License <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/sponsorship">Sponsorship <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/security">Security <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/thanks">Thanks <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct <i class="fa fa-external-link"></i></a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12">
+
+ <div id="non-canonical" style="display: none; background-color: #F0E68C; padding-left: 1em;">
+ Visit the official site at: <a href="https://accumulo.apache.org">https://accumulo.apache.org</a>
+ </div>
+ <div id="content">
+
+ <div class="alert alert-danger" role="alert">This documentation is for an unreleased version of Apache Accumulo that is currently under development! Check out the <a href="/1.8/accumulo_user_manual.html">documentation for the latest release</a>.</div>
+
+<div class="row">
+ <div class="col-md-3">
+ <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true">
+ <div class="panel panel-default">
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsegetting-started" aria-expanded="false" aria-controls="collapsegetting-started">
+ Getting started
+ </a>
+ </h4>
+ </div>
+ <div id="collapsegetting-started" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/design">Accumulo Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/clients">Accumulo Clients</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/shell">Accumulo Shell</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_design">Table Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_configuration">Table Configuration</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsedevelopment" aria-expanded="false" aria-controls="collapsedevelopment">
+ Development
+ </a>
+ </h4>
+ </div>
+ <div id="collapsedevelopment" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/sampling">Sampling</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/summaries">Summary Statistics</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/security">Security</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapseadministration" aria-expanded="true" aria-controls="collapseadministration">
+ Administration
+ </a>
+ </h4>
+ </div>
+ <div id="collapseadministration" class="panel-collapse collapse in" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/multivolume">Multi-Volume Installations</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsetroubleshooting" aria-expanded="false" aria-controls="collapsetroubleshooting">
+ Troubleshooting
+ </a>
+ </h4>
+ </div>
+ <div id="collapsetroubleshooting" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
+
+ </div>
+ </div>
+
+
+
+ </div>
+ </div>
+ </div>
+ <div class="col-md-9">
+
+ <p><a href="/docs/unreleased/">Accumulo unreleased docs</a> >> Administration >> Monitoring & Metrics</p>
+
+
+ <h1>Monitoring & Metrics</h1>
+
+ <h2 id="monitoring">Monitoring</h2>
+
+<h3 id="accumulo-monitor">Accumulo Monitor</h3>
+
+<p>The Accumulo Monitor provides an interface for monitoring the status and health of
+Accumulo components. The Accumulo Monitor provides a web UI for accessing this information at
+<code class="highlighter-rouge">http://_monitorhost_:9995/</code>.</p>
+
+<p>Things highlighted in yellow may be in need of attention.
+If anything is highlighted in red on the monitor page, it is something that definitely needs attention.</p>
+
+<p>The Overview page contains some summary information about the Accumulo instance, including the version, instance name, and instance ID.
+There is a table labeled Accumulo Master with current status, a table listing the active Zookeeper servers, and graphs displaying various metrics over time.
+These include ingest and scan performance and other useful measurements.</p>
+
+<p>The Master Server, Tablet Servers, and Tables pages display metrics grouped in different ways (e.g. by tablet server or by table).
+Metrics typically include number of entries (key/value pairs), ingest and query rates.
+The number of running scans, major and minor compactions are in the form <em>number_running</em> (<em>number_queued</em>).
+Another important metric is hold time, which is the amount of time a tablet has been waiting but unable to flush its memory in a minor compaction.</p>
+
+<p>The Server Activity page graphically displays tablet server status, with each server represented as a circle or square.
+Different metrics may be assigned to the nodes’ color and speed of oscillation.
+The Overall Avg metric is only used on the Server Activity page, and represents the average of all the other metrics (after normalization).
+Similarly, the Overall Max metric picks the metric with the maximum normalized value.</p>
+
+<p>The Garbage Collector page displays a list of garbage collection cycles, the number of files found of each type (including deletion candidates in use and files actually deleted), and the length of the deletion cycle.
+The Traces page displays data for recent traces performed (see the following section for information on <a href="/docs/unreleased/administration/tracing">tracing</a>).
+The Recent Logs page displays warning and error logs forwarded to the monitor from all Accumulo processes.
+Also, the XML and JSON links provide metrics in XML and JSON formats, respectively.</p>
+
+<h3 id="ssl">SSL</h3>
+
+<p>SSL may be enabled for the monitor page by setting the following properties in the <code class="highlighter-rouge">accumulo-site.xml</code> file:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>monitor.ssl.keyStore
+monitor.ssl.keyStorePassword
+monitor.ssl.trustStore
+monitor.ssl.trustStorePassword
+</code></pre>
+</div>
+
+<p>If the Accumulo conf directory has been configured (in particular the <code class="highlighter-rouge">accumulo-env.sh</code> file must be set up), the
+<code class="highlighter-rouge">accumulo-util gen-monitor-cert</code> command can be used to create the keystore and truststore files with random passwords. The command
+will print out the properties that need to be added to the <code class="highlighter-rouge">accumulo-site.xml</code> file. The stores can also be generated manually with the
+Java <code class="highlighter-rouge">keytool</code> command, whose usage can be seen in the <code class="highlighter-rouge">accumulo-util</code> script.</p>
+
+<p>If desired, the SSL ciphers allowed for connections can be controlled via the following properties in <code class="highlighter-rouge">accumulo-site.xml</code>:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>monitor.ssl.include.ciphers
+monitor.ssl.exclude.ciphers
+</code></pre>
+</div>
+
+<p>If SSL is enabled, the monitor URL can only be accessed via https.
+This also allows you to access the Accumulo shell through the monitor page.
+The left navigation bar will have a new link to Shell.
+An Accumulo user name and password must be entered for access to the shell.</p>
+
+<h2 id="metrics">Metrics</h2>
+
+<p>Accumulo can expose metrics through a legacy metrics library and using the Hadoop Metrics2 library.</p>
+
+<h3 id="legacy-metrics">Legacy Metrics</h3>
+
+<p>Accumulo has a legacy metrics library that can be exposes metrics using JMX endpoints or file-based logging. These metrics can
+be enabled by setting <code class="highlighter-rouge">general.legacy.metrics</code> to <code class="highlighter-rouge">true</code> in <code class="highlighter-rouge">accumulo-site.xml</code> and placing the <code class="highlighter-rouge">accumulo-metrics.xml</code>
+configuration file on the classpath (which is typically done by placing the file in the <code class="highlighter-rouge">conf/</code> directory). A template for
+<code class="highlighter-rouge">accumulo-metrics.xml</code> can be found in <code class="highlighter-rouge">conf/templates</code> of the Accumulo tarball.</p>
+
+<h3 id="hadoop-metrics2">Hadoop Metrics2</h3>
+
+<p>Hadoop Metrics2 is a library which allows for routing of metrics generated by registered MetricsSources to
+configured MetricsSinks. Examples of sinks that are implemented by Hadoop include file-based logging, Graphite and Ganglia.
+All metric sources are exposed via JMX when using Metrics2.</p>
+
+<p>Metrics2 is configured by examining the classpath for a file that matches <code class="highlighter-rouge">hadoop-metrics2*.properties</code>. The Accumulo tarball
+contains an example <code class="highlighter-rouge">hadoop-metrics2-accumulo.properties</code> file in <code class="highlighter-rouge">conf/templates</code> which can be copied to <code class="highlighter-rouge">conf/</code> to place
+on classpath. This file is used to enable file, Graphite or Ganglia sinks (some minimal configuration required for Graphite
+and Ganglia). Because the Hadoop configuration is also on the Accumulo classpath, be sure that you do not have multiple
+Metrics2 configuration files. It is recommended to consolidate metrics in a single properties file in a central location to
+remove ambiguity. The contents of <code class="highlighter-rouge">hadoop-metrics2-accumulo.properties</code> can be added to a central <code class="highlighter-rouge">hadoop-metrics2.properties</code>
+in <code class="highlighter-rouge">$HADOOP_CONF_DIR</code>.</p>
+
+<p>As a note for configuring the file sink, the provided path should be absolute. A relative path or file name will be created relative
+to the directory in which the Accumulo process was started. External tools, such as logrotate, can be used to prevent these files
+from growing without bound.</p>
+
+<p>Each server process should have log messages from the Metrics2 library about the sinks that were created. Be sure to check
+the Accumulo processes log files when debugging missing metrics output.</p>
+
+<p>For additional information on configuring Metrics2, visit the <a href="https://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html">Javadoc page for Metrics2</a>.</p>
+
+
+ </div>
+</div>
+
+ </div>
+
+
+<footer>
+
+ <p><a href="https://www.apache.org/foundation/contributing"><img src="https://www.apache.org/images/SupportApache-small.png" alt="Support the ASF" id="asf-logo" height="100" /></a></p>
+
+ <p>Copyright © 2011-2017 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.</p>
+
+</footer>
+
+
+ </div>
+ </div>
+ </div>
+</body>
+</html>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/administration/multivolume.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/administration/multivolume.html b/docs/unreleased/administration/multivolume.html
index e5abe91..8ed1b03 100644
--- a/docs/unreleased/administration/multivolume.html
+++ b/docs/unreleased/administration/multivolume.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
[04/10] accumulo-website git commit: Jekyll build from master:036341f
Posted by mw...@apache.org.
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/troubleshooting/tools.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/troubleshooting/tools.html b/docs/unreleased/troubleshooting/tools.html
new file mode 100644
index 0000000..00369a1
--- /dev/null
+++ b/docs/unreleased/troubleshooting/tools.html
@@ -0,0 +1,489 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<meta charset="utf-8">
+<meta http-equiv="X-UA-Compatible" content="IE=edge">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<link href="https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/paper/bootstrap.min.css" rel="stylesheet" integrity="sha384-awusxf8AUojygHf2+joICySzB780jVvQaVCAt1clU3QsyAitLGul28Qxb2r1e5g+" crossorigin="anonymous">
+<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
+<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.css">
+<link href="/css/accumulo.css" rel="stylesheet" type="text/css">
+
+<title>Accumulo Documentation - Troubleshooting Tools</title>
+
+<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
+<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
+<script type="text/javascript" src="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.js"></script>
+<script>
+ // show location of canonical site if not currently on the canonical site
+ $(function() {
+ var host = window.location.host;
+ if (typeof host !== 'undefined' && host !== 'accumulo.apache.org') {
+ $('#non-canonical').show();
+ }
+ });
+
+ $(function() {
+ // decorate section headers with anchors
+ return $("h2, h3, h4, h5, h6").each(function(i, el) {
+ var $el, icon, id;
+ $el = $(el);
+ id = $el.attr('id');
+ icon = '<i class="fa fa-link"></i>';
+ if (id) {
+ return $el.append($("<a />").addClass("header-link").attr("href", "#" + id).html(icon));
+ }
+ });
+ });
+
+ // configure Google Analytics
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ if (ga.hasOwnProperty('loaded') && ga.loaded === true) {
+ ga('create', 'UA-50934829-1', 'apache.org');
+ ga('send', 'pageview');
+ }
+</script>
+
+</head>
+<body style="padding-top: 100px">
+
+ <nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="/"><img id="nav-logo" alt="Apache Accumulo" class="img-responsive" src="/images/accumulo-logo.png" width="200"
+ /></a>
+ </div>
+ <div class="collapse navbar-collapse" id="navbar-items">
+ <ul class="nav navbar-nav">
+ <li class="nav-link"><a href="/downloads">Download</a></li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Releases<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/release/accumulo-1.8.1/">1.8.1 (Latest)</a></li>
+ <li><a href="/release/accumulo-1.7.3/">1.7.3</a></li>
+ <li><a href="/release/accumulo-1.6.6/">1.6.6</a></li>
+ <li><a href="/release/">Archive</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/1.8/accumulo_user_manual.html">User Manual (1.8)</a></li>
+ <li><a href="/1.8/apidocs">Javadocs (1.8)</a></li>
+ <li><a href="/1.8/examples">Examples (1.8)</a></li>
+ <li><a href="/features">Features</a></li>
+ <li><a href="/glossary">Glossary</a></li>
+ <li><a href="/external-docs">External Docs</a></li>
+ <li><a href="/docs-archive/">Archive</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/get_involved">Get Involved</a></li>
+ <li><a href="/mailing_list">Mailing Lists</a></li>
+ <li><a href="/people">People</a></li>
+ <li><a href="/related-projects">Related Projects</a></li>
+ <li><a href="/contributor/">Contributor Guide</a></li>
+ </ul>
+ </li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://www.apache.org">Apache Homepage <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/licenses/LICENSE-2.0">License <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/sponsorship">Sponsorship <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/security">Security <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/thanks">Thanks <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct <i class="fa fa-external-link"></i></a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12">
+
+ <div id="non-canonical" style="display: none; background-color: #F0E68C; padding-left: 1em;">
+ Visit the official site at: <a href="https://accumulo.apache.org">https://accumulo.apache.org</a>
+ </div>
+ <div id="content">
+
+ <div class="alert alert-danger" role="alert">This documentation is for an unreleased version of Apache Accumulo that is currently under development! Check out the <a href="/1.8/accumulo_user_manual.html">documentation for the latest release</a>.</div>
+
+<div class="row">
+ <div class="col-md-3">
+ <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true">
+ <div class="panel panel-default">
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsegetting-started" aria-expanded="false" aria-controls="collapsegetting-started">
+ Getting started
+ </a>
+ </h4>
+ </div>
+ <div id="collapsegetting-started" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/design">Accumulo Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/clients">Accumulo Clients</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/shell">Accumulo Shell</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_design">Table Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_configuration">Table Configuration</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsedevelopment" aria-expanded="false" aria-controls="collapsedevelopment">
+ Development
+ </a>
+ </h4>
+ </div>
+ <div id="collapsedevelopment" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/sampling">Sampling</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/summaries">Summary Statistics</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/security">Security</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapseadministration" aria-expanded="false" aria-controls="collapseadministration">
+ Administration
+ </a>
+ </h4>
+ </div>
+ <div id="collapseadministration" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/multivolume">Multi-Volume Installations</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsetroubleshooting" aria-expanded="true" aria-controls="collapsetroubleshooting">
+ Troubleshooting
+ </a>
+ </h4>
+ </div>
+ <div id="collapsetroubleshooting" class="panel-collapse collapse in" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
+
+ </div>
+ </div>
+
+
+
+ </div>
+ </div>
+ </div>
+ <div class="col-md-9">
+
+ <p><a href="/docs/unreleased/">Accumulo unreleased docs</a> >> Troubleshooting >> Troubleshooting Tools</p>
+
+
+ <h1>Troubleshooting Tools</h1>
+
+ <p>The <code class="highlighter-rouge">accumulo</code> command can be used to run classes from the command line.</p>
+
+<h2 id="printinfo">PrintInfo</h2>
+
+<p>The <code class="highlighter-rouge">PrintInfo</code> tool will examine an Accumulo storage file and print out basic metadata.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.core.file.rfile.PrintInfo /accumulo/tables/1/default_tablet/A000000n.rf
+2013-07-16 08:17:14,778 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
+Locality group : <DEFAULT>
+ Start block : 0
+ Num blocks : 1
+ Index level 0 : 62 bytes 1 blocks
+ First key : 288be9ab4052fe9e span:34078a86a723e5d3:3da450f02108ced5 [] 1373373521623 false
+ Last key : start:13fc375709e id:615f5ee2dd822d7a [] 1373373821660 false
+ Num entries : 466
+ Column families : [waitForCommits, start, md major compactor 1, md major compactor 2, md major compactor 3,
+ bringOnline, prep, md major compactor 4, md major compactor 5, md root major compactor 3,
+ minorCompaction, wal, compactFiles, md root major compactor 4, md root major compactor 1,
+ md root major compactor 2, compact, id, client:update, span, update, commit, write,
+ majorCompaction]
+
+Meta block : BCFile.index
+ Raw size : 4 bytes
+ Compressed size : 12 bytes
+ Compression type : gz
+
+Meta block : RFile.index
+ Raw size : 780 bytes
+ Compressed size : 344 bytes
+ Compression type : gz
+</code></pre>
+</div>
+
+<p>When trying to diagnose problems related to key size, the <code class="highlighter-rouge">PrintInfo</code> tool can provide a histogram of the individual key sizes:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.core.file.rfile.PrintInfo --histogram /accumulo/tables/1/default_tablet/A000000n.rf
+...
+Up to size count %-age
+ 10 : 222 28.23%
+ 100 : 244 71.77%
+ 1000 : 0 0.00%
+ 10000 : 0 0.00%
+ 100000 : 0 0.00%
+ 1000000 : 0 0.00%
+ 10000000 : 0 0.00%
+ 100000000 : 0 0.00%
+ 1000000000 : 0 0.00%
+10000000000 : 0 0.00%
+</code></pre>
+</div>
+
+<p>Likewise, <code class="highlighter-rouge">PrintInfo</code> will dump the key-value pairs and show you the contents of the RFile:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.core.file.rfile.PrintInfo --dump /accumulo/tables/1/default_tablet/A000000n.rf
+row columnFamily:columnQualifier [visibility] timestamp deleteFlag -> Value
+...
+</code></pre>
+</div>
+
+<h2 id="getmasterstats">GetMasterStats</h2>
+
+<p>The <code class="highlighter-rouge">GetMasterStats</code> tool can be used to retrieve Accumulo state and statistics:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.test.GetMasterStats | grep Load
+ OS Load Average: 0.27
+</code></pre>
+</div>
+
+<h2 id="findofflinetablets">FindOfflineTablets</h2>
+
+<p>If the Accumulo monitor shows an offline tablet, use <code class="highlighter-rouge">FindOfflineTablets</code> to find out which
+tablet it is.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.server.util.FindOfflineTablets
+2<<@(null,null,localhost:9997) is UNASSIGNED #walogs:2
+</code></pre>
+</div>
+
+<p>Here’s what the output means:</p>
+
+<ul>
+ <li>
+ <p><code class="highlighter-rouge">2<<</code> -
+ This is the tablet from (-inf, pass:[+]inf) for the
+ table with id 2. The command <code class="highlighter-rouge">tables -l</code> in the shell will show table ids for
+ tables.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">@(null, null, localhost:9997)</code> -
+ Location information. The
+ format is <code class="highlighter-rouge">@(assigned, hosted, last)</code>. In this case, the
+ tablet has not been assigned, is not hosted anywhere, and was once
+ hosted on localhost.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">#walogs:2</code> -
+ The number of write-ahead logs that this tablet requires for recovery.</p>
+ </li>
+</ul>
+
+<p>An unassigned tablet with write-ahead logs is probably waiting for
+logs to be sorted for efficient recovery.</p>
+
+<h2 id="checkformetadataproblems">CheckForMetadataProblems</h2>
+
+<p>The <code class="highlighter-rouge">CheckForMetadataProblems</code> tool can be used to make sure metadata
+tables are up and consistent. It will verify the start/end of
+every tablet matches, and the start and stop for the table is empty:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.server.util.CheckForMetadataProblems -u root --password
+Enter the connection password:
+All is well for table !0
+All is well for table 1
+</code></pre>
+</div>
+
+<h2 id="removeentriesformissingfiles">RemoveEntriesForMissingFiles</h2>
+
+<p>If your Hadoop cluster has a lost a file due to a NameNode failure, you can remove the
+the file reference using <code class="highlighter-rouge">RemoveEntriesForMissingFiles</code>. It will check every file reference
+and ensure that the file exists in HDFS. Optionally, it will remove the reference:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.server.util.RemoveEntriesForMissingFiles -u root --password
+Enter the connection password:
+2013-07-16 13:10:57,293 [util.RemoveEntriesForMissingFiles] INFO : File /accumulo/tables/2/default_tablet/F0000005.rf
+ is missing
+2013-07-16 13:10:57,296 [util.RemoveEntriesForMissingFiles] INFO : 1 files of 3 missing
+</code></pre>
+</div>
+
+<h2 id="cleanzookeeper">CleanZookeeper</h2>
+
+<p>If you have entries in zookeeper for old instances that you no longer need, remove them using CleanZookeeper:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.server.util.CleanZookeeper
+</code></pre>
+</div>
+
+<p>This command will not delete the instance pointed to by the local <code class="highlighter-rouge">accumulo-site.xml</code> file.</p>
+
+<h2 id="dumpzookeeper--restorezookeeper">DumpZookeeper & RestoreZookeeper</h2>
+
+<p>If you would like to backup, or otherwise examine the contents of Zookeeper, there are commands to dump and load to/from XML.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.server.util.DumpZookeeper --root /accumulo >dump.xml
+$ accumulo org.apache.accumulo.server.util.RestoreZookeeper --overwrite < dump.xml
+</code></pre>
+</div>
+
+
+ </div>
+</div>
+
+ </div>
+
+
+<footer>
+
+ <p><a href="https://www.apache.org/foundation/contributing"><img src="https://www.apache.org/images/SupportApache-small.png" alt="Support the ASF" id="asf-logo" height="100" /></a></p>
+
+ <p>Copyright © 2011-2017 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.</p>
+
+</footer>
+
+
+ </div>
+ </div>
+ </div>
+</body>
+</html>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/feed.xml
----------------------------------------------------------------------
diff --git a/feed.xml b/feed.xml
index e89a61e..c68428d 100644
--- a/feed.xml
+++ b/feed.xml
@@ -6,8 +6,8 @@
</description>
<link>https://accumulo.apache.org/</link>
<atom:link href="https://accumulo.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Mon, 22 May 2017 14:01:06 -0400</pubDate>
- <lastBuildDate>Mon, 22 May 2017 14:01:06 -0400</lastBuildDate>
+ <pubDate>Mon, 22 May 2017 16:09:46 -0400</pubDate>
+ <lastBuildDate>Mon, 22 May 2017 16:09:46 -0400</lastBuildDate>
<generator>Jekyll v3.3.1</generator>
<item>
[10/10] accumulo-website git commit: Jekyll build from master:036341f
Posted by mw...@apache.org.
Jekyll build from master:036341f
Split up long administration and troubleshooting overview pages
Project: http://git-wip-us.apache.org/repos/asf/accumulo-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo-website/commit/3e4443c1
Tree: http://git-wip-us.apache.org/repos/asf/accumulo-website/tree/3e4443c1
Diff: http://git-wip-us.apache.org/repos/asf/accumulo-website/diff/3e4443c1
Branch: refs/heads/asf-site
Commit: 3e4443c117adf5c8f84f64e17009620b4498d2db
Parents: 652e2a5
Author: Mike Walch <mw...@apache.org>
Authored: Mon May 22 16:10:05 2017 -0400
Committer: Mike Walch <mw...@apache.org>
Committed: Mon May 22 16:10:05 2017 -0400
----------------------------------------------------------------------
.../configuration-management.html | 18 +-
.../configuration-properties.html | 18 +-
docs/unreleased/administration/fate.html | 18 +-
.../unreleased/administration/installation.html | 1128 +++++++++++++
docs/unreleased/administration/kerberos.html | 18 +-
.../administration/monitoring-metrics.html | 432 +++++
docs/unreleased/administration/multivolume.html | 18 +-
docs/unreleased/administration/overview.html | 1586 ------------------
docs/unreleased/administration/replication.html | 18 +-
docs/unreleased/administration/ssl.html | 18 +-
docs/unreleased/administration/tracing.html | 714 ++++++++
docs/unreleased/development/analytics.html | 18 +-
.../development/development_tools.html | 18 +-
.../development/high_speed_ingest.html | 18 +-
.../unreleased/development/iterator_design.html | 18 +-
.../development/iterator_testing.html | 18 +-
docs/unreleased/development/sampling.html | 18 +-
docs/unreleased/development/security.html | 18 +-
docs/unreleased/development/summaries.html | 18 +-
docs/unreleased/getting-started/clients.html | 18 +-
docs/unreleased/getting-started/design.html | 18 +-
docs/unreleased/getting-started/shell.html | 18 +-
.../getting-started/table_configuration.html | 18 +-
.../getting-started/table_design.html | 18 +-
docs/unreleased/index.html | 18 +-
docs/unreleased/troubleshooting/advanced.html | 728 ++++++++
docs/unreleased/troubleshooting/basic.html | 576 +++++++
docs/unreleased/troubleshooting/overview.html | 1238 --------------
.../troubleshooting/system-metadata-tables.html | 474 ++++++
docs/unreleased/troubleshooting/tools.html | 489 ++++++
feed.xml | 4 +-
31 files changed, 4837 insertions(+), 2910 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/administration/configuration-management.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/administration/configuration-management.html b/docs/unreleased/administration/configuration-management.html
index 0429701..a4e3b9a 100644
--- a/docs/unreleased/administration/configuration-management.html
+++ b/docs/unreleased/administration/configuration-management.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/administration/configuration-properties.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/administration/configuration-properties.html b/docs/unreleased/administration/configuration-properties.html
index 92aa6a2..d39e643 100644
--- a/docs/unreleased/administration/configuration-properties.html
+++ b/docs/unreleased/administration/configuration-properties.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/administration/fate.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/administration/fate.html b/docs/unreleased/administration/fate.html
index 215063b..44fb258 100644
--- a/docs/unreleased/administration/fate.html
+++ b/docs/unreleased/administration/fate.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/administration/installation.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/administration/installation.html b/docs/unreleased/administration/installation.html
new file mode 100644
index 0000000..5d3d55d
--- /dev/null
+++ b/docs/unreleased/administration/installation.html
@@ -0,0 +1,1128 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<meta charset="utf-8">
+<meta http-equiv="X-UA-Compatible" content="IE=edge">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<link href="https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/paper/bootstrap.min.css" rel="stylesheet" integrity="sha384-awusxf8AUojygHf2+joICySzB780jVvQaVCAt1clU3QsyAitLGul28Qxb2r1e5g+" crossorigin="anonymous">
+<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
+<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.css">
+<link href="/css/accumulo.css" rel="stylesheet" type="text/css">
+
+<title>Accumulo Documentation - Installation</title>
+
+<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
+<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
+<script type="text/javascript" src="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.js"></script>
+<script>
+ // show location of canonical site if not currently on the canonical site
+ $(function() {
+ var host = window.location.host;
+ if (typeof host !== 'undefined' && host !== 'accumulo.apache.org') {
+ $('#non-canonical').show();
+ }
+ });
+
+ $(function() {
+ // decorate section headers with anchors
+ return $("h2, h3, h4, h5, h6").each(function(i, el) {
+ var $el, icon, id;
+ $el = $(el);
+ id = $el.attr('id');
+ icon = '<i class="fa fa-link"></i>';
+ if (id) {
+ return $el.append($("<a />").addClass("header-link").attr("href", "#" + id).html(icon));
+ }
+ });
+ });
+
+ // configure Google Analytics
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ if (ga.hasOwnProperty('loaded') && ga.loaded === true) {
+ ga('create', 'UA-50934829-1', 'apache.org');
+ ga('send', 'pageview');
+ }
+</script>
+
+</head>
+<body style="padding-top: 100px">
+
+ <nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="/"><img id="nav-logo" alt="Apache Accumulo" class="img-responsive" src="/images/accumulo-logo.png" width="200"
+ /></a>
+ </div>
+ <div class="collapse navbar-collapse" id="navbar-items">
+ <ul class="nav navbar-nav">
+ <li class="nav-link"><a href="/downloads">Download</a></li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Releases<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/release/accumulo-1.8.1/">1.8.1 (Latest)</a></li>
+ <li><a href="/release/accumulo-1.7.3/">1.7.3</a></li>
+ <li><a href="/release/accumulo-1.6.6/">1.6.6</a></li>
+ <li><a href="/release/">Archive</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/1.8/accumulo_user_manual.html">User Manual (1.8)</a></li>
+ <li><a href="/1.8/apidocs">Javadocs (1.8)</a></li>
+ <li><a href="/1.8/examples">Examples (1.8)</a></li>
+ <li><a href="/features">Features</a></li>
+ <li><a href="/glossary">Glossary</a></li>
+ <li><a href="/external-docs">External Docs</a></li>
+ <li><a href="/docs-archive/">Archive</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/get_involved">Get Involved</a></li>
+ <li><a href="/mailing_list">Mailing Lists</a></li>
+ <li><a href="/people">People</a></li>
+ <li><a href="/related-projects">Related Projects</a></li>
+ <li><a href="/contributor/">Contributor Guide</a></li>
+ </ul>
+ </li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://www.apache.org">Apache Homepage <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/licenses/LICENSE-2.0">License <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/sponsorship">Sponsorship <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/security">Security <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/thanks">Thanks <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct <i class="fa fa-external-link"></i></a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12">
+
+ <div id="non-canonical" style="display: none; background-color: #F0E68C; padding-left: 1em;">
+ Visit the official site at: <a href="https://accumulo.apache.org">https://accumulo.apache.org</a>
+ </div>
+ <div id="content">
+
+ <div class="alert alert-danger" role="alert">This documentation is for an unreleased version of Apache Accumulo that is currently under development! Check out the <a href="/1.8/accumulo_user_manual.html">documentation for the latest release</a>.</div>
+
+<div class="row">
+ <div class="col-md-3">
+ <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true">
+ <div class="panel panel-default">
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsegetting-started" aria-expanded="false" aria-controls="collapsegetting-started">
+ Getting started
+ </a>
+ </h4>
+ </div>
+ <div id="collapsegetting-started" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/design">Accumulo Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/clients">Accumulo Clients</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/shell">Accumulo Shell</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_design">Table Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_configuration">Table Configuration</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsedevelopment" aria-expanded="false" aria-controls="collapsedevelopment">
+ Development
+ </a>
+ </h4>
+ </div>
+ <div id="collapsedevelopment" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/sampling">Sampling</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/summaries">Summary Statistics</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/security">Security</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapseadministration" aria-expanded="true" aria-controls="collapseadministration">
+ Administration
+ </a>
+ </h4>
+ </div>
+ <div id="collapseadministration" class="panel-collapse collapse in" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/multivolume">Multi-Volume Installations</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsetroubleshooting" aria-expanded="false" aria-controls="collapsetroubleshooting">
+ Troubleshooting
+ </a>
+ </h4>
+ </div>
+ <div id="collapsetroubleshooting" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
+
+ </div>
+ </div>
+
+
+
+ </div>
+ </div>
+ </div>
+ <div class="col-md-9">
+
+ <p><a href="/docs/unreleased/">Accumulo unreleased docs</a> >> Administration >> Installation</p>
+
+
+ <h1>Installation</h1>
+
+ <h2 id="hardware">Hardware</h2>
+
+<p>Because we are running essentially two or three systems simultaneously layered
+across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to
+consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have
+at least one core and 2 - 4 GB each.</p>
+
+<p>One core running HDFS can typically keep 2 to 4 disks busy, so each machine may
+typically have as little as 2 x 300GB disks and as much as 4 x 1TB or 2TB disks.</p>
+
+<p>It is possible to do with less than this, such as with 1u servers with 2 cores and 4GB
+each, but in this case it is recommended to only run up to two processes per
+machine – i.e. DataNode and TabletServer or DataNode and MapReduce worker but
+not all three. The constraint here is having enough available heap space for all the
+processes on a machine.</p>
+
+<h2 id="network">Network</h2>
+
+<p>Accumulo communicates via remote procedure calls over TCP/IP for both passing
+data and control messages. In addition, Accumulo uses HDFS clients to
+communicate with HDFS. To achieve good ingest and query performance, sufficient
+network bandwidth must be available between any two machines.</p>
+
+<p>In addition to needing access to ports associated with HDFS and ZooKeeper, Accumulo will
+use the following default ports. Please make sure that they are open, or change
+their value in accumulo-site.xml.</p>
+
+<table>
+ <thead>
+ <tr>
+ <th>Port</th>
+ <th>Description</th>
+ <th>Property Name</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>4445</td>
+ <td>Shutdown Port (Accumulo MiniCluster)</td>
+ <td>n/a</td>
+ </tr>
+ <tr>
+ <td>4560</td>
+ <td>Accumulo monitor (for centralized log display)</td>
+ <td>monitor.port.log4j</td>
+ </tr>
+ <tr>
+ <td>9995</td>
+ <td>Accumulo HTTP monitor</td>
+ <td>monitor.port.client</td>
+ </tr>
+ <tr>
+ <td>9997</td>
+ <td>Tablet Server</td>
+ <td>tserver.port.client</td>
+ </tr>
+ <tr>
+ <td>9998</td>
+ <td>Accumulo GC</td>
+ <td>gc.port.client</td>
+ </tr>
+ <tr>
+ <td>9999</td>
+ <td>Master Server</td>
+ <td>master.port.client</td>
+ </tr>
+ <tr>
+ <td>12234</td>
+ <td>Accumulo Tracer</td>
+ <td>trace.port.client</td>
+ </tr>
+ <tr>
+ <td>42424</td>
+ <td>Accumulo Proxy Server</td>
+ <td>n/a</td>
+ </tr>
+ <tr>
+ <td>10001</td>
+ <td>Master Replication service</td>
+ <td>master.replication.coordinator.port</td>
+ </tr>
+ <tr>
+ <td>10002</td>
+ <td>TabletServer Replication service</td>
+ <td>replication.receipt.service.port</td>
+ </tr>
+ </tbody>
+</table>
+
+<p>In addition, the user can provide <code class="highlighter-rouge">0</code> and an ephemeral port will be chosen instead. This
+ephemeral port is likely to be unique and not already bound. Thus, configuring ports to
+use <code class="highlighter-rouge">0</code> instead of an explicit value, should, in most cases, work around any issues of
+running multiple distinct Accumulo instances (or any other process which tries to use the
+same default ports) on the same hardware. Finally, the *.port.client properties will work
+with the port range syntax (M-N) allowing the user to specify a range of ports for the
+service to attempt to bind. The ports in the range will be tried in a 1-up manner starting
+at the low end of the range to, and including, the high end of the range.</p>
+
+<h2 id="download-tarball">Download Tarball</h2>
+
+<p>Download a binary distribution of Accumulo and install it to a directory on a disk with
+sufficient space:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>cd <install directory>
+tar xzf accumulo-X.Y.Z-bin.tar.gz # Replace 'X.Y.Z' with your Accumulo version
+cd accumulo-X.Y.Z
+</code></pre>
+</div>
+
+<p>Repeat this step on each machine in your cluster. Typically, the same <code class="highlighter-rouge"><install directory></code>
+is chosen for all machines in the cluster.</p>
+
+<p>There are four scripts in the <code class="highlighter-rouge">bin/</code> directory that are used to manage Accumulo:</p>
+
+<ol>
+ <li><code class="highlighter-rouge">accumulo</code> - Runs Accumulo command-line tools and starts Accumulo processes</li>
+ <li><code class="highlighter-rouge">accumulo-service</code> - Runs Accumulo processes as services</li>
+ <li><code class="highlighter-rouge">accumulo-cluster</code> - Manages Accumulo cluster on a single node or several nodes</li>
+ <li><code class="highlighter-rouge">accumulo-util</code> - Accumulo utilities for creating configuration, native libraries, etc.</li>
+</ol>
+
+<p>These scripts will be used in the remaining instructions to configure and run Accumulo.</p>
+
+<h2 id="dependencies">Dependencies</h2>
+
+<p>Accumulo requires HDFS and ZooKeeper to be configured and running
+before starting. Password-less SSH should be configured between at least the
+Accumulo master and TabletServer machines. It is also a good idea to run Network
+Time Protocol (NTP) within the cluster to ensure nodes’ clocks don’t get too out of
+sync, which can cause problems with automatically timestamped data.</p>
+
+<h2 id="configuration">Configuration</h2>
+
+<p>The Accumulo tarball contains a <code class="highlighter-rouge">conf/</code> directory where Accumulo looks for configuration. If you
+installed Accumulo using downstream packaging, the <code class="highlighter-rouge">conf/</code> could be something else like
+<code class="highlighter-rouge">/etc/accumulo/</code>.</p>
+
+<p>Before starting Accumulo, the configuration files <code class="highlighter-rouge">accumulo-env.sh</code> and <code class="highlighter-rouge">accumulo-site.xml</code> must
+exist in <code class="highlighter-rouge">conf/</code> and be properly configured. If you are using <code class="highlighter-rouge">accumulo-cluster</code> to launch
+a cluster, the <code class="highlighter-rouge">conf/</code> directory must also contain hosts file for Accumulo services (i.e <code class="highlighter-rouge">gc</code>,
+<code class="highlighter-rouge">masters</code>, <code class="highlighter-rouge">monitor</code>, <code class="highlighter-rouge">tservers</code>, <code class="highlighter-rouge">tracers</code>). You can either create these files manually or run
+<code class="highlighter-rouge">accumulo-cluster create-config</code>.</p>
+
+<p>Logging is configured in <code class="highlighter-rouge">accumulo-env.sh</code> to use three log4j configuration files in <code class="highlighter-rouge">conf/</code>. The
+file used depends on the Accumulo command or service being run. Logging for most Accumulo services
+(i.e Master, TabletServer, Garbage Collector) is configured by <code class="highlighter-rouge">log4j-service.properties</code> except for
+the Monitor which is configured by <code class="highlighter-rouge">log4j-monitor.properties</code>. All Accumulo commands (i.e <code class="highlighter-rouge">init</code>,
+<code class="highlighter-rouge">shell</code>, etc) are configured by <code class="highlighter-rouge">log4j.properties</code>.</p>
+
+<h3 id="configure-accumulo-envsh">Configure accumulo-env.sh</h3>
+
+<p>Accumulo needs to know where to find the software it depends on. Edit accumulo-env.sh
+and specify the following:</p>
+
+<ol>
+ <li>Enter the location of Hadoop for <code class="highlighter-rouge">$HADOOP_PREFIX</code></li>
+ <li>Enter the location of ZooKeeper for <code class="highlighter-rouge">$ZOOKEEPER_HOME</code></li>
+ <li>Optionally, choose a different location for Accumulo logs using <code class="highlighter-rouge">$ACCUMULO_LOG_DIR</code></li>
+</ol>
+
+<p>Accumulo uses <code class="highlighter-rouge">HADOOP_PREFIX</code> and <code class="highlighter-rouge">ZOOKEEPER_HOME</code> to locate Hadoop and Zookeeper jars
+and add them the <code class="highlighter-rouge">CLASSPATH</code> variable. If you are running a vendor-specific release of Hadoop
+or Zookeeper, you may need to change how your <code class="highlighter-rouge">CLASSPATH</code> is built in <code class="highlighter-rouge">accumulo-env.sh</code>. If
+Accumulo has problems later on finding jars, run <code class="highlighter-rouge">accumulo classpath -d</code> to debug and print
+Accumulo’s classpath.</p>
+
+<p>You may want to change the default memory settings for Accumulo’s TabletServer which are
+by set in the <code class="highlighter-rouge">JAVA_OPTS</code> settings for ‘tservers’ in <code class="highlighter-rouge">accumulo-env.sh</code>. Note the
+syntax is that of the Java JVM command line options. This value should be less than the
+physical memory of the machines running TabletServers.</p>
+
+<p>There are similar options for the master’s memory usage and the garbage collector
+process. Reduce these if they exceed the physical RAM of your hardware and
+increase them, within the bounds of the physical RAM, if a process fails because of
+insufficient memory.</p>
+
+<p>Note that you will be specifying the Java heap space in accumulo-env.sh. You should
+make sure that the total heap space used for the Accumulo tserver and the Hadoop
+DataNode and TaskTracker is less than the available memory on each worker node in
+the cluster. On large clusters, it is recommended that the Accumulo master, Hadoop
+NameNode, secondary NameNode, and Hadoop JobTracker all be run on separate
+machines to allow them to use more heap space. If you are running these on the
+same machine on a small cluster, likewise make sure their heap space settings fit
+within the available memory.</p>
+
+<h3 id="native-map">Native Map</h3>
+
+<p>The tablet server uses a data structure called a MemTable to store sorted key/value
+pairs in memory when they are first received from the client. When a minor compaction
+occurs, this data structure is written to HDFS. The MemTable will default to using
+memory in the JVM but a JNI version, called the native map, can be used to significantly
+speed up performance by utilizing the memory space of the native operating system. The
+native map also avoids the performance implications brought on by garbage collection
+in the JVM by causing it to pause much less frequently.</p>
+
+<h4 id="building">Building</h4>
+
+<p>32-bit and 64-bit Linux and Mac OS X versions of the native map can be built by executing
+<code class="highlighter-rouge">accumulo-util build-native</code>. If your system’s default compiler options are insufficient,
+you can add additional compiler options to the command line, such as options for the
+architecture. These will be passed to the Makefile in the environment variable <code class="highlighter-rouge">USERFLAGS</code>.</p>
+
+<p>Examples:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>accumulo-util build-native
+accumulo-util build-native -m32
+</code></pre>
+</div>
+
+<p>After building the native map from the source, you will find the artifact in
+<code class="highlighter-rouge">lib/native</code>. Upon starting up, the tablet server will look
+in this directory for the map library. If the file is renamed or moved from its
+target directory, the tablet server may not be able to find it. The system can
+also locate the native maps shared library by setting <code class="highlighter-rouge">LD_LIBRARY_PATH</code>
+(or <code class="highlighter-rouge">DYLD_LIBRARY_PATH</code> on Mac OS X) in <code class="highlighter-rouge">accumulo-env.sh</code>.</p>
+
+<h4 id="native-maps-configuration">Native Maps Configuration</h4>
+
+<p>As mentioned, Accumulo will use the native libraries if they are found in the expected
+location and <code class="highlighter-rouge">tserver.memory.maps.native.enabled</code> is set to <code class="highlighter-rouge">true</code> (which is the default).
+Using the native maps over JVM Maps nets a noticeable improvement in ingest rates; however,
+certain configuration variables are important to modify when increasing the size of the
+native map.</p>
+
+<p>To adjust the size of the native map, increase the value of <code class="highlighter-rouge">tserver.memory.maps.max</code>.
+By default, the maximum size of the native map is 1GB. When increasing this value, it is
+also important to adjust the values of <code class="highlighter-rouge">table.compaction.minor.logs.threshold</code> and
+<code class="highlighter-rouge">tserver.walog.max.size</code>. <code class="highlighter-rouge">table.compaction.minor.logs.threshold</code> is the maximum
+number of write-ahead log files that a tablet can reference before they will be automatically
+minor compacted. <code class="highlighter-rouge">tserver.walog.max.size</code> is the maximum size of a write-ahead log.</p>
+
+<p>The maximum size of the native maps for a server should be less than the product
+of the write-ahead log maximum size and minor compaction threshold for log files:</p>
+
+<p><code class="highlighter-rouge">$table.compaction.minor.logs.threshold * $tserver.walog.max.size >= $tserver.memory.maps.max</code></p>
+
+<p>This formula ensures that minor compactions won’t be automatically triggered before the native
+maps can be completely saturated.</p>
+
+<p>Subsequently, when increasing the size of the write-ahead logs, it can also be important
+to increase the HDFS block size that Accumulo uses when creating the files for the write-ahead log.
+This is controlled via <code class="highlighter-rouge">tserver.wal.blocksize</code>. A basic recommendation is that when
+<code class="highlighter-rouge">tserver.walog.max.size</code> is larger than 2GB in size, set <code class="highlighter-rouge">tserver.wal.blocksize</code> to 2GB.
+Increasing the block size to a value larger than 2GB can result in decreased write
+performance to the write-ahead log file which will slow ingest.</p>
+
+<h3 id="cluster-specification">Cluster Specification</h3>
+
+<p>If you are using <code class="highlighter-rouge">accumulo-cluster</code> to start a cluster, configure the following on the
+machine that will serve as the Accumulo master:</p>
+
+<ol>
+ <li>Write the IP address or domain name of the Accumulo Master to the <code class="highlighter-rouge">conf/masters</code> file.</li>
+ <li>Write the IP addresses or domain name of the machines that will be TabletServers in <code class="highlighter-rouge">conf/tservers</code>, one per line.</li>
+</ol>
+
+<p>Note that if using domain names rather than IP addresses, DNS must be configured
+properly for all machines participating in the cluster. DNS can be a confusing source
+of errors.</p>
+
+<h3 id="configure-accumulo-sitexml">Configure accumulo-site.xml</h3>
+
+<p>Specify appropriate values for the following settings in <code class="highlighter-rouge">accumulo-site.xml</code>:</p>
+
+<div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="nt"><property></span>
+ <span class="nt"><name></span>instance.zookeeper.host<span class="nt"></name></span>
+ <span class="nt"><value></span>zooserver-one:2181,zooserver-two:2181<span class="nt"></value></span>
+ <span class="nt"><description></span>list of zookeeper servers<span class="nt"></description></span>
+<span class="nt"></property></span>
+</code></pre>
+</div>
+
+<p>This enables Accumulo to find ZooKeeper. Accumulo uses ZooKeeper to coordinate
+settings between processes and helps finalize TabletServer failure.</p>
+
+<div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="nt"><property></span>
+ <span class="nt"><name></span>instance.secret<span class="nt"></name></span>
+ <span class="nt"><value></span>DEFAULT<span class="nt"></value></span>
+<span class="nt"></property></span>
+</code></pre>
+</div>
+
+<p>The instance needs a secret to enable secure communication between servers. Configure your
+secret and make sure that the <code class="highlighter-rouge">accumulo-site.xml</code> file is not readable to other users.
+For alternatives to storing the <code class="highlighter-rouge">instance.secret</code> in plaintext, please read the
+<code class="highlighter-rouge">Sensitive Configuration Values</code> section.</p>
+
+<p>Some settings can be modified via the Accumulo shell and take effect immediately, but
+some settings require a process restart to take effect. See the <a href="/docs/unreleased/administration/configuration-management">configuration management</a>
+documentation for details.</p>
+
+<h3 id="hostnames-in-configuration-files">Hostnames in configuration files</h3>
+
+<p>Accumulo has a number of configuration files which can contain references to other hosts in your
+network. All of the “host” configuration files for Accumulo (<code class="highlighter-rouge">gc</code>, <code class="highlighter-rouge">masters</code>, <code class="highlighter-rouge">tservers</code>, <code class="highlighter-rouge">monitor</code>,
+<code class="highlighter-rouge">tracers</code>) as well as <code class="highlighter-rouge">instance.volumes</code> in accumulo-site.xml must contain some host reference.</p>
+
+<p>While IP address, short hostnames, or fully qualified domain names (FQDN) are all technically valid, it
+is good practice to always use FQDNs for both Accumulo and other processes in your Hadoop cluster.
+Failing to consistently use FQDNs can have unexpected consequences in how Accumulo uses the FileSystem.</p>
+
+<p>A common way for this problem can be observed is via applications that use Bulk Ingest. The Accumulo
+Master coordinates moving the input files to Bulk Ingest to an Accumulo-managed directory. However,
+Accumulo cannot safely move files across different Hadoop FileSystems. This is problematic because
+Accumulo also cannot make reliable assertions across what is the same FileSystem which is specified
+with different names. Naively, while 127.0.0.1:8020 might be a valid identifier for an HDFS instance,
+Accumulo identifies <code class="highlighter-rouge">localhost:8020</code> as a different HDFS instance than <code class="highlighter-rouge">127.0.0.1:8020</code>.</p>
+
+<h3 id="deploy-configuration">Deploy Configuration</h3>
+
+<p>Copy accumulo-env.sh and accumulo-site.xml from the <code class="highlighter-rouge">conf/</code> directory on the master to all Accumulo
+tablet servers. The “host” configuration files files <code class="highlighter-rouge">accumulo-cluster</code> only need to be on servers
+where that command is run.</p>
+
+<h3 id="sensitive-configuration-values">Sensitive Configuration Values</h3>
+
+<p>Accumulo has a number of properties that can be specified via the accumulo-site.xml
+file which are sensitive in nature, instance.secret and trace.token.property.password
+are two common examples. Both of these properties, if compromised, have the ability
+to result in data being leaked to users who should not have access to that data.</p>
+
+<p>In Hadoop-2.6.0, a new CredentialProvider class was introduced which serves as a common
+implementation to abstract away the storage and retrieval of passwords from plaintext
+storage in configuration files. Any Property marked with the <code class="highlighter-rouge">Sensitive</code> annotation
+is a candidate for use with these CredentialProviders. For version of Hadoop which lack
+these classes, the feature will just be unavailable for use.</p>
+
+<p>A comma separated list of CredentialProviders can be configured using the Accumulo Property
+<code class="highlighter-rouge">general.security.credential.provider.paths</code>. Each configured URL will be consulted
+when the Configuration object for accumulo-site.xml is accessed.</p>
+
+<h3 id="using-a-javakeystorecredentialprovider-for-storage">Using a JavaKeyStoreCredentialProvider for storage</h3>
+
+<p>One of the implementations provided in Hadoop-2.6.0 is a Java KeyStore CredentialProvider.
+Each entry in the KeyStore is the Accumulo Property key name. For example, to store the
+<code class="highlighter-rouge">instance.secret</code>, the following command can be used:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>hadoop credential create instance.secret --provider jceks://file/etc/accumulo/conf/accumulo.jceks
+</code></pre>
+</div>
+
+<p>The command will then prompt you to enter the secret to use and create a keystore in:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>/path/to/accumulo/conf/accumulo.jceks
+</code></pre>
+</div>
+
+<p>Then, accumulo-site.xml must be configured to use this KeyStore as a CredentialProvider:</p>
+
+<div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="nt"><property></span>
+ <span class="nt"><name></span>general.security.credential.provider.paths<span class="nt"></name></span>
+ <span class="nt"><value></span>jceks://file/path/to/accumulo/conf/accumulo.jceks<span class="nt"></value></span>
+<span class="nt"></property></span>
+</code></pre>
+</div>
+
+<p>This configuration will then transparently extract the <code class="highlighter-rouge">instance.secret</code> from
+the configured KeyStore and alleviates a human readable storage of the sensitive
+property.</p>
+
+<p>A KeyStore can also be stored in HDFS, which will make the KeyStore readily available to
+all Accumulo servers. If the local filesystem is used, be aware that each Accumulo server
+will expect the KeyStore in the same location.</p>
+
+<h3 id="client-configuration">Client Configuration</h3>
+
+<p>In version 1.6.0, Accumulo included a new type of configuration file known as a client
+configuration file. One problem with the traditional “site.xml” file that is prevalent
+through Hadoop is that it is a single file used by both clients and servers. This makes
+it very difficult to protect secrets that are only meant for the server processes while
+allowing the clients to connect to the servers.</p>
+
+<p>The client configuration file is a subset of the information stored in accumulo-site.xml
+meant only for consumption by clients of Accumulo. By default, Accumulo checks a number
+of locations for a client configuration by default:</p>
+
+<ul>
+ <li><code class="highlighter-rouge">/path/to/accumulo/conf/client.conf</code></li>
+ <li><code class="highlighter-rouge">/etc/accumulo/client.conf</code></li>
+ <li><code class="highlighter-rouge">/etc/accumulo/conf/client.conf</code></li>
+ <li><code class="highlighter-rouge">~/.accumulo/config</code></li>
+</ul>
+
+<p>These files are <a href="https://en.wikipedia.org/wiki/.properties">Java Properties files</a>. These files
+can currently contain information about ZooKeeper servers, RPC properties (such as SSL or SASL
+connectors), distributed tracing properties. Valid properties are defined by the <a href="https://github.com/apache/accumulo/blob/f1d0ec93d9f13ff84844b5ac81e4a7b383ced467/core/src/main/java/org/apache/accumulo/core/client/ClientConfiguration.java#L54">ClientProperty</a>
+enum contained in the client API.</p>
+
+<h4 id="custom-table-tags">Custom Table Tags</h4>
+
+<p>Accumulo has the ability for users to add custom tags to tables. This allows
+applications to set application-level metadata about a table. These tags can be
+anything from a table description, administrator notes, date created, etc.
+This is done by naming and setting a property with a prefix <code class="highlighter-rouge">table.custom.*</code>.</p>
+
+<p>Currently, table properties are stored in ZooKeeper. This means that the number
+and size of custom properties should be restricted on the order of 10’s of properties
+at most without any properties exceeding 1MB in size. ZooKeeper’s performance can be
+very sensitive to an excessive number of nodes and the sizes of the nodes. Applications
+which leverage the user of custom properties should take these warnings into
+consideration. There is no enforcement of these warnings via the API.</p>
+
+<h4 id="configuring-the-classloader">Configuring the ClassLoader</h4>
+
+<p>Accumulo builds its Java classpath in <code class="highlighter-rouge">accumulo-env.sh</code>. After an Accumulo application has started, it will load classes from the locations
+specified in the deprecated <code class="highlighter-rouge">general.classpaths</code> property. Additionally, Accumulo will load classes from the locations specified in the
+<code class="highlighter-rouge">general.dynamic.classpaths</code> property and will monitor and reload them if they change. The reloading feature is useful during the development
+and testing of iterators as new or modified iterator classes can be deployed to Accumulo without having to restart the database.
+/
+Accumulo also has an alternate configuration for the classloader which will allow it to load classes from remote locations. This mechanism
+uses Apache Commons VFS which enables locations such as http and hdfs to be used. This alternate configuration also uses the
+<code class="highlighter-rouge">general.classpaths</code> property in the same manner described above. It differs in that you need to configure the
+<code class="highlighter-rouge">general.vfs.classpaths</code> property instead of the <code class="highlighter-rouge">general.dynamic.classpath</code> property. As in the default configuration, this alternate
+configuration will also monitor the vfs locations for changes and reload if necessary.</p>
+
+<p>The Accumulo classpath can be viewed in human readable format by running <code class="highlighter-rouge">accumulo classpath -d</code>.</p>
+
+<h5 id="classloader-contexts">ClassLoader Contexts</h5>
+
+<p>With the addition of the VFS based classloader, we introduced the notion of classloader contexts. A context is identified
+by a name and references a set of locations from which to load classes and can be specified in the accumulo-site.xml file or added
+using the <code class="highlighter-rouge">config</code> command in the shell. Below is an example for specify the app1 context in the accumulo-site.xml file:</p>
+
+<div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="nt"><property></span>
+ <span class="nt"><name></span>general.vfs.context.classpath.app1<span class="nt"></name></span>
+ <span class="nt"><value></span>hdfs://localhost:8020/applicationA/classpath/.*.jar,file:///opt/applicationA/lib/.*.jar<span class="nt"></value></span>
+ <span class="nt"><description></span>Application A classpath, loads jars from HDFS and local file system<span class="nt"></description></span>
+<span class="nt"></property></span>
+</code></pre>
+</div>
+
+<p>The default behavior follows the Java ClassLoader contract in that classes, if they exists, are loaded from the parent classloader first.
+You can override this behavior by delegating to the parent classloader after looking in this classloader first. An example of this
+configuration is:</p>
+
+<div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="nt"><property></span>
+ <span class="nt"><name></span>general.vfs.context.classpath.app1.delegation=post<span class="nt"></name></span>
+ <span class="nt"><value></span>hdfs://localhost:8020/applicationA/classpath/.*.jar,file:///opt/applicationA/lib/.*.jar<span class="nt"></value></span>
+ <span class="nt"><description></span>Application A classpath, loads jars from HDFS and local file system<span class="nt"></description></span>
+<span class="nt"></property></span>
+</code></pre>
+</div>
+
+<p>To use contexts in your application you can set the <code class="highlighter-rouge">table.classpath.context</code> on your tables or use the <code class="highlighter-rouge">setClassLoaderContext()</code> method on Scanner
+and BatchScanner passing in the name of the context, app1 in the example above. Setting the property on the table allows your minc, majc, and scan
+iterators to load classes from the locations defined by the context. Passing the context name to the scanners allows you to override the table setting
+to load only scan time iterators from a different location.</p>
+
+<h2 id="initialization">Initialization</h2>
+
+<p>Accumulo must be initialized to create the structures it uses internally to locate
+data across the cluster. HDFS is required to be configured and running before
+Accumulo can be initialized.</p>
+
+<p>Once HDFS is started, initialization can be performed by executing
+<code class="highlighter-rouge">accumulo init</code> . This script will prompt for a name
+for this instance of Accumulo. The instance name is used to identify a set of tables
+and instance-specific settings. The script will then write some information into
+HDFS so Accumulo can start properly.</p>
+
+<p>The initialization script will prompt you to set a root password. Once Accumulo is
+initialized it can be started.</p>
+
+<h2 id="running">Running</h2>
+
+<h3 id="starting-accumulo">Starting Accumulo</h3>
+
+<p>Make sure Hadoop is configured on all of the machines in the cluster, including
+access to a shared HDFS instance. Make sure HDFS and ZooKeeper are running.
+Make sure ZooKeeper is configured and running on at least one machine in the
+cluster.
+Start Accumulo using <code class="highlighter-rouge">accumulo-cluster start</code>.</p>
+
+<p>To verify that Accumulo is running, check the <a href="/docs/unreleased/administration/monitoring-metrics#monitor">Accumulo monitor</a>.
+In addition, the Shell can provide some information about the status of tables via reading the metadata tables.</p>
+
+<h3 id="stopping-accumulo">Stopping Accumulo</h3>
+
+<p>To shutdown cleanly, run <code class="highlighter-rouge">accumulo-cluster stop</code> and the master will orchestrate the
+shutdown of all the tablet servers. Shutdown waits for all minor compactions to finish, so it may
+take some time for particular configurations.</p>
+
+<h3 id="adding-a-tablet-server">Adding a Tablet Server</h3>
+
+<p>Update your <code class="highlighter-rouge">conf/tservers</code> file to account for the addition.</p>
+
+<p>Next, ssh to each of the hosts you want to add and run:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>accumulo-service tserver start
+</code></pre>
+</div>
+
+<p>Make sure the host in question has the new configuration, or else the tablet
+server won’t start; at a minimum this needs to be on the host(s) being added,
+but in practice it’s good to ensure consistent configuration across all nodes.</p>
+
+<h3 id="decomissioning-a-tablet-server">Decomissioning a Tablet Server</h3>
+
+<p>If you need to take a node out of operation, you can trigger a graceful shutdown of a tablet
+server. Accumulo will automatically rebalance the tablets across the available tablet servers.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>accumulo admin stop <host(s)> {<host> ...}
+</code></pre>
+</div>
+
+<p>Alternatively, you can ssh to each of the hosts you want to remove and run:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>accumulo-service tserver stop
+</code></pre>
+</div>
+
+<p>Be sure to update your <code class="highlighter-rouge">conf/tservers</code> file to
+account for the removal of these hosts. Bear in mind that the monitor will not re-read the
+tservers file automatically, so it will report the decommissioned servers as down; it’s
+recommended that you restart the monitor so that the node list is up to date.</p>
+
+<p>The steps described to decommission a node can also be used (without removal of the host
+from the <code class="highlighter-rouge">conf/tservers</code> file) to gracefully stop a node. This will
+ensure that the tabletserver is cleanly stopped and recovery will not need to be performed
+when the tablets are re-hosted.</p>
+
+<h3 id="restarting-process-on-a-node">Restarting process on a node</h3>
+
+<p>Occasionally, it might be necessary to restart the processes on a specific node. In addition
+to the <code class="highlighter-rouge">accumulo-cluster</code> script, Accumulo has a <code class="highlighter-rouge">accumulo-service</code> script that
+can be use to start/stop processes on a node.</p>
+
+<h4 id="a-note-on-rolling-restarts">A note on rolling restarts</h4>
+
+<p>For sufficiently large Accumulo clusters, restarting multiple TabletServers within a short window can place significant
+load on the Master server. If slightly lower availability is acceptable, this load can be reduced by globally setting
+<code class="highlighter-rouge">table.suspend.duration</code> to a positive value.</p>
+
+<p>With <code class="highlighter-rouge">table.suspend.duration</code> set to, say, <code class="highlighter-rouge">5m</code>, Accumulo will wait
+for 5 minutes for any dead TabletServer to return before reassigning that TabletServer’s responsibilities to other TabletServers.
+If the TabletServer returns to the cluster before the specified timeout has elapsed, Accumulo will assign the TabletServer
+its original responsibilities.</p>
+
+<p>It is important not to choose too large a value for <code class="highlighter-rouge">table.suspend.duration</code>, as during this time, all scans against the
+data that TabletServer had hosted will block (or time out).</p>
+
+<h3 id="running-multiple-tabletservers-on-a-single-node">Running multiple TabletServers on a single node</h3>
+
+<p>With very powerful nodes, it may be beneficial to run more than one TabletServer on a given
+node. This decision should be made carefully and with much deliberation as Accumulo is designed
+to be able to scale to using 10’s of GB of RAM and 10’s of CPU cores.</p>
+
+<p>Accumulo TabletServers bind certain ports on the host to accommodate remote procedure calls to/from
+other nodes. Running more than one TabletServer on a host requires that you set the environment variable
+<code class="highlighter-rouge">ACCUMULO_SERVICE_INSTANCE</code> to an instance number (i.e 1, 2) for each instance that is started. Also, set
+these properties in <code class="highlighter-rouge">accumulo-site.xml</code>:</p>
+
+<div class="language-xml highlighter-rouge"><pre class="highlight"><code> <span class="nt"><property></span>
+ <span class="nt"><name></span>tserver.port.search<span class="nt"></name></span>
+ <span class="nt"><value></span>true<span class="nt"></value></span>
+ <span class="nt"></property></span>
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>replication.receipt.service.port<span class="nt"></name></span>
+ <span class="nt"><value></span>0<span class="nt"></value></span>
+ <span class="nt"></property></span>
+</code></pre>
+</div>
+
+<h2 id="logging">Logging</h2>
+
+<p>Accumulo processes each write to a set of log files. By default, these logs are found at directory
+set by <code class="highlighter-rouge">ACCUMULO_LOG_DIR</code> in <code class="highlighter-rouge">accumulo-env.sh</code>.</p>
+
+<h2 id="recovery">Recovery</h2>
+
+<p>In the event of TabletServer failure or error on shutting Accumulo down, some
+mutations may not have been minor compacted to HDFS properly. In this case,
+Accumulo will automatically reapply such mutations from the write-ahead log
+either when the tablets from the failed server are reassigned by the Master (in the
+case of a single TabletServer failure) or the next time Accumulo starts (in the event of
+failure during shutdown).</p>
+
+<p>Recovery is performed by asking a tablet server to sort the logs so that tablets can easily find their missing
+updates. The sort status of each file is displayed on
+Accumulo monitor status page. Once the recovery is complete any
+tablets involved should return to an <code class="highlighter-rouge">online</code> state. Until then those tablets will be
+unavailable to clients.</p>
+
+<p>The Accumulo client library is configured to retry failed mutations and in many
+cases clients will be able to continue processing after the recovery process without
+throwing an exception.</p>
+
+<h2 id="migrating-accumulo-from-non-ha-namenode-to-ha-namenode">Migrating Accumulo from non-HA Namenode to HA Namenode</h2>
+
+<p>The following steps will allow a non-HA instance to be migrated to an HA instance. Consider an HDFS URL
+<code class="highlighter-rouge">hdfs://namenode.example.com:8020</code> which is going to be moved to <code class="highlighter-rouge">hdfs://nameservice1</code>.</p>
+
+<p>Before moving HDFS over to the HA namenode, use <code class="highlighter-rouge">accumulo admin volumes</code> to confirm
+that the only volume displayed is the volume from the current namenode’s HDFS URL.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>Listing volumes referenced in zookeeper
+ Volume : hdfs://namenode.example.com:8020/accumulo
+
+Listing volumes referenced in accumulo.root tablets section
+ Volume : hdfs://namenode.example.com:8020/accumulo
+Listing volumes referenced in accumulo.root deletes section (volume replacement occurrs at deletion time)
+
+Listing volumes referenced in accumulo.metadata tablets section
+ Volume : hdfs://namenode.example.com:8020/accumulo
+
+Listing volumes referenced in accumulo.metadata deletes section (volume replacement occurrs at deletion time)
+</code></pre>
+</div>
+
+<p>After verifying the current volume is correct, shut down the cluster and transition HDFS to the HA nameservice.</p>
+
+<p>Edit <code class="highlighter-rouge">accumulo-site.xml</code> to notify accumulo that a volume is being replaced. First,
+add the new nameservice volume to the <code class="highlighter-rouge">instance.volumes</code> property. Next, add the
+<code class="highlighter-rouge">instance.volumes.replacements</code> property in the form of <code class="highlighter-rouge">old new</code>. It’s important to not include
+the volume that’s being replaced in <code class="highlighter-rouge">instance.volumes</code>, otherwise it’s possible accumulo could continue
+to write to the volume.</p>
+
+<div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="c"><!-- instance.dfs.uri and instance.dfs.dir should not be set--></span>
+<span class="nt"><property></span>
+ <span class="nt"><name></span>instance.volumes<span class="nt"></name></span>
+ <span class="nt"><value></span>hdfs://nameservice1/accumulo<span class="nt"></value></span>
+<span class="nt"></property></span>
+<span class="nt"><property></span>
+ <span class="nt"><name></span>instance.volumes.replacements<span class="nt"></name></span>
+ <span class="nt"><value></span>hdfs://namenode.example.com:8020/accumulo hdfs://nameservice1/accumulo<span class="nt"></value></span>
+<span class="nt"></property></span>
+</code></pre>
+</div>
+
+<p>Run <code class="highlighter-rouge">accumulo init --add-volumes</code> and start up the accumulo cluster. Verify that the
+new nameservice volume shows up with <code class="highlighter-rouge">accumulo admin volumes</code>.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>Listing volumes referenced in zookeeper
+ Volume : hdfs://namenode.example.com:8020/accumulo
+ Volume : hdfs://nameservice1/accumulo
+
+Listing volumes referenced in accumulo.root tablets section
+ Volume : hdfs://namenode.example.com:8020/accumulo
+ Volume : hdfs://nameservice1/accumulo
+Listing volumes referenced in accumulo.root deletes section (volume replacement occurrs at deletion time)
+
+Listing volumes referenced in accumulo.metadata tablets section
+ Volume : hdfs://namenode.example.com:8020/accumulo
+ Volume : hdfs://nameservice1/accumulo
+Listing volumes referenced in accumulo.metadata deletes section (volume replacement occurrs at deletion time)
+</code></pre>
+</div>
+
+<p>Some erroneous GarbageCollector messages may still be seen for a small period while data is transitioning to
+the new volumes. This is expected and can usually be ignored.</p>
+
+<h2 id="achieving-stability-in-a-vm-environment">Achieving Stability in a VM Environment</h2>
+
+<p>For testing, demonstration, and even operation uses, Accumulo is often
+installed and run in a virtual machine (VM) environment. The majority of
+long-term operational uses of Accumulo are on bare-metal cluster. However, the
+core design of Accumulo and its dependencies do not preclude running stably for
+long periods within a VM. Many of Accumulo’s operational robustness features to
+handle failures like periodic network partitioning in a large cluster carry
+over well to VM environments. This guide covers general recommendations for
+maximizing stability in a VM environment, including some of the common failure
+modes that are more common when running in VMs.</p>
+
+<h3 id="known-failure-modes-setup-and-troubleshooting">Known failure modes: Setup and Troubleshooting</h3>
+
+<p>In addition to the general failure modes of running Accumulo, VMs can introduce a
+couple of environmental challenges that can affect process stability. Clock
+drift is something that is more common in VMs, especially when VMs are
+suspended and resumed. Clock drift can cause Accumulo servers to assume that
+they have lost connectivity to the other Accumulo processes and/or lose their
+locks in Zookeeper. VM environments also frequently have constrained resources,
+such as CPU, RAM, network, and disk throughput and capacity. Accumulo generally
+deals well with constrained resources from a stability perspective (optimizing
+performance will require additional tuning, which is not covered in this
+section), however there are some limits.</p>
+
+<h4 id="physical-memory">Physical Memory</h4>
+
+<p>One of those limits has to do with the Linux out of memory killer. A common
+failure mode in VM environments (and in some bare metal installations) is when
+the Linux out of memory killer decides to kill processes in order to avoid a
+kernel panic when provisioning a memory page. This often happens in VMs due to
+the large number of processes that must run in a small memory footprint. In
+addition to the Linux core processes, a single-node Accumulo setup requires a
+Hadoop Namenode, a Hadoop Secondary Namenode a Hadoop Datanode, a Zookeeper
+server, an Accumulo Master, an Accumulo GC and an Accumulo TabletServer.
+Typical setups also include an Accumulo Monitor, an Accumulo Tracer, a Hadoop
+ResourceManager, a Hadoop NodeManager, provisioning software, and client
+applications. Between all of these processes, it is not uncommon to
+over-subscribe the available RAM in a VM. We recommend setting up VMs without
+swap enabled, so rather than performance grinding to a halt when physical
+memory is exhausted the kernel will randomly* select processes to kill in order
+to free up memory.</p>
+
+<p>Calculating the maximum possible memory usage is essential in creating a stable
+Accumulo VM setup. Safely engineering memory allocation for stability is a
+matter of then bringing the calculated maximum memory usage under the physical
+memory by a healthy margin. The margin is to account for operating system-level
+operations, such as managing process, maintaining virtual memory pages, and
+file system caching. When the java out-of-memory killer finds your process, you
+will probably only see evidence of that in /var/log/messages. Out-of-memory
+process kills do not show up in Accumulo or Hadoop logs.</p>
+
+<p>To calculate the max memory usage of all java virtual machine (JVM) processes
+add the maximum heap size (often limited by a -Xmx… argument, such as in
+accumulo-site.xml) and the off-heap memory usage. Off-heap memory usage
+includes the following:</p>
+
+<ul>
+ <li>“Permanent Space”, where the JVM stores Classes, Methods, and other code elements. This can be limited by a JVM flag such as <code class="highlighter-rouge">-XX:MaxPermSize:100m</code>, and is typically tens of megabytes.</li>
+ <li>Code generation space, where the JVM stores just-in-time compiled code. This is typically small enough to ignore</li>
+ <li>Socket buffers, where the JVM stores send and receive buffers for each socket.</li>
+ <li>Thread stacks, where the JVM allocates memory to manage each thread.</li>
+ <li>Direct memory space and JNI code, where applications can allocate memory outside of the JVM-managed space. For Accumulo, this includes the native in-memory maps that are allocated with the memory.maps.max parameter in accumulo-site.xml.</li>
+ <li>Garbage collection space, where the JVM stores information used for garbage collection.</li>
+</ul>
+
+<p>You can assume that each Hadoop and Accumulo process will use ~100-150MB for
+Off-heap memory, plus the in-memory map of the Accumulo TServer process. A
+simple calculation for physical memory requirements follows:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code> Physical memory needed
+ = (per-process off-heap memory) + (heap memory) + (other processes) + (margin)
+ = (number of java processes * 150M + native map) + (sum of -Xmx settings for java process) + (total applications memory, provisioning memory, etc.) + (1G)
+ = (11*150M +500M) + (1G +1G +1G +256M +1G +256M +512M +512M +512M +512M +512M) + (2G) + (1G)
+ = (2150M) + (7G) + (2G) + (1G)
+ = ~12GB
+</code></pre>
+</div>
+
+<p>These calculations can add up quickly with the large number of processes,
+especially in constrained VM environments. To reduce the physical memory
+requirements, it is a good idea to reduce maximum heap limits and turn off
+unnecessary processes. If you’re not using YARN in your application, you can
+turn off the ResourceManager and NodeManager. If you’re not expecting to
+re-provision the cluster frequently you can turn off or reduce provisioning
+processes such as Salt Stack minions and masters.</p>
+
+<h4 id="disk-space">Disk Space</h4>
+
+<p>Disk space is primarily used for two operations: storing data and storing logs.
+While Accumulo generally stores all of its key/value data in HDFS, Accumulo,
+Hadoop, and Zookeeper all store a significant amount of logs in a directory on
+a local file system. Care should be taken to make sure that (a) limitations to
+the amount of logs generated are in place, and (b) enough space is available to
+host the generated logs on the partitions that they are assigned. When space is
+not available to log, processes will hang. This can cause interruptions in
+availability of Accumulo, as well as cascade into failures of various
+processes.</p>
+
+<p>Hadoop, Accumulo, and Zookeeper use log4j as a logging mechanism, and each of
+them has a way of limiting the logs and directing them to a particular
+directory. Logs are generated independently for each process, so when
+considering the total space you need to add up the maximum logs generated by
+each process. Typically, a rolling log setup in which each process can generate
+something like 10 100MB files is instituted, resulting in a maximum file system
+usage of 1GB per process. Default setups for Hadoop and Zookeeper are often
+unbounded, so it is important to set these limits in the logging configuration
+files for each subsystem. Consult the user manual for each system for
+instructions on how to limit generated logs.</p>
+
+<h4 id="zookeeper-interaction">Zookeeper Interaction</h4>
+
+<p>Accumulo is designed to scale up to thousands of nodes. At that scale,
+intermittent interruptions in network service and other rare failures of
+compute nodes become more common. To limit the impact of node failures on
+overall service availability, Accumulo uses a heartbeat monitoring system that
+leverages Zookeeper’s ephemeral locks. There are several conditions that can
+occur that cause Accumulo process to lose their Zookeeper locks, some of which
+are true interruptions to availability and some of which are false positives.
+Several of these conditions become more common in VM environments, where they
+can be exacerbated by resource constraints and clock drift.</p>
+
+<h4 id="tested-versions">Tested Versions</h4>
+
+<p>Each release of Accumulo is built with a specific version of Apache
+Hadoop, Apache ZooKeeper and Apache Thrift. We expect Accumulo to
+work with versions that are API compatible with those versions.
+However this compatibility is not guaranteed because Hadoop, ZooKeeper
+and Thift may not provide guarantees between their own versions. We
+have also found that certain versions of Accumulo and Hadoop included
+bugs that greatly affected overall stability. Thrift is particularly
+prone to compatibility changes between versions and you must use the
+same version your Accumulo is built with.</p>
+
+<p>Please check the release notes for your Accumulo version or use the
+mailing lists at https://accumulo.apache.org for more info.</p>
+
+
+ </div>
+</div>
+
+ </div>
+
+
+<footer>
+
+ <p><a href="https://www.apache.org/foundation/contributing"><img src="https://www.apache.org/images/SupportApache-small.png" alt="Support the ASF" id="asf-logo" height="100" /></a></p>
+
+ <p>Copyright © 2011-2017 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.</p>
+
+</footer>
+
+
+ </div>
+ </div>
+ </div>
+</body>
+</html>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/administration/kerberos.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/administration/kerberos.html b/docs/unreleased/administration/kerberos.html
index 6656df4..2c31e5f 100644
--- a/docs/unreleased/administration/kerberos.html
+++ b/docs/unreleased/administration/kerberos.html
@@ -240,15 +240,15 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
@@ -256,6 +256,10 @@
<div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
</div>
</div>
@@ -290,7 +294,13 @@
<div class="panel-body">
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
</div>
</div>
[08/10] accumulo-website git commit: Jekyll build from master:036341f
Posted by mw...@apache.org.
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/administration/overview.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/administration/overview.html b/docs/unreleased/administration/overview.html
deleted file mode 100644
index f985509..0000000
--- a/docs/unreleased/administration/overview.html
+++ /dev/null
@@ -1,1586 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-<meta charset="utf-8">
-<meta http-equiv="X-UA-Compatible" content="IE=edge">
-<meta name="viewport" content="width=device-width, initial-scale=1">
-<link href="https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/paper/bootstrap.min.css" rel="stylesheet" integrity="sha384-awusxf8AUojygHf2+joICySzB780jVvQaVCAt1clU3QsyAitLGul28Qxb2r1e5g+" crossorigin="anonymous">
-<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.css">
-<link href="/css/accumulo.css" rel="stylesheet" type="text/css">
-
-<title>Accumulo Documentation - Overview</title>
-
-<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
-<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
-<script type="text/javascript" src="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.js"></script>
-<script>
- // show location of canonical site if not currently on the canonical site
- $(function() {
- var host = window.location.host;
- if (typeof host !== 'undefined' && host !== 'accumulo.apache.org') {
- $('#non-canonical').show();
- }
- });
-
- $(function() {
- // decorate section headers with anchors
- return $("h2, h3, h4, h5, h6").each(function(i, el) {
- var $el, icon, id;
- $el = $(el);
- id = $el.attr('id');
- icon = '<i class="fa fa-link"></i>';
- if (id) {
- return $el.append($("<a />").addClass("header-link").attr("href", "#" + id).html(icon));
- }
- });
- });
-
- // configure Google Analytics
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
-
- if (ga.hasOwnProperty('loaded') && ga.loaded === true) {
- ga('create', 'UA-50934829-1', 'apache.org');
- ga('send', 'pageview');
- }
-</script>
-
-</head>
-<body style="padding-top: 100px">
-
- <nav class="navbar navbar-default navbar-fixed-top">
- <div class="container">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- <a href="/"><img id="nav-logo" alt="Apache Accumulo" class="img-responsive" src="/images/accumulo-logo.png" width="200"
- /></a>
- </div>
- <div class="collapse navbar-collapse" id="navbar-items">
- <ul class="nav navbar-nav">
- <li class="nav-link"><a href="/downloads">Download</a></li>
- <li class="dropdown">
- <a class="dropdown-toggle" data-toggle="dropdown" href="#">Releases<span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li><a href="/release/accumulo-1.8.1/">1.8.1 (Latest)</a></li>
- <li><a href="/release/accumulo-1.7.3/">1.7.3</a></li>
- <li><a href="/release/accumulo-1.6.6/">1.6.6</a></li>
- <li><a href="/release/">Archive</a></li>
- </ul>
- </li>
- <li class="dropdown">
- <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li><a href="/1.8/accumulo_user_manual.html">User Manual (1.8)</a></li>
- <li><a href="/1.8/apidocs">Javadocs (1.8)</a></li>
- <li><a href="/1.8/examples">Examples (1.8)</a></li>
- <li><a href="/features">Features</a></li>
- <li><a href="/glossary">Glossary</a></li>
- <li><a href="/external-docs">External Docs</a></li>
- <li><a href="/docs-archive/">Archive</a></li>
- </ul>
- </li>
- <li class="dropdown">
- <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li><a href="/get_involved">Get Involved</a></li>
- <li><a href="/mailing_list">Mailing Lists</a></li>
- <li><a href="/people">People</a></li>
- <li><a href="/related-projects">Related Projects</a></li>
- <li><a href="/contributor/">Contributor Guide</a></li>
- </ul>
- </li>
- </ul>
- <ul class="nav navbar-nav navbar-right">
- <li class="dropdown">
- <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation<span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li><a href="https://www.apache.org">Apache Homepage <i class="fa fa-external-link"></i></a></li>
- <li><a href="https://www.apache.org/licenses/LICENSE-2.0">License <i class="fa fa-external-link"></i></a></li>
- <li><a href="https://www.apache.org/foundation/sponsorship">Sponsorship <i class="fa fa-external-link"></i></a></li>
- <li><a href="https://www.apache.org/security">Security <i class="fa fa-external-link"></i></a></li>
- <li><a href="https://www.apache.org/foundation/thanks">Thanks <i class="fa fa-external-link"></i></a></li>
- <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct <i class="fa fa-external-link"></i></a></li>
- </ul>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
- <div class="container">
- <div class="row">
- <div class="col-md-12">
-
- <div id="non-canonical" style="display: none; background-color: #F0E68C; padding-left: 1em;">
- Visit the official site at: <a href="https://accumulo.apache.org">https://accumulo.apache.org</a>
- </div>
- <div id="content">
-
- <div class="alert alert-danger" role="alert">This documentation is for an unreleased version of Apache Accumulo that is currently under development! Check out the <a href="/1.8/accumulo_user_manual.html">documentation for the latest release</a>.</div>
-
-<div class="row">
- <div class="col-md-3">
- <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true">
- <div class="panel panel-default">
-
-
-
-
-
-
-
-
-
- <div class="panel-heading" role="tab" id="headingOne">
- <h4 class="panel-title">
- <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsegetting-started" aria-expanded="false" aria-controls="collapsegetting-started">
- Getting started
- </a>
- </h4>
- </div>
- <div id="collapsegetting-started" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
- <div class="panel-body">
-
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/design">Accumulo Design</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/clients">Accumulo Clients</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/shell">Accumulo Shell</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_design">Table Design</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_configuration">Table Configuration</a></div>
-
- </div>
- </div>
-
-
-
-
-
-
-
-
-
-
-
- <div class="panel-heading" role="tab" id="headingOne">
- <h4 class="panel-title">
- <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsedevelopment" aria-expanded="false" aria-controls="collapsedevelopment">
- Development
- </a>
- </h4>
- </div>
- <div id="collapsedevelopment" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
- <div class="panel-body">
-
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/sampling">Sampling</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/summaries">Summary Statistics</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/security">Security</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div>
-
- </div>
- </div>
-
-
-
-
-
-
-
-
-
-
-
- <div class="panel-heading" role="tab" id="headingOne">
- <h4 class="panel-title">
- <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapseadministration" aria-expanded="true" aria-controls="collapseadministration">
- Administration
- </a>
- </h4>
- </div>
- <div id="collapseadministration" class="panel-collapse collapse in" role="tabpanel" aria-labelledby="headingOne">
- <div class="panel-body">
-
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/multivolume">Multi-Volume Installations</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
-
- </div>
- </div>
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <div class="panel-heading" role="tab" id="headingOne">
- <h4 class="panel-title">
- <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsetroubleshooting" aria-expanded="false" aria-controls="collapsetroubleshooting">
- Troubleshooting
- </a>
- </h4>
- </div>
- <div id="collapsetroubleshooting" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
- <div class="panel-body">
-
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
-
- </div>
- </div>
-
-
-
- </div>
- </div>
- </div>
- <div class="col-md-9">
-
- <p><a href="/docs/unreleased/">Accumulo unreleased docs</a> >> Administration >> Overview</p>
-
-
- <h1>Overview</h1>
-
- <h2 id="hardware">Hardware</h2>
-
-<p>Because we are running essentially two or three systems simultaneously layered
-across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to
-consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have
-at least one core and 2 - 4 GB each.</p>
-
-<p>One core running HDFS can typically keep 2 to 4 disks busy, so each machine may
-typically have as little as 2 x 300GB disks and as much as 4 x 1TB or 2TB disks.</p>
-
-<p>It is possible to do with less than this, such as with 1u servers with 2 cores and 4GB
-each, but in this case it is recommended to only run up to two processes per
-machine – i.e. DataNode and TabletServer or DataNode and MapReduce worker but
-not all three. The constraint here is having enough available heap space for all the
-processes on a machine.</p>
-
-<h2 id="network">Network</h2>
-
-<p>Accumulo communicates via remote procedure calls over TCP/IP for both passing
-data and control messages. In addition, Accumulo uses HDFS clients to
-communicate with HDFS. To achieve good ingest and query performance, sufficient
-network bandwidth must be available between any two machines.</p>
-
-<p>In addition to needing access to ports associated with HDFS and ZooKeeper, Accumulo will
-use the following default ports. Please make sure that they are open, or change
-their value in accumulo-site.xml.</p>
-
-<table>
- <thead>
- <tr>
- <th>Port</th>
- <th>Description</th>
- <th>Property Name</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>4445</td>
- <td>Shutdown Port (Accumulo MiniCluster)</td>
- <td>n/a</td>
- </tr>
- <tr>
- <td>4560</td>
- <td>Accumulo monitor (for centralized log display)</td>
- <td>monitor.port.log4j</td>
- </tr>
- <tr>
- <td>9995</td>
- <td>Accumulo HTTP monitor</td>
- <td>monitor.port.client</td>
- </tr>
- <tr>
- <td>9997</td>
- <td>Tablet Server</td>
- <td>tserver.port.client</td>
- </tr>
- <tr>
- <td>9998</td>
- <td>Accumulo GC</td>
- <td>gc.port.client</td>
- </tr>
- <tr>
- <td>9999</td>
- <td>Master Server</td>
- <td>master.port.client</td>
- </tr>
- <tr>
- <td>12234</td>
- <td>Accumulo Tracer</td>
- <td>trace.port.client</td>
- </tr>
- <tr>
- <td>42424</td>
- <td>Accumulo Proxy Server</td>
- <td>n/a</td>
- </tr>
- <tr>
- <td>10001</td>
- <td>Master Replication service</td>
- <td>master.replication.coordinator.port</td>
- </tr>
- <tr>
- <td>10002</td>
- <td>TabletServer Replication service</td>
- <td>replication.receipt.service.port</td>
- </tr>
- </tbody>
-</table>
-
-<p>In addition, the user can provide <code class="highlighter-rouge">0</code> and an ephemeral port will be chosen instead. This
-ephemeral port is likely to be unique and not already bound. Thus, configuring ports to
-use <code class="highlighter-rouge">0</code> instead of an explicit value, should, in most cases, work around any issues of
-running multiple distinct Accumulo instances (or any other process which tries to use the
-same default ports) on the same hardware. Finally, the *.port.client properties will work
-with the port range syntax (M-N) allowing the user to specify a range of ports for the
-service to attempt to bind. The ports in the range will be tried in a 1-up manner starting
-at the low end of the range to, and including, the high end of the range.</p>
-
-<h2 id="installation">Installation</h2>
-
-<p>Download a binary distribution of Accumulo and install it to a directory on a disk with
-sufficient space:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>cd <install directory>
-tar xzf accumulo-X.Y.Z-bin.tar.gz # Replace 'X.Y.Z' with your Accumulo version
-cd accumulo-X.Y.Z
-</code></pre>
-</div>
-
-<p>Repeat this step on each machine in your cluster. Typically, the same <code class="highlighter-rouge"><install directory></code>
-is chosen for all machines in the cluster.</p>
-
-<p>There are four scripts in the <code class="highlighter-rouge">bin/</code> directory that are used to manage Accumulo:</p>
-
-<ol>
- <li><code class="highlighter-rouge">accumulo</code> - Runs Accumulo command-line tools and starts Accumulo processes</li>
- <li><code class="highlighter-rouge">accumulo-service</code> - Runs Accumulo processes as services</li>
- <li><code class="highlighter-rouge">accumulo-cluster</code> - Manages Accumulo cluster on a single node or several nodes</li>
- <li><code class="highlighter-rouge">accumulo-util</code> - Accumulo utilities for creating configuration, native libraries, etc.</li>
-</ol>
-
-<p>These scripts will be used in the remaining instructions to configure and run Accumulo.</p>
-
-<h2 id="dependencies">Dependencies</h2>
-
-<p>Accumulo requires HDFS and ZooKeeper to be configured and running
-before starting. Password-less SSH should be configured between at least the
-Accumulo master and TabletServer machines. It is also a good idea to run Network
-Time Protocol (NTP) within the cluster to ensure nodes’ clocks don’t get too out of
-sync, which can cause problems with automatically timestamped data.</p>
-
-<h2 id="configuration">Configuration</h2>
-
-<p>The Accumulo tarball contains a <code class="highlighter-rouge">conf/</code> directory where Accumulo looks for configuration. If you
-installed Accumulo using downstream packaging, the <code class="highlighter-rouge">conf/</code> could be something else like
-<code class="highlighter-rouge">/etc/accumulo/</code>.</p>
-
-<p>Before starting Accumulo, the configuration files <code class="highlighter-rouge">accumulo-env.sh</code> and <code class="highlighter-rouge">accumulo-site.xml</code> must
-exist in <code class="highlighter-rouge">conf/</code> and be properly configured. If you are using <code class="highlighter-rouge">accumulo-cluster</code> to launch
-a cluster, the <code class="highlighter-rouge">conf/</code> directory must also contain hosts file for Accumulo services (i.e <code class="highlighter-rouge">gc</code>,
-<code class="highlighter-rouge">masters</code>, <code class="highlighter-rouge">monitor</code>, <code class="highlighter-rouge">tservers</code>, <code class="highlighter-rouge">tracers</code>). You can either create these files manually or run
-<code class="highlighter-rouge">accumulo-cluster create-config</code>.</p>
-
-<p>Logging is configured in <code class="highlighter-rouge">accumulo-env.sh</code> to use three log4j configuration files in <code class="highlighter-rouge">conf/</code>. The
-file used depends on the Accumulo command or service being run. Logging for most Accumulo services
-(i.e Master, TabletServer, Garbage Collector) is configured by <code class="highlighter-rouge">log4j-service.properties</code> except for
-the Monitor which is configured by <code class="highlighter-rouge">log4j-monitor.properties</code>. All Accumulo commands (i.e <code class="highlighter-rouge">init</code>,
-<code class="highlighter-rouge">shell</code>, etc) are configured by <code class="highlighter-rouge">log4j.properties</code>.</p>
-
-<h3 id="configure-accumulo-envsh">Configure accumulo-env.sh</h3>
-
-<p>Accumulo needs to know where to find the software it depends on. Edit accumulo-env.sh
-and specify the following:</p>
-
-<ol>
- <li>Enter the location of Hadoop for <code class="highlighter-rouge">$HADOOP_PREFIX</code></li>
- <li>Enter the location of ZooKeeper for <code class="highlighter-rouge">$ZOOKEEPER_HOME</code></li>
- <li>Optionally, choose a different location for Accumulo logs using <code class="highlighter-rouge">$ACCUMULO_LOG_DIR</code></li>
-</ol>
-
-<p>Accumulo uses <code class="highlighter-rouge">HADOOP_PREFIX</code> and <code class="highlighter-rouge">ZOOKEEPER_HOME</code> to locate Hadoop and Zookeeper jars
-and add them the <code class="highlighter-rouge">CLASSPATH</code> variable. If you are running a vendor-specific release of Hadoop
-or Zookeeper, you may need to change how your <code class="highlighter-rouge">CLASSPATH</code> is built in <code class="highlighter-rouge">accumulo-env.sh</code>. If
-Accumulo has problems later on finding jars, run <code class="highlighter-rouge">accumulo classpath -d</code> to debug and print
-Accumulo’s classpath.</p>
-
-<p>You may want to change the default memory settings for Accumulo’s TabletServer which are
-by set in the <code class="highlighter-rouge">JAVA_OPTS</code> settings for ‘tservers’ in <code class="highlighter-rouge">accumulo-env.sh</code>. Note the
-syntax is that of the Java JVM command line options. This value should be less than the
-physical memory of the machines running TabletServers.</p>
-
-<p>There are similar options for the master’s memory usage and the garbage collector
-process. Reduce these if they exceed the physical RAM of your hardware and
-increase them, within the bounds of the physical RAM, if a process fails because of
-insufficient memory.</p>
-
-<p>Note that you will be specifying the Java heap space in accumulo-env.sh. You should
-make sure that the total heap space used for the Accumulo tserver and the Hadoop
-DataNode and TaskTracker is less than the available memory on each worker node in
-the cluster. On large clusters, it is recommended that the Accumulo master, Hadoop
-NameNode, secondary NameNode, and Hadoop JobTracker all be run on separate
-machines to allow them to use more heap space. If you are running these on the
-same machine on a small cluster, likewise make sure their heap space settings fit
-within the available memory.</p>
-
-<h3 id="native-map">Native Map</h3>
-
-<p>The tablet server uses a data structure called a MemTable to store sorted key/value
-pairs in memory when they are first received from the client. When a minor compaction
-occurs, this data structure is written to HDFS. The MemTable will default to using
-memory in the JVM but a JNI version, called the native map, can be used to significantly
-speed up performance by utilizing the memory space of the native operating system. The
-native map also avoids the performance implications brought on by garbage collection
-in the JVM by causing it to pause much less frequently.</p>
-
-<h4 id="building">Building</h4>
-
-<p>32-bit and 64-bit Linux and Mac OS X versions of the native map can be built by executing
-<code class="highlighter-rouge">accumulo-util build-native</code>. If your system’s default compiler options are insufficient,
-you can add additional compiler options to the command line, such as options for the
-architecture. These will be passed to the Makefile in the environment variable <code class="highlighter-rouge">USERFLAGS</code>.</p>
-
-<p>Examples:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>accumulo-util build-native
-accumulo-util build-native -m32
-</code></pre>
-</div>
-
-<p>After building the native map from the source, you will find the artifact in
-<code class="highlighter-rouge">lib/native</code>. Upon starting up, the tablet server will look
-in this directory for the map library. If the file is renamed or moved from its
-target directory, the tablet server may not be able to find it. The system can
-also locate the native maps shared library by setting <code class="highlighter-rouge">LD_LIBRARY_PATH</code>
-(or <code class="highlighter-rouge">DYLD_LIBRARY_PATH</code> on Mac OS X) in <code class="highlighter-rouge">accumulo-env.sh</code>.</p>
-
-<h4 id="native-maps-configuration">Native Maps Configuration</h4>
-
-<p>As mentioned, Accumulo will use the native libraries if they are found in the expected
-location and <code class="highlighter-rouge">tserver.memory.maps.native.enabled</code> is set to <code class="highlighter-rouge">true</code> (which is the default).
-Using the native maps over JVM Maps nets a noticeable improvement in ingest rates; however,
-certain configuration variables are important to modify when increasing the size of the
-native map.</p>
-
-<p>To adjust the size of the native map, increase the value of <code class="highlighter-rouge">tserver.memory.maps.max</code>.
-By default, the maximum size of the native map is 1GB. When increasing this value, it is
-also important to adjust the values of <code class="highlighter-rouge">table.compaction.minor.logs.threshold</code> and
-<code class="highlighter-rouge">tserver.walog.max.size</code>. <code class="highlighter-rouge">table.compaction.minor.logs.threshold</code> is the maximum
-number of write-ahead log files that a tablet can reference before they will be automatically
-minor compacted. <code class="highlighter-rouge">tserver.walog.max.size</code> is the maximum size of a write-ahead log.</p>
-
-<p>The maximum size of the native maps for a server should be less than the product
-of the write-ahead log maximum size and minor compaction threshold for log files:</p>
-
-<p><code class="highlighter-rouge">$table.compaction.minor.logs.threshold * $tserver.walog.max.size >= $tserver.memory.maps.max</code></p>
-
-<p>This formula ensures that minor compactions won’t be automatically triggered before the native
-maps can be completely saturated.</p>
-
-<p>Subsequently, when increasing the size of the write-ahead logs, it can also be important
-to increase the HDFS block size that Accumulo uses when creating the files for the write-ahead log.
-This is controlled via <code class="highlighter-rouge">tserver.wal.blocksize</code>. A basic recommendation is that when
-<code class="highlighter-rouge">tserver.walog.max.size</code> is larger than 2GB in size, set <code class="highlighter-rouge">tserver.wal.blocksize</code> to 2GB.
-Increasing the block size to a value larger than 2GB can result in decreased write
-performance to the write-ahead log file which will slow ingest.</p>
-
-<h3 id="cluster-specification">Cluster Specification</h3>
-
-<p>If you are using <code class="highlighter-rouge">accumulo-cluster</code> to start a cluster, configure the following on the
-machine that will serve as the Accumulo master:</p>
-
-<ol>
- <li>Write the IP address or domain name of the Accumulo Master to the <code class="highlighter-rouge">conf/masters</code> file.</li>
- <li>Write the IP addresses or domain name of the machines that will be TabletServers in <code class="highlighter-rouge">conf/tservers</code>, one per line.</li>
-</ol>
-
-<p>Note that if using domain names rather than IP addresses, DNS must be configured
-properly for all machines participating in the cluster. DNS can be a confusing source
-of errors.</p>
-
-<h3 id="configure-accumulo-sitexml">Configure accumulo-site.xml</h3>
-
-<p>Specify appropriate values for the following settings in <code class="highlighter-rouge">accumulo-site.xml</code>:</p>
-
-<div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="nt"><property></span>
- <span class="nt"><name></span>instance.zookeeper.host<span class="nt"></name></span>
- <span class="nt"><value></span>zooserver-one:2181,zooserver-two:2181<span class="nt"></value></span>
- <span class="nt"><description></span>list of zookeeper servers<span class="nt"></description></span>
-<span class="nt"></property></span>
-</code></pre>
-</div>
-
-<p>This enables Accumulo to find ZooKeeper. Accumulo uses ZooKeeper to coordinate
-settings between processes and helps finalize TabletServer failure.</p>
-
-<div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="nt"><property></span>
- <span class="nt"><name></span>instance.secret<span class="nt"></name></span>
- <span class="nt"><value></span>DEFAULT<span class="nt"></value></span>
-<span class="nt"></property></span>
-</code></pre>
-</div>
-
-<p>The instance needs a secret to enable secure communication between servers. Configure your
-secret and make sure that the <code class="highlighter-rouge">accumulo-site.xml</code> file is not readable to other users.
-For alternatives to storing the <code class="highlighter-rouge">instance.secret</code> in plaintext, please read the
-<code class="highlighter-rouge">Sensitive Configuration Values</code> section.</p>
-
-<p>Some settings can be modified via the Accumulo shell and take effect immediately, but
-some settings require a process restart to take effect. See the <a href="/docs/unreleased/administration/configuration-management">configuration management</a>
-documentation for details.</p>
-
-<h3 id="hostnames-in-configuration-files">Hostnames in configuration files</h3>
-
-<p>Accumulo has a number of configuration files which can contain references to other hosts in your
-network. All of the “host” configuration files for Accumulo (<code class="highlighter-rouge">gc</code>, <code class="highlighter-rouge">masters</code>, <code class="highlighter-rouge">tservers</code>, <code class="highlighter-rouge">monitor</code>,
-<code class="highlighter-rouge">tracers</code>) as well as <code class="highlighter-rouge">instance.volumes</code> in accumulo-site.xml must contain some host reference.</p>
-
-<p>While IP address, short hostnames, or fully qualified domain names (FQDN) are all technically valid, it
-is good practice to always use FQDNs for both Accumulo and other processes in your Hadoop cluster.
-Failing to consistently use FQDNs can have unexpected consequences in how Accumulo uses the FileSystem.</p>
-
-<p>A common way for this problem can be observed is via applications that use Bulk Ingest. The Accumulo
-Master coordinates moving the input files to Bulk Ingest to an Accumulo-managed directory. However,
-Accumulo cannot safely move files across different Hadoop FileSystems. This is problematic because
-Accumulo also cannot make reliable assertions across what is the same FileSystem which is specified
-with different names. Naively, while 127.0.0.1:8020 might be a valid identifier for an HDFS instance,
-Accumulo identifies <code class="highlighter-rouge">localhost:8020</code> as a different HDFS instance than <code class="highlighter-rouge">127.0.0.1:8020</code>.</p>
-
-<h3 id="deploy-configuration">Deploy Configuration</h3>
-
-<p>Copy accumulo-env.sh and accumulo-site.xml from the <code class="highlighter-rouge">conf/</code> directory on the master to all Accumulo
-tablet servers. The “host” configuration files files <code class="highlighter-rouge">accumulo-cluster</code> only need to be on servers
-where that command is run.</p>
-
-<h3 id="sensitive-configuration-values">Sensitive Configuration Values</h3>
-
-<p>Accumulo has a number of properties that can be specified via the accumulo-site.xml
-file which are sensitive in nature, instance.secret and trace.token.property.password
-are two common examples. Both of these properties, if compromised, have the ability
-to result in data being leaked to users who should not have access to that data.</p>
-
-<p>In Hadoop-2.6.0, a new CredentialProvider class was introduced which serves as a common
-implementation to abstract away the storage and retrieval of passwords from plaintext
-storage in configuration files. Any Property marked with the <code class="highlighter-rouge">Sensitive</code> annotation
-is a candidate for use with these CredentialProviders. For version of Hadoop which lack
-these classes, the feature will just be unavailable for use.</p>
-
-<p>A comma separated list of CredentialProviders can be configured using the Accumulo Property
-<code class="highlighter-rouge">general.security.credential.provider.paths</code>. Each configured URL will be consulted
-when the Configuration object for accumulo-site.xml is accessed.</p>
-
-<h3 id="using-a-javakeystorecredentialprovider-for-storage">Using a JavaKeyStoreCredentialProvider for storage</h3>
-
-<p>One of the implementations provided in Hadoop-2.6.0 is a Java KeyStore CredentialProvider.
-Each entry in the KeyStore is the Accumulo Property key name. For example, to store the
-<code class="highlighter-rouge">instance.secret</code>, the following command can be used:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>hadoop credential create instance.secret --provider jceks://file/etc/accumulo/conf/accumulo.jceks
-</code></pre>
-</div>
-
-<p>The command will then prompt you to enter the secret to use and create a keystore in:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>/path/to/accumulo/conf/accumulo.jceks
-</code></pre>
-</div>
-
-<p>Then, accumulo-site.xml must be configured to use this KeyStore as a CredentialProvider:</p>
-
-<div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="nt"><property></span>
- <span class="nt"><name></span>general.security.credential.provider.paths<span class="nt"></name></span>
- <span class="nt"><value></span>jceks://file/path/to/accumulo/conf/accumulo.jceks<span class="nt"></value></span>
-<span class="nt"></property></span>
-</code></pre>
-</div>
-
-<p>This configuration will then transparently extract the <code class="highlighter-rouge">instance.secret</code> from
-the configured KeyStore and alleviates a human readable storage of the sensitive
-property.</p>
-
-<p>A KeyStore can also be stored in HDFS, which will make the KeyStore readily available to
-all Accumulo servers. If the local filesystem is used, be aware that each Accumulo server
-will expect the KeyStore in the same location.</p>
-
-<h3 id="client-configuration">Client Configuration</h3>
-
-<p>In version 1.6.0, Accumulo included a new type of configuration file known as a client
-configuration file. One problem with the traditional “site.xml” file that is prevalent
-through Hadoop is that it is a single file used by both clients and servers. This makes
-it very difficult to protect secrets that are only meant for the server processes while
-allowing the clients to connect to the servers.</p>
-
-<p>The client configuration file is a subset of the information stored in accumulo-site.xml
-meant only for consumption by clients of Accumulo. By default, Accumulo checks a number
-of locations for a client configuration by default:</p>
-
-<ul>
- <li><code class="highlighter-rouge">/path/to/accumulo/conf/client.conf</code></li>
- <li><code class="highlighter-rouge">/etc/accumulo/client.conf</code></li>
- <li><code class="highlighter-rouge">/etc/accumulo/conf/client.conf</code></li>
- <li><code class="highlighter-rouge">~/.accumulo/config</code></li>
-</ul>
-
-<p>These files are <a href="https://en.wikipedia.org/wiki/.properties">Java Properties files</a>. These files
-can currently contain information about ZooKeeper servers, RPC properties (such as SSL or SASL
-connectors), distributed tracing properties. Valid properties are defined by the <a href="https://github.com/apache/accumulo/blob/f1d0ec93d9f13ff84844b5ac81e4a7b383ced467/core/src/main/java/org/apache/accumulo/core/client/ClientConfiguration.java#L54">ClientProperty</a>
-enum contained in the client API.</p>
-
-<h4 id="custom-table-tags">Custom Table Tags</h4>
-
-<p>Accumulo has the ability for users to add custom tags to tables. This allows
-applications to set application-level metadata about a table. These tags can be
-anything from a table description, administrator notes, date created, etc.
-This is done by naming and setting a property with a prefix <code class="highlighter-rouge">table.custom.*</code>.</p>
-
-<p>Currently, table properties are stored in ZooKeeper. This means that the number
-and size of custom properties should be restricted on the order of 10’s of properties
-at most without any properties exceeding 1MB in size. ZooKeeper’s performance can be
-very sensitive to an excessive number of nodes and the sizes of the nodes. Applications
-which leverage the user of custom properties should take these warnings into
-consideration. There is no enforcement of these warnings via the API.</p>
-
-<h4 id="configuring-the-classloader">Configuring the ClassLoader</h4>
-
-<p>Accumulo builds its Java classpath in <code class="highlighter-rouge">accumulo-env.sh</code>. After an Accumulo application has started, it will load classes from the locations
-specified in the deprecated <code class="highlighter-rouge">general.classpaths</code> property. Additionally, Accumulo will load classes from the locations specified in the
-<code class="highlighter-rouge">general.dynamic.classpaths</code> property and will monitor and reload them if they change. The reloading feature is useful during the development
-and testing of iterators as new or modified iterator classes can be deployed to Accumulo without having to restart the database.
-/
-Accumulo also has an alternate configuration for the classloader which will allow it to load classes from remote locations. This mechanism
-uses Apache Commons VFS which enables locations such as http and hdfs to be used. This alternate configuration also uses the
-<code class="highlighter-rouge">general.classpaths</code> property in the same manner described above. It differs in that you need to configure the
-<code class="highlighter-rouge">general.vfs.classpaths</code> property instead of the <code class="highlighter-rouge">general.dynamic.classpath</code> property. As in the default configuration, this alternate
-configuration will also monitor the vfs locations for changes and reload if necessary.</p>
-
-<p>The Accumulo classpath can be viewed in human readable format by running <code class="highlighter-rouge">accumulo classpath -d</code>.</p>
-
-<h5 id="classloader-contexts">ClassLoader Contexts</h5>
-
-<p>With the addition of the VFS based classloader, we introduced the notion of classloader contexts. A context is identified
-by a name and references a set of locations from which to load classes and can be specified in the accumulo-site.xml file or added
-using the <code class="highlighter-rouge">config</code> command in the shell. Below is an example for specify the app1 context in the accumulo-site.xml file:</p>
-
-<div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="nt"><property></span>
- <span class="nt"><name></span>general.vfs.context.classpath.app1<span class="nt"></name></span>
- <span class="nt"><value></span>hdfs://localhost:8020/applicationA/classpath/.*.jar,file:///opt/applicationA/lib/.*.jar<span class="nt"></value></span>
- <span class="nt"><description></span>Application A classpath, loads jars from HDFS and local file system<span class="nt"></description></span>
-<span class="nt"></property></span>
-</code></pre>
-</div>
-
-<p>The default behavior follows the Java ClassLoader contract in that classes, if they exists, are loaded from the parent classloader first.
-You can override this behavior by delegating to the parent classloader after looking in this classloader first. An example of this
-configuration is:</p>
-
-<div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="nt"><property></span>
- <span class="nt"><name></span>general.vfs.context.classpath.app1.delegation=post<span class="nt"></name></span>
- <span class="nt"><value></span>hdfs://localhost:8020/applicationA/classpath/.*.jar,file:///opt/applicationA/lib/.*.jar<span class="nt"></value></span>
- <span class="nt"><description></span>Application A classpath, loads jars from HDFS and local file system<span class="nt"></description></span>
-<span class="nt"></property></span>
-</code></pre>
-</div>
-
-<p>To use contexts in your application you can set the <code class="highlighter-rouge">table.classpath.context</code> on your tables or use the <code class="highlighter-rouge">setClassLoaderContext()</code> method on Scanner
-and BatchScanner passing in the name of the context, app1 in the example above. Setting the property on the table allows your minc, majc, and scan
-iterators to load classes from the locations defined by the context. Passing the context name to the scanners allows you to override the table setting
-to load only scan time iterators from a different location.</p>
-
-<h2 id="initialization">Initialization</h2>
-
-<p>Accumulo must be initialized to create the structures it uses internally to locate
-data across the cluster. HDFS is required to be configured and running before
-Accumulo can be initialized.</p>
-
-<p>Once HDFS is started, initialization can be performed by executing
-<code class="highlighter-rouge">accumulo init</code> . This script will prompt for a name
-for this instance of Accumulo. The instance name is used to identify a set of tables
-and instance-specific settings. The script will then write some information into
-HDFS so Accumulo can start properly.</p>
-
-<p>The initialization script will prompt you to set a root password. Once Accumulo is
-initialized it can be started.</p>
-
-<h2 id="running">Running</h2>
-
-<h3 id="starting-accumulo">Starting Accumulo</h3>
-
-<p>Make sure Hadoop is configured on all of the machines in the cluster, including
-access to a shared HDFS instance. Make sure HDFS and ZooKeeper are running.
-Make sure ZooKeeper is configured and running on at least one machine in the
-cluster.
-Start Accumulo using <code class="highlighter-rouge">accumulo-cluster start</code>.</p>
-
-<p>To verify that Accumulo is running, check the <a href="/docs/unreleased/administration/overview#monitoring">Accumulo monitor</a>.
-In addition, the Shell can provide some information about the status of tables via reading the metadata tables.</p>
-
-<h3 id="stopping-accumulo">Stopping Accumulo</h3>
-
-<p>To shutdown cleanly, run <code class="highlighter-rouge">accumulo-cluster stop</code> and the master will orchestrate the
-shutdown of all the tablet servers. Shutdown waits for all minor compactions to finish, so it may
-take some time for particular configurations.</p>
-
-<h3 id="adding-a-tablet-server">Adding a Tablet Server</h3>
-
-<p>Update your <code class="highlighter-rouge">conf/tservers</code> file to account for the addition.</p>
-
-<p>Next, ssh to each of the hosts you want to add and run:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>accumulo-service tserver start
-</code></pre>
-</div>
-
-<p>Make sure the host in question has the new configuration, or else the tablet
-server won’t start; at a minimum this needs to be on the host(s) being added,
-but in practice it’s good to ensure consistent configuration across all nodes.</p>
-
-<h3 id="decomissioning-a-tablet-server">Decomissioning a Tablet Server</h3>
-
-<p>If you need to take a node out of operation, you can trigger a graceful shutdown of a tablet
-server. Accumulo will automatically rebalance the tablets across the available tablet servers.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>accumulo admin stop <host(s)> {<host> ...}
-</code></pre>
-</div>
-
-<p>Alternatively, you can ssh to each of the hosts you want to remove and run:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>accumulo-service tserver stop
-</code></pre>
-</div>
-
-<p>Be sure to update your <code class="highlighter-rouge">conf/tservers</code> file to
-account for the removal of these hosts. Bear in mind that the monitor will not re-read the
-tservers file automatically, so it will report the decommissioned servers as down; it’s
-recommended that you restart the monitor so that the node list is up to date.</p>
-
-<p>The steps described to decommission a node can also be used (without removal of the host
-from the <code class="highlighter-rouge">conf/tservers</code> file) to gracefully stop a node. This will
-ensure that the tabletserver is cleanly stopped and recovery will not need to be performed
-when the tablets are re-hosted.</p>
-
-<h3 id="restarting-process-on-a-node">Restarting process on a node</h3>
-
-<p>Occasionally, it might be necessary to restart the processes on a specific node. In addition
-to the <code class="highlighter-rouge">accumulo-cluster</code> script, Accumulo has a <code class="highlighter-rouge">accumulo-service</code> script that
-can be use to start/stop processes on a node.</p>
-
-<h4 id="a-note-on-rolling-restarts">A note on rolling restarts</h4>
-
-<p>For sufficiently large Accumulo clusters, restarting multiple TabletServers within a short window can place significant
-load on the Master server. If slightly lower availability is acceptable, this load can be reduced by globally setting
-<code class="highlighter-rouge">table.suspend.duration</code> to a positive value.</p>
-
-<p>With <code class="highlighter-rouge">table.suspend.duration</code> set to, say, <code class="highlighter-rouge">5m</code>, Accumulo will wait
-for 5 minutes for any dead TabletServer to return before reassigning that TabletServer’s responsibilities to other TabletServers.
-If the TabletServer returns to the cluster before the specified timeout has elapsed, Accumulo will assign the TabletServer
-its original responsibilities.</p>
-
-<p>It is important not to choose too large a value for <code class="highlighter-rouge">table.suspend.duration</code>, as during this time, all scans against the
-data that TabletServer had hosted will block (or time out).</p>
-
-<h3 id="running-multiple-tabletservers-on-a-single-node">Running multiple TabletServers on a single node</h3>
-
-<p>With very powerful nodes, it may be beneficial to run more than one TabletServer on a given
-node. This decision should be made carefully and with much deliberation as Accumulo is designed
-to be able to scale to using 10’s of GB of RAM and 10’s of CPU cores.</p>
-
-<p>Accumulo TabletServers bind certain ports on the host to accommodate remote procedure calls to/from
-other nodes. Running more than one TabletServer on a host requires that you set the environment variable
-<code class="highlighter-rouge">ACCUMULO_SERVICE_INSTANCE</code> to an instance number (i.e 1, 2) for each instance that is started. Also, set
-these properties in <code class="highlighter-rouge">accumulo-site.xml</code>:</p>
-
-<div class="language-xml highlighter-rouge"><pre class="highlight"><code> <span class="nt"><property></span>
- <span class="nt"><name></span>tserver.port.search<span class="nt"></name></span>
- <span class="nt"><value></span>true<span class="nt"></value></span>
- <span class="nt"></property></span>
- <span class="nt"><property></span>
- <span class="nt"><name></span>replication.receipt.service.port<span class="nt"></name></span>
- <span class="nt"><value></span>0<span class="nt"></value></span>
- <span class="nt"></property></span>
-</code></pre>
-</div>
-
-<h2 id="monitoring">Monitoring</h2>
-
-<h3 id="accumulo-monitor">Accumulo Monitor</h3>
-
-<p>The Accumulo Monitor provides an interface for monitoring the status and health of
-Accumulo components. The Accumulo Monitor provides a web UI for accessing this information at
-<code class="highlighter-rouge">http://_monitorhost_:9995/</code>.</p>
-
-<p>Things highlighted in yellow may be in need of attention.
-If anything is highlighted in red on the monitor page, it is something that definitely needs attention.</p>
-
-<p>The Overview page contains some summary information about the Accumulo instance, including the version, instance name, and instance ID.
-There is a table labeled Accumulo Master with current status, a table listing the active Zookeeper servers, and graphs displaying various metrics over time.
-These include ingest and scan performance and other useful measurements.</p>
-
-<p>The Master Server, Tablet Servers, and Tables pages display metrics grouped in different ways (e.g. by tablet server or by table).
-Metrics typically include number of entries (key/value pairs), ingest and query rates.
-The number of running scans, major and minor compactions are in the form <em>number_running</em> (<em>number_queued</em>).
-Another important metric is hold time, which is the amount of time a tablet has been waiting but unable to flush its memory in a minor compaction.</p>
-
-<p>The Server Activity page graphically displays tablet server status, with each server represented as a circle or square.
-Different metrics may be assigned to the nodes’ color and speed of oscillation.
-The Overall Avg metric is only used on the Server Activity page, and represents the average of all the other metrics (after normalization).
-Similarly, the Overall Max metric picks the metric with the maximum normalized value.</p>
-
-<p>The Garbage Collector page displays a list of garbage collection cycles, the number of files found of each type (including deletion candidates in use and files actually deleted), and the length of the deletion cycle.
-The Traces page displays data for recent traces performed (see the following section for information on <a href="/docs/unreleased/administration/overview#tracing">tracing</a>).
-The Recent Logs page displays warning and error logs forwarded to the monitor from all Accumulo processes.
-Also, the XML and JSON links provide metrics in XML and JSON formats, respectively.</p>
-
-<h3 id="ssl">SSL</h3>
-
-<p>SSL may be enabled for the monitor page by setting the following properties in the <code class="highlighter-rouge">accumulo-site.xml</code> file:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>monitor.ssl.keyStore
-monitor.ssl.keyStorePassword
-monitor.ssl.trustStore
-monitor.ssl.trustStorePassword
-</code></pre>
-</div>
-
-<p>If the Accumulo conf directory has been configured (in particular the <code class="highlighter-rouge">accumulo-env.sh</code> file must be set up), the
-<code class="highlighter-rouge">accumulo-util gen-monitor-cert</code> command can be used to create the keystore and truststore files with random passwords. The command
-will print out the properties that need to be added to the <code class="highlighter-rouge">accumulo-site.xml</code> file. The stores can also be generated manually with the
-Java <code class="highlighter-rouge">keytool</code> command, whose usage can be seen in the <code class="highlighter-rouge">accumulo-util</code> script.</p>
-
-<p>If desired, the SSL ciphers allowed for connections can be controlled via the following properties in <code class="highlighter-rouge">accumulo-site.xml</code>:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>monitor.ssl.include.ciphers
-monitor.ssl.exclude.ciphers
-</code></pre>
-</div>
-
-<p>If SSL is enabled, the monitor URL can only be accessed via https.
-This also allows you to access the Accumulo shell through the monitor page.
-The left navigation bar will have a new link to Shell.
-An Accumulo user name and password must be entered for access to the shell.</p>
-
-<h2 id="metrics">Metrics</h2>
-
-<p>Accumulo can expose metrics through a legacy metrics library and using the Hadoop Metrics2 library.</p>
-
-<h3 id="legacy-metrics">Legacy Metrics</h3>
-
-<p>Accumulo has a legacy metrics library that can be exposes metrics using JMX endpoints or file-based logging. These metrics can
-be enabled by setting <code class="highlighter-rouge">general.legacy.metrics</code> to <code class="highlighter-rouge">true</code> in <code class="highlighter-rouge">accumulo-site.xml</code> and placing the <code class="highlighter-rouge">accumulo-metrics.xml</code>
-configuration file on the classpath (which is typically done by placing the file in the <code class="highlighter-rouge">conf/</code> directory). A template for
-<code class="highlighter-rouge">accumulo-metrics.xml</code> can be found in <code class="highlighter-rouge">conf/templates</code> of the Accumulo tarball.</p>
-
-<h3 id="hadoop-metrics2">Hadoop Metrics2</h3>
-
-<p>Hadoop Metrics2 is a library which allows for routing of metrics generated by registered MetricsSources to
-configured MetricsSinks. Examples of sinks that are implemented by Hadoop include file-based logging, Graphite and Ganglia.
-All metric sources are exposed via JMX when using Metrics2.</p>
-
-<p>Metrics2 is configured by examining the classpath for a file that matches <code class="highlighter-rouge">hadoop-metrics2*.properties</code>. The Accumulo tarball
-contains an example <code class="highlighter-rouge">hadoop-metrics2-accumulo.properties</code> file in <code class="highlighter-rouge">conf/templates</code> which can be copied to <code class="highlighter-rouge">conf/</code> to place
-on classpath. This file is used to enable file, Graphite or Ganglia sinks (some minimal configuration required for Graphite
-and Ganglia). Because the Hadoop configuration is also on the Accumulo classpath, be sure that you do not have multiple
-Metrics2 configuration files. It is recommended to consolidate metrics in a single properties file in a central location to
-remove ambiguity. The contents of <code class="highlighter-rouge">hadoop-metrics2-accumulo.properties</code> can be added to a central <code class="highlighter-rouge">hadoop-metrics2.properties</code>
-in <code class="highlighter-rouge">$HADOOP_CONF_DIR</code>.</p>
-
-<p>As a note for configuring the file sink, the provided path should be absolute. A relative path or file name will be created relative
-to the directory in which the Accumulo process was started. External tools, such as logrotate, can be used to prevent these files
-from growing without bound.</p>
-
-<p>Each server process should have log messages from the Metrics2 library about the sinks that were created. Be sure to check
-the Accumulo processes log files when debugging missing metrics output.</p>
-
-<p>For additional information on configuring Metrics2, visit the <a href="https://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html">Javadoc page for Metrics2</a>.</p>
-
-<h2 id="tracing">Tracing</h2>
-
-<p>It can be difficult to determine why some operations are taking longer
-than expected. For example, you may be looking up items with very low
-latency, but sometimes the lookups take much longer. Determining the
-cause of the delay is difficult because the system is distributed, and
-the typical lookup is fast.</p>
-
-<p>Accumulo has been instrumented to record the time that various
-operations take when tracing is turned on. The fact that tracing is
-enabled follows all the requests made on behalf of the user throughout
-the distributed infrastructure of accumulo, and across all threads of
-execution.</p>
-
-<p>These time spans will be inserted into the <code class="highlighter-rouge">trace</code> table in
-Accumulo. You can browse recent traces from the Accumulo monitor
-page. You can also read the <code class="highlighter-rouge">trace</code> table directly like any
-other table.</p>
-
-<p>The design of Accumulo’s distributed tracing follows that of <a href="http://research.google.com/pubs/pub36356.html">Google’s Dapper</a>.</p>
-
-<h3 id="tracers">Tracers</h3>
-
-<p>To collect traces, Accumulo needs at least one tracer server running. If you are using <code class="highlighter-rouge">accumulo-cluster</code> to start your cluster,
-configure your server in <code class="highlighter-rouge">conf/tracers</code>. The server collects traces from clients and writes them to the <code class="highlighter-rouge">trace</code> table. The Accumulo
-user that the tracer connects to Accumulo with can be configured with the following properties (see the <a href="/docs/unreleased/administration/configuration-management">configuration management</a>
-page for setting Accumulo server properties)</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>trace.user
-trace.token.property.password
-</code></pre>
-</div>
-
-<p>Other tracer configuration properties include</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>trace.port.client - port tracer listens on
-trace.table - table tracer writes to
-trace.zookeeper.path - zookeeper path where tracers register
-</code></pre>
-</div>
-
-<p>The zookeeper path is configured to /tracers by default. If
-multiple Accumulo instances are sharing the same ZooKeeper
-quorum, take care to configure Accumulo with unique values for
-this property.</p>
-
-<h3 id="configuring-tracing">Configuring Tracing</h3>
-
-<p>Traces are collected via SpanReceivers. The default SpanReceiver
-configured is org.apache.accumulo.core.trace.ZooTraceClient, which
-sends spans to an Accumulo Tracer process, as discussed in the
-previous section. This default can be changed to a different span
-receiver, or additional span receivers can be added in a
-comma-separated list, by modifying the property</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>trace.span.receivers
-</code></pre>
-</div>
-
-<p>Individual span receivers may require their own configuration
-parameters, which are grouped under the trace.span.receiver.*
-prefix. ZooTraceClient uses the following properties. The first
-three properties are populated from other Accumulo properties,
-while the remaining ones should be prefixed with
-trace.span.receiver. when set in the Accumulo configuration.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>tracer.zookeeper.host - populated from instance.zookeepers
-tracer.zookeeper.timeout - populated from instance.zookeeper.timeout
-tracer.zookeeper.path - populated from trace.zookeeper.path
-tracer.send.timer.millis - timer for flushing send queue (in ms, default 1000)
-tracer.queue.size - max queue size (default 5000)
-tracer.span.min.ms - minimum span length to store (in ms, default 1)
-</code></pre>
-</div>
-
-<p>Note that to configure an Accumulo client for tracing, including
-the Accumulo shell, the client configuration must be given the same
-trace.span.receivers, trace.span.receiver.*, and trace.zookeeper.path
-properties as the servers have.</p>
-
-<p>Hadoop can also be configured to send traces to Accumulo, as of
-Hadoop 2.6.0, by setting properties in Hadoop’s core-site.xml
-file. Instead of using the trace.span.receiver.* prefix, Hadoop
-uses hadoop.htrace.*. The Hadoop configuration does not have
-access to Accumulo’s properties, so the
-hadoop.htrace.tracer.zookeeper.host property must be specified.
-The zookeeper timeout defaults to 30000 (30 seconds), and the
-zookeeper path defaults to /tracers. An example of configuring
-Hadoop to send traces to ZooTraceClient is</p>
-
-<div class="language-xml highlighter-rouge"><pre class="highlight"><code> <span class="nt"><property></span>
- <span class="nt"><name></span>hadoop.htrace.spanreceiver.classes<span class="nt"></name></span>
- <span class="nt"><value></span>org.apache.accumulo.core.trace.ZooTraceClient<span class="nt"></value></span>
- <span class="nt"></property></span>
- <span class="nt"><property></span>
- <span class="nt"><name></span>hadoop.htrace.tracer.zookeeper.host<span class="nt"></name></span>
- <span class="nt"><value></span>zookeeperHost:2181<span class="nt"></value></span>
- <span class="nt"></property></span>
- <span class="nt"><property></span>
- <span class="nt"><name></span>hadoop.htrace.tracer.zookeeper.path<span class="nt"></name></span>
- <span class="nt"><value></span>/tracers<span class="nt"></value></span>
- <span class="nt"></property></span>
- <span class="nt"><property></span>
- <span class="nt"><name></span>hadoop.htrace.tracer.span.min.ms<span class="nt"></name></span>
- <span class="nt"><value></span>1<span class="nt"></value></span>
- <span class="nt"></property></span>
-</code></pre>
-</div>
-
-<p>The accumulo-core, accumulo-tracer, accumulo-fate and libthrift
-jars must also be placed on Hadoop’s classpath.</p>
-
-<h5 id="adding-additional-spanreceivers">Adding additional SpanReceivers</h5>
-
-<p><a href="https://github.com/openzipkin/zipkin">Zipkin</a> has a SpanReceiver supported by HTrace and popularized by Twitter
-that users looking for a more graphical trace display may opt to use.
-The following steps configure Accumulo to use <code class="highlighter-rouge">org.apache.htrace.impl.ZipkinSpanReceiver</code>
-in addition to the Accumulo’s default ZooTraceClient, and they serve as a template
-for adding any SpanReceiver to Accumulo:</p>
-
-<ol>
- <li>
- <p>Add the Jar containing the ZipkinSpanReceiver class file to the
-<code class="highlighter-rouge">lib/</code> directory. It is critical that the Jar is placed in
-<code class="highlighter-rouge">lib/</code> and NOT in <code class="highlighter-rouge">lib/ext/</code> so that the new SpanReceiver class
-is visible to the same class loader of htrace-core.</p>
- </li>
- <li>
- <p>Add the following to <code class="highlighter-rouge">accumulo-site.xml</code>:</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code> <property>
- <name>trace.span.receivers</name>
- <value>org.apache.accumulo.tracer.ZooTraceClient,org.apache.htrace.impl.ZipkinSpanReceiver</value>
- </property>
-</code></pre>
- </div>
- </li>
- <li>
- <p>Restart your Accumulo tablet servers.</p>
- </li>
-</ol>
-
-<p>In order to use ZipkinSpanReceiver from a client as well as the Accumulo server,</p>
-
-<ol>
- <li>
- <p>Ensure your client can see the ZipkinSpanReceiver class at runtime. For Maven projects,
-this is easily done by adding to your client’s pom.xml (taking care to specify a good version)</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code> <dependency>
- <groupId>org.apache.htrace</groupId>
- <artifactId>htrace-zipkin</artifactId>
- <version>3.1.0-incubating</version>
- <scope>runtime</scope>
- </dependency>
-</code></pre>
- </div>
- </li>
- <li>
- <p>Add the following to your client configuration.</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code> trace.span.receivers=org.apache.accumulo.tracer.ZooTraceClient,org.apache.htrace.impl.ZipkinSpanReceiver
-</code></pre>
- </div>
- </li>
- <li>
- <p>Instrument your client as in the next section.</p>
- </li>
-</ol>
-
-<p>Your SpanReceiver may require additional properties, and if so these should likewise
-be placed in the ClientConfiguration (if applicable) and Accumulo’s <code class="highlighter-rouge">accumulo-site.xml</code>.
-Two such properties for ZipkinSpanReceiver, listed with their default values, are</p>
-
-<div class="language-xml highlighter-rouge"><pre class="highlight"><code> <span class="nt"><property></span>
- <span class="nt"><name></span>trace.span.receiver.zipkin.collector-hostname<span class="nt"></name></span>
- <span class="nt"><value></span>localhost<span class="nt"></value></span>
- <span class="nt"></property></span>
- <span class="nt"><property></span>
- <span class="nt"><name></span>trace.span.receiver.zipkin.collector-port<span class="nt"></name></span>
- <span class="nt"><value></span>9410<span class="nt"></value></span>
- <span class="nt"></property></span>
-</code></pre>
-</div>
-
-<h4 id="instrumenting-a-client">Instrumenting a Client</h4>
-
-<p>Tracing can be used to measure a client operation, such as a scan, as
-the operation traverses the distributed system. To enable tracing for
-your application call</p>
-
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.accumulo.core.trace.DistributedTrace</span><span class="o">;</span>
-<span class="o">...</span>
-<span class="n">DistributedTrace</span><span class="o">.</span><span class="na">enable</span><span class="o">(</span><span class="n">hostname</span><span class="o">,</span> <span class="s">"myApplication"</span><span class="o">);</span>
-<span class="c1">// do some tracing</span>
-<span class="o">...</span>
-<span class="n">DistributedTrace</span><span class="o">.</span><span class="na">disable</span><span class="o">();</span>
-</code></pre>
-</div>
-
-<p>Once tracing has been enabled, a client can wrap an operation in a trace.</p>
-
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.htrace.Sampler</span><span class="o">;</span>
-<span class="kn">import</span> <span class="nn">org.apache.htrace.Trace</span><span class="o">;</span>
-<span class="kn">import</span> <span class="nn">org.apache.htrace.TraceScope</span><span class="o">;</span>
-<span class="o">...</span>
-<span class="n">TraceScope</span> <span class="n">scope</span> <span class="o">=</span> <span class="n">Trace</span><span class="o">.</span><span class="na">startSpan</span><span class="o">(</span><span class="s">"Client Scan"</span><span class="o">,</span> <span class="n">Sampler</span><span class="o">.</span><span class="na">ALWAYS</span><span class="o">);</span>
-<span class="n">BatchScanner</span> <span class="n">scanner</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="na">createBatchScanner</span><span class="o">(...);</span>
-<span class="c1">// Configure your scanner</span>
-<span class="k">for</span> <span class="o">(</span><span class="n">Entry</span> <span class="n">entry</span> <span class="o">:</span> <span class="n">scanner</span><span class="o">)</span> <span class="o">{</span>
-<span class="o">}</span>
-<span class="n">scope</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
-</code></pre>
-</div>
-
-<p>The user can create additional Spans within a Trace.</p>
-
-<p>The sampler (such as <code class="highlighter-rouge">Sampler.ALWAYS</code>) for the trace should only be specified with a top-level span,
-and subsequent spans will be collected depending on whether that first span was sampled.
-Don’t forget to specify a Sampler at the top-level span
-because the default Sampler only samples when part of a pre-existing trace,
-which will never occur in a client that never specifies a Sampler.</p>
-
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">TraceScope</span> <span class="n">scope</span> <span class="o">=</span> <span class="n">Trace</span><span class="o">.</span><span class="na">startSpan</span><span class="o">(</span><span class="s">"Client Update"</span><span class="o">,</span> <span class="n">Sampler</span><span class="o">.</span><span class="na">ALWAYS</span><span class="o">);</span>
-<span class="o">...</span>
-<span class="n">TraceScope</span> <span class="n">readScope</span> <span class="o">=</span> <span class="n">Trace</span><span class="o">.</span><span class="na">startSpan</span><span class="o">(</span><span class="s">"Read"</span><span class="o">);</span>
-<span class="o">...</span>
-<span class="n">readScope</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
-<span class="o">...</span>
-<span class="n">TraceScope</span> <span class="n">writeScope</span> <span class="o">=</span> <span class="n">Trace</span><span class="o">.</span><span class="na">startSpan</span><span class="o">(</span><span class="s">"Write"</span><span class="o">);</span>
-<span class="o">...</span>
-<span class="n">writeScope</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
-<span class="n">scope</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
-</code></pre>
-</div>
-
-<p>Like Dapper, Accumulo tracing supports user defined annotations to associate additional data with a Trace.
-Checking whether currently tracing is necessary when using a sampler other than Sampler.ALWAYS.</p>
-
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="o">...</span>
-<span class="kt">int</span> <span class="n">numberOfEntriesRead</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
-<span class="n">TraceScope</span> <span class="n">readScope</span> <span class="o">=</span> <span class="n">Trace</span><span class="o">.</span><span class="na">startSpan</span><span class="o">(</span><span class="s">"Read"</span><span class="o">);</span>
-<span class="c1">// Do the read, update the counter</span>
-<span class="o">...</span>
-<span class="k">if</span> <span class="o">(</span><span class="n">Trace</span><span class="o">.</span><span class="na">isTracing</span><span class="o">)</span>
- <span class="n">readScope</span><span class="o">.</span><span class="na">getSpan</span><span class="o">().</span><span class="na">addKVAnnotation</span><span class="o">(</span><span class="s">"Number of Entries Read"</span><span class="o">.</span><span class="na">getBytes</span><span class="o">(</span><span class="n">StandardCharsets</span><span class="o">.</span><span class="na">UTF_8</span><span class="o">),</span>
- <span class="n">String</span><span class="o">.</span><span class="na">valueOf</span><span class="o">(</span><span class="n">numberOfEntriesRead</span><span class="o">).</span><span class="na">getBytes</span><span class="o">(</span><span class="n">StandardCharsets</span><span class="o">.</span><span class="na">UTF_8</span><span class="o">));</span>
-</code></pre>
-</div>
-
-<p>It is also possible to add timeline annotations to your spans.
-This associates a string with a given timestamp between the start and stop times for a span.</p>
-
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="o">...</span>
-<span class="n">writeScope</span><span class="o">.</span><span class="na">getSpan</span><span class="o">().</span><span class="na">addTimelineAnnotation</span><span class="o">(</span><span class="s">"Initiating Flush"</span><span class="o">);</span>
-</code></pre>
-</div>
-
-<p>Some client operations may have a high volume within your
-application. As such, you may wish to only sample a percentage of
-operations for tracing. As seen below, the CountSampler can be used to
-help enable tracing for 1-in-1000 operations</p>
-
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.htrace.impl.CountSampler</span><span class="o">;</span>
-<span class="o">...</span>
-<span class="n">Sampler</span> <span class="n">sampler</span> <span class="o">=</span> <span class="k">new</span> <span class="n">CountSampler</span><span class="o">(</span><span class="n">HTraceConfiguration</span><span class="o">.</span><span class="na">fromMap</span><span class="o">(</span>
- <span class="n">Collections</span><span class="o">.</span><span class="na">singletonMap</span><span class="o">(</span><span class="n">CountSampler</span><span class="o">.</span><span class="na">SAMPLER_FREQUENCY_CONF_KEY</span><span class="o">,</span> <span class="s">"1000"</span><span class="o">)));</span>
-<span class="o">...</span>
-<span class="n">TraceScope</span> <span class="n">readScope</span> <span class="o">=</span> <span class="n">Trace</span><span class="o">.</span><span class="na">startSpan</span><span class="o">(</span><span class="s">"Read"</span><span class="o">,</span> <span class="n">sampler</span><span class="o">);</span>
-<span class="o">...</span>
-<span class="n">readScope</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
-</code></pre>
-</div>
-
-<p>Remember to close all spans and disable tracing when finished.</p>
-
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">DistributedTrace</span><span class="o">.</span><span class="na">disable</span><span class="o">();</span>
-</code></pre>
-</div>
-
-<h3 id="viewing-collected-traces">Viewing Collected Traces</h3>
-
-<p>To view collected traces, use the “Recent Traces” link on the Monitor
-UI. You can also programmatically access and print traces using the
-<code class="highlighter-rouge">TraceDump</code> class.</p>
-
-<h4 id="trace-table-format">Trace Table Format</h4>
-
-<p>This section is for developers looking to use data recorded in the trace table
-directly, above and beyond the default services of the Accumulo monitor.
-Please note the trace table format and its supporting classes
-are not in the public API and may be subject to change in future versions.</p>
-
-<p>Each span received by a tracer’s ZooTraceClient is recorded in the trace table
-in the form of three entries: span entries, index entries, and start time entries.
-Span and start time entries record full span information,
-whereas index entries provide indexing into span information
-useful for quickly finding spans by type or start time.</p>
-
-<p>Each entry is illustrated by a description and sample of data.
-In the description, a token in quotes is a String literal,
-whereas other other tokens are span variables.
-Parentheses group parts together, to distinguish colon characters inside the
-column family or qualifier from the colon that separates column family and qualifier.
-We use the format <code class="highlighter-rouge">row columnFamily:columnQualifier columnVisibility value</code>
-(omitting timestamp which records the time an entry is written to the trace table).</p>
-
-<p>Span entries take the following form:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>traceId "span":(parentSpanId:spanId) [] spanBinaryEncoding
-63b318de80de96d1 span:4b8f66077df89de1:3778c6739afe4e1 [] %18;%09;...
-</code></pre>
-</div>
-
-<p>The parentSpanId is “” for the root span of a trace.
-The spanBinaryEncoding is a compact Apache Thrift encoding of the original Span object.
-This allows clients (and the Accumulo monitor) to recover all the details of the original Span
-at a later time, by scanning the trace table and decoding the value of span entries
-via <code class="highlighter-rouge">TraceFormatter.getRemoteSpan(entry)</code>.</p>
-
-<p>The trace table has a formatter class by default (org.apache.accumulo.tracer.TraceFormatter)
-that changes how span entries appear from the Accumulo shell.
-Normal scans to the trace table do not use this formatter representation;
-it exists only to make span entries easier to view inside the Accumulo shell.</p>
-
-<p>Index entries take the following form:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>"idx":service:startTime description:sender [] traceId:elapsedTime
-idx:tserver:14f3828f58b startScan:localhost [] 63b318de80de96d1:1
-</code></pre>
-</div>
-
-<p>The service and sender are set by the first call of each Accumulo process
-(and instrumented client processes) to <code class="highlighter-rouge">DistributedTrace.enable(...)</code>
-(the sender is autodetected if not specified).
-The description is specified in each span.
-Start time and the elapsed time (start - stop, 1 millisecond in the example above)
-are recorded in milliseconds as long values serialized to a string in hex.</p>
-
-<p>Start time entries take the following form:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>"start":startTime "id":traceId [] spanBinaryEncoding
-start:14f3828a351 id:63b318de80de96d1 [] %18;%09;...
-</code></pre>
-</div>
-
-<p>The following classes may be run while Accumulo is running to provide insight into trace statistics. These require
-accumulo-trace-VERSION.jar to be provided on the Accumulo classpath (<code class="highlighter-rouge">lib/ext</code> is fine).</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.tracer.TraceTableStats -u username -p password -i instancename
-$ accumulo org.apache.accumulo.tracer.TraceDump -u username -p password -i instancename -r
-</code></pre>
-</div>
-
-<h3 id="tracing-from-the-shell">Tracing from the Shell</h3>
-<p>You can enable tracing for operations run from the shell by using the
-<code class="highlighter-rouge">trace on</code> and <code class="highlighter-rouge">trace off</code> commands.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>root@test test> trace on
-
-root@test test> scan
-a b:c [] d
-
-root@test test> trace off
-Waiting for trace information
-Waiting for trace information
-Trace started at 2013/08/26 13:24:08.332
-Time Start Service@Location Name
- 3628+0 shell@localhost shell:root
- 8+1690 shell@localhost scan
- 7+1691 shell@localhost scan:location
- 6+1692 tserver@localhost startScan
- 5+1692 tserver@localhost tablet read ahead 6
-</code></pre>
-</div>
-
-<h2 id="logging">Logging</h2>
-
-<p>Accumulo processes each write to a set of log files. By default, these logs are found at directory
-set by <code class="highlighter-rouge">ACCUMULO_LOG_DIR</code> in <code class="highlighter-rouge">accumulo-env.sh</code>.</p>
-
-<h2 id="recovery">Recovery</h2>
-
-<p>In the event of TabletServer failure or error on shutting Accumulo down, some
-mutations may not have been minor compacted to HDFS properly. In this case,
-Accumulo will automatically reapply such mutations from the write-ahead log
-either when the tablets from the failed server are reassigned by the Master (in the
-case of a single TabletServer failure) or the next time Accumulo starts (in the event of
-failure during shutdown).</p>
-
-<p>Recovery is performed by asking a tablet server to sort the logs so that tablets can easily find their missing
-updates. The sort status of each file is displayed on
-Accumulo monitor status page. Once the recovery is complete any
-tablets involved should return to an ``online’’ state. Until then those tablets will be
-unavailable to clients.</p>
-
-<p>The Accumulo client library is configured to retry failed mutations and in many
-cases clients will be able to continue processing after the recovery process without
-throwing an exception.</p>
-
-<h2 id="migrating-accumulo-from-non-ha-namenode-to-ha-namenode">Migrating Accumulo from non-HA Namenode to HA Namenode</h2>
-
-<p>The following steps will allow a non-HA instance to be migrated to an HA instance. Consider an HDFS URL
-<code class="highlighter-rouge">hdfs://namenode.example.com:8020</code> which is going to be moved to <code class="highlighter-rouge">hdfs://nameservice1</code>.</p>
-
-<p>Before moving HDFS over to the HA namenode, use <code class="highlighter-rouge">accumulo admin volumes</code> to confirm
-that the only volume displayed is the volume from the current namenode’s HDFS URL.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>Listing volumes referenced in zookeeper
- Volume : hdfs://namenode.example.com:8020/accumulo
-
-Listing volumes referenced in accumulo.root tablets section
- Volume : hdfs://namenode.example.com:8020/accumulo
-Listing volumes referenced in accumulo.root deletes section (volume replacement occurrs at deletion time)
-
-Listing volumes referenced in accumulo.metadata tablets section
- Volume : hdfs://namenode.example.com:8020/accumulo
-
-Listing volumes referenced in accumulo.metadata deletes section (volume replacement occurrs at deletion time)
-</code></pre>
-</div>
-
-<p>After verifying the current volume is correct, shut down the cluster and transition HDFS to the HA nameservice.</p>
-
-<p>Edit <code class="highlighter-rouge">accumulo-site.xml</code> to notify accumulo that a volume is being replaced. First,
-add the new nameservice volume to the <code class="highlighter-rouge">instance.volumes</code> property. Next, add the
-<code class="highlighter-rouge">instance.volumes.replacements</code> property in the form of <code class="highlighter-rouge">old new</code>. It’s important to not include
-the volume that’s being replaced in <code class="highlighter-rouge">instance.volumes</code>, otherwise it’s possible accumulo could continue
-to write to the volume.</p>
-
-<div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="c"><!-- instance.dfs.uri and instance.dfs.dir should not be set--></span>
-<span class="nt"><property></span>
- <span class="nt"><name></span>instance.volumes<span class="nt"></name></span>
- <span class="nt"><value></span>hdfs://nameservice1/accumulo<span class="nt"></value></span>
-<span class="nt"></property></span>
-<span class="nt"><property></span>
- <span class="nt"><name></span>instance.volumes.replacements<span class="nt"></name></span>
- <span class="nt"><value></span>hdfs://namenode.example.com:8020/accumulo hdfs://nameservice1/accumulo<span class="nt"></value></span>
-<span class="nt"></property></span>
-</code></pre>
-</div>
-
-<p>Run <code class="highlighter-rouge">accumulo init --add-volumes</code> and start up the accumulo cluster. Verify that the
-new nameservice volume shows up with <code class="highlighter-rouge">accumulo admin volumes</code>.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>Listing volumes referenced in zookeeper
- Volume : hdfs://namenode.example.com:8020/accumulo
- Volume : hdfs://nameservice1/accumulo
-
-Listing volumes referenced in accumulo.root tablets section
- Volume : hdfs://namenode.example.com:8020/accumulo
- Volume : hdfs://nameservice1/accumulo
-Listing volumes referenced in accumulo.root deletes section (volume replacement occurrs at deletion time)
-
-Listing volumes referenced in accumulo.metadata tablets section
- Volume : hdfs://namenode.example.com:8020/accumulo
- Volume : hdfs://nameservice1/accumulo
-Listing volumes referenced in accumulo.metadata deletes section (volume replacement occurrs at deletion time)
-</code></pre>
-</div>
-
-<p>Some erroneous GarbageCollector messages may still be seen for a small period while data is transitioning to
-the new volumes. This is expected and can usually be ignored.</p>
-
-<h2 id="achieving-stability-in-a-vm-environment">Achieving Stability in a VM Environment</h2>
-
-<p>For testing, demonstration, and even operation uses, Accumulo is often
-installed and run in a virtual machine (VM) environment. The majority of
-long-term operational uses of Accumulo are on bare-metal cluster. However, the
-core design of Accumulo and its dependencies do not preclude running stably for
-long periods within a VM. Many of Accumulo’s operational robustness features to
-handle failures like periodic network partitioning in a large cluster carry
-over well to VM environments. This guide covers general recommendations for
-maximizing stability in a VM environment, including some of the common failure
-modes that are more common when running in VMs.</p>
-
-<h3 id="known-failure-modes-setup-and-troubleshooting">Known failure modes: Setup and Troubleshooting</h3>
-
-<p>In addition to the general failure modes of running Accumulo, VMs can introduce a
-couple of environmental challenges that can affect process stability. Clock
-drift is something that is more common in VMs, especially when VMs are
-suspended and resumed. Clock drift can cause Accumulo servers to assume that
-they have lost connectivity to the other Accumulo processes and/or lose their
-locks in Zookeeper. VM environments also frequently have constrained resources,
-such as CPU, RAM, network, and disk throughput and capacity. Accumulo generally
-deals well with constrained resources from a stability perspective (optimizing
-performance will require additional tuning, which is not covered in this
-section), however there are some limits.</p>
-
-<h4 id="physical-memory">Physical Memory</h4>
-
-<p>One of those limits has to do with the Linux out of memory killer. A common
-failure mode in VM environments (and in some bare metal installations) is when
-the Linux out of memory killer decides to kill processes in order to avoid a
-kernel panic when provisioning a memory page. This often happens in VMs due to
-the large number of processes that must run in a small memory footprint. In
-addition to the Linux core processes, a single-node Accumulo setup requires a
-Hadoop Namenode, a Hadoop Secondary Namenode a Hadoop Datanode, a Zookeeper
-server, an Accumulo Master, an Accumulo GC and an Accumulo TabletServer.
-Typical setups also include an Accumulo Monitor, an Accumulo Tracer, a Hadoop
-ResourceManager, a Hadoop NodeManager, provisioning software, and client
-applications. Between all of these processes, it is not uncommon to
-over-subscribe the available RAM in a VM. We recommend setting up VMs without
-swap enabled, so rather than performance grinding to a halt when physical
-memory is exhausted the kernel will randomly* select processes to kill in order
-to free up memory.</p>
-
-<p>Calculating the maximum possible memory usage is essential in creating a stable
-Accumulo VM setup. Safely engineering memory allocation for stability is a
-matter of then bringing the calculated maximum memory usage under the physical
-memory by a healthy margin. The margin is to account for operating system-level
-operations, such as managing process, maintaining virtual memory pages, and
-file system caching. When the java out-of-memory killer finds your process, you
-will probably only see evidence of that in /var/log/messages. Out-of-memory
-process kills do not show up in Accumulo or Hadoop logs.</p>
-
-<p>To calculate the max memory usage of all java virtual machine (JVM) processes
-add the maximum heap size (often limited by a -Xmx… argument, such as in
-accumulo-site.xml) and the off-heap memory usage. Off-heap memory usage
-includes the following:</p>
-
-<ul>
- <li>“Permanent Space”, where the JVM stores Classes, Methods, and other code elements. This can be limited by a JVM flag such as <code class="highlighter-rouge">-XX:MaxPermSize:100m</code>, and is typically tens of megabytes.</li>
- <li>Code generation space, where the JVM stores just-in-time compiled code. This is typically small enough to ignore</li>
- <li>Socket buffers, where the JVM stores send and receive buffers for each socket.</li>
- <li>Thread stacks, where the JVM allocates memory to manage each thread.</li>
- <li>Direct memory space and JNI code, where applications can allocate memory outside of the JVM-managed space. For Accumulo, this includes the native in-memory maps that are allocated with the memory.maps.max parameter in accumulo-site.xml.</li>
- <li>Garbage collection space, where the JVM stores information used for garbage collection.</li>
-</ul>
-
-<p>You can assume that each Hadoop and Accumulo process will use ~100-150MB for
-Off-heap memory, plus the in-memory map of the Accumulo TServer process. A
-simple calculation for physical memory requirements follows:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code> Physical memory needed
- = (per-process off-heap memory) + (heap memory) + (other processes) + (margin)
-
<TRUNCATED>
[02/10] accumulo-website git commit: Split up long administration and
troubleshooting overview pages
Posted by mw...@apache.org.
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/036341f2/_docs-unreleased/administration/overview.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/administration/overview.md b/_docs-unreleased/administration/overview.md
deleted file mode 100644
index 39ff163..0000000
--- a/_docs-unreleased/administration/overview.md
+++ /dev/null
@@ -1,1151 +0,0 @@
----
-title: Overview
-category: administration
-order: 1
----
-
-## Hardware
-
-Because we are running essentially two or three systems simultaneously layered
-across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to
-consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have
-at least one core and 2 - 4 GB each.
-
-One core running HDFS can typically keep 2 to 4 disks busy, so each machine may
-typically have as little as 2 x 300GB disks and as much as 4 x 1TB or 2TB disks.
-
-It is possible to do with less than this, such as with 1u servers with 2 cores and 4GB
-each, but in this case it is recommended to only run up to two processes per
-machine -- i.e. DataNode and TabletServer or DataNode and MapReduce worker but
-not all three. The constraint here is having enough available heap space for all the
-processes on a machine.
-
-## Network
-
-Accumulo communicates via remote procedure calls over TCP/IP for both passing
-data and control messages. In addition, Accumulo uses HDFS clients to
-communicate with HDFS. To achieve good ingest and query performance, sufficient
-network bandwidth must be available between any two machines.
-
-In addition to needing access to ports associated with HDFS and ZooKeeper, Accumulo will
-use the following default ports. Please make sure that they are open, or change
-their value in accumulo-site.xml.
-
-|Port | Description | Property Name
-|-----|-------------|--------------
-|4445 | Shutdown Port (Accumulo MiniCluster) | n/a
-|4560 | Accumulo monitor (for centralized log display) | monitor.port.log4j
-|9995 | Accumulo HTTP monitor | monitor.port.client
-|9997 | Tablet Server | tserver.port.client
-|9998 | Accumulo GC | gc.port.client
-|9999 | Master Server | master.port.client
-|12234 | Accumulo Tracer | trace.port.client
-|42424 | Accumulo Proxy Server | n/a
-|10001 | Master Replication service | master.replication.coordinator.port
-|10002 | TabletServer Replication service | replication.receipt.service.port
-
-In addition, the user can provide `0` and an ephemeral port will be chosen instead. This
-ephemeral port is likely to be unique and not already bound. Thus, configuring ports to
-use `0` instead of an explicit value, should, in most cases, work around any issues of
-running multiple distinct Accumulo instances (or any other process which tries to use the
-same default ports) on the same hardware. Finally, the *.port.client properties will work
-with the port range syntax (M-N) allowing the user to specify a range of ports for the
-service to attempt to bind. The ports in the range will be tried in a 1-up manner starting
-at the low end of the range to, and including, the high end of the range.
-
-## Installation
-
-Download a binary distribution of Accumulo and install it to a directory on a disk with
-sufficient space:
-
- cd <install directory>
- tar xzf accumulo-X.Y.Z-bin.tar.gz # Replace 'X.Y.Z' with your Accumulo version
- cd accumulo-X.Y.Z
-
-Repeat this step on each machine in your cluster. Typically, the same `<install directory>`
-is chosen for all machines in the cluster.
-
-There are four scripts in the `bin/` directory that are used to manage Accumulo:
-
-1. `accumulo` - Runs Accumulo command-line tools and starts Accumulo processes
-2. `accumulo-service` - Runs Accumulo processes as services
-3. `accumulo-cluster` - Manages Accumulo cluster on a single node or several nodes
-4. `accumulo-util` - Accumulo utilities for creating configuration, native libraries, etc.
-
-These scripts will be used in the remaining instructions to configure and run Accumulo.
-
-## Dependencies
-
-Accumulo requires HDFS and ZooKeeper to be configured and running
-before starting. Password-less SSH should be configured between at least the
-Accumulo master and TabletServer machines. It is also a good idea to run Network
-Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of
-sync, which can cause problems with automatically timestamped data.
-
-## Configuration
-
-The Accumulo tarball contains a `conf/` directory where Accumulo looks for configuration. If you
-installed Accumulo using downstream packaging, the `conf/` could be something else like
-`/etc/accumulo/`.
-
-Before starting Accumulo, the configuration files `accumulo-env.sh` and `accumulo-site.xml` must
-exist in `conf/` and be properly configured. If you are using `accumulo-cluster` to launch
-a cluster, the `conf/` directory must also contain hosts file for Accumulo services (i.e `gc`,
-`masters`, `monitor`, `tservers`, `tracers`). You can either create these files manually or run
-`accumulo-cluster create-config`.
-
-Logging is configured in `accumulo-env.sh` to use three log4j configuration files in `conf/`. The
-file used depends on the Accumulo command or service being run. Logging for most Accumulo services
-(i.e Master, TabletServer, Garbage Collector) is configured by `log4j-service.properties` except for
-the Monitor which is configured by `log4j-monitor.properties`. All Accumulo commands (i.e `init`,
-`shell`, etc) are configured by `log4j.properties`.
-
-### Configure accumulo-env.sh
-
-Accumulo needs to know where to find the software it depends on. Edit accumulo-env.sh
-and specify the following:
-
-1. Enter the location of Hadoop for `$HADOOP_PREFIX`
-2. Enter the location of ZooKeeper for `$ZOOKEEPER_HOME`
-3. Optionally, choose a different location for Accumulo logs using `$ACCUMULO_LOG_DIR`
-
-Accumulo uses `HADOOP_PREFIX` and `ZOOKEEPER_HOME` to locate Hadoop and Zookeeper jars
-and add them the `CLASSPATH` variable. If you are running a vendor-specific release of Hadoop
-or Zookeeper, you may need to change how your `CLASSPATH` is built in `accumulo-env.sh`. If
-Accumulo has problems later on finding jars, run `accumulo classpath -d` to debug and print
-Accumulo's classpath.
-
-You may want to change the default memory settings for Accumulo's TabletServer which are
-by set in the `JAVA_OPTS` settings for 'tservers' in `accumulo-env.sh`. Note the
-syntax is that of the Java JVM command line options. This value should be less than the
-physical memory of the machines running TabletServers.
-
-There are similar options for the master's memory usage and the garbage collector
-process. Reduce these if they exceed the physical RAM of your hardware and
-increase them, within the bounds of the physical RAM, if a process fails because of
-insufficient memory.
-
-Note that you will be specifying the Java heap space in accumulo-env.sh. You should
-make sure that the total heap space used for the Accumulo tserver and the Hadoop
-DataNode and TaskTracker is less than the available memory on each worker node in
-the cluster. On large clusters, it is recommended that the Accumulo master, Hadoop
-NameNode, secondary NameNode, and Hadoop JobTracker all be run on separate
-machines to allow them to use more heap space. If you are running these on the
-same machine on a small cluster, likewise make sure their heap space settings fit
-within the available memory.
-
-### Native Map
-
-The tablet server uses a data structure called a MemTable to store sorted key/value
-pairs in memory when they are first received from the client. When a minor compaction
-occurs, this data structure is written to HDFS. The MemTable will default to using
-memory in the JVM but a JNI version, called the native map, can be used to significantly
-speed up performance by utilizing the memory space of the native operating system. The
-native map also avoids the performance implications brought on by garbage collection
-in the JVM by causing it to pause much less frequently.
-
-#### Building
-
-32-bit and 64-bit Linux and Mac OS X versions of the native map can be built by executing
-`accumulo-util build-native`. If your system's default compiler options are insufficient,
-you can add additional compiler options to the command line, such as options for the
-architecture. These will be passed to the Makefile in the environment variable `USERFLAGS`.
-
-Examples:
-
- accumulo-util build-native
- accumulo-util build-native -m32
-
-After building the native map from the source, you will find the artifact in
-`lib/native`. Upon starting up, the tablet server will look
-in this directory for the map library. If the file is renamed or moved from its
-target directory, the tablet server may not be able to find it. The system can
-also locate the native maps shared library by setting `LD_LIBRARY_PATH`
-(or `DYLD_LIBRARY_PATH` on Mac OS X) in `accumulo-env.sh`.
-
-#### Native Maps Configuration
-
-As mentioned, Accumulo will use the native libraries if they are found in the expected
-location and `tserver.memory.maps.native.enabled` is set to `true` (which is the default).
-Using the native maps over JVM Maps nets a noticeable improvement in ingest rates; however,
-certain configuration variables are important to modify when increasing the size of the
-native map.
-
-To adjust the size of the native map, increase the value of `tserver.memory.maps.max`.
-By default, the maximum size of the native map is 1GB. When increasing this value, it is
-also important to adjust the values of `table.compaction.minor.logs.threshold` and
-`tserver.walog.max.size`. `table.compaction.minor.logs.threshold` is the maximum
-number of write-ahead log files that a tablet can reference before they will be automatically
-minor compacted. `tserver.walog.max.size` is the maximum size of a write-ahead log.
-
-The maximum size of the native maps for a server should be less than the product
-of the write-ahead log maximum size and minor compaction threshold for log files:
-
-`$table.compaction.minor.logs.threshold * $tserver.walog.max.size >= $tserver.memory.maps.max`
-
-This formula ensures that minor compactions won't be automatically triggered before the native
-maps can be completely saturated.
-
-Subsequently, when increasing the size of the write-ahead logs, it can also be important
-to increase the HDFS block size that Accumulo uses when creating the files for the write-ahead log.
-This is controlled via `tserver.wal.blocksize`. A basic recommendation is that when
-`tserver.walog.max.size` is larger than 2GB in size, set `tserver.wal.blocksize` to 2GB.
-Increasing the block size to a value larger than 2GB can result in decreased write
-performance to the write-ahead log file which will slow ingest.
-
-### Cluster Specification
-
-If you are using `accumulo-cluster` to start a cluster, configure the following on the
-machine that will serve as the Accumulo master:
-
-1. Write the IP address or domain name of the Accumulo Master to the `conf/masters` file.
-2. Write the IP addresses or domain name of the machines that will be TabletServers in `conf/tservers`, one per line.
-
-Note that if using domain names rather than IP addresses, DNS must be configured
-properly for all machines participating in the cluster. DNS can be a confusing source
-of errors.
-
-### Configure accumulo-site.xml
-
-Specify appropriate values for the following settings in `accumulo-site.xml`:
-
-```xml
-<property>
- <name>instance.zookeeper.host</name>
- <value>zooserver-one:2181,zooserver-two:2181</value>
- <description>list of zookeeper servers</description>
-</property>
-```
-
-This enables Accumulo to find ZooKeeper. Accumulo uses ZooKeeper to coordinate
-settings between processes and helps finalize TabletServer failure.
-
-```xml
-<property>
- <name>instance.secret</name>
- <value>DEFAULT</value>
-</property>
-```
-
-The instance needs a secret to enable secure communication between servers. Configure your
-secret and make sure that the `accumulo-site.xml` file is not readable to other users.
-For alternatives to storing the `instance.secret` in plaintext, please read the
-`Sensitive Configuration Values` section.
-
-Some settings can be modified via the Accumulo shell and take effect immediately, but
-some settings require a process restart to take effect. See the [configuration management][config-mgmt]
-documentation for details.
-
-### Hostnames in configuration files
-
-Accumulo has a number of configuration files which can contain references to other hosts in your
-network. All of the "host" configuration files for Accumulo (`gc`, `masters`, `tservers`, `monitor`,
-`tracers`) as well as `instance.volumes` in accumulo-site.xml must contain some host reference.
-
-While IP address, short hostnames, or fully qualified domain names (FQDN) are all technically valid, it
-is good practice to always use FQDNs for both Accumulo and other processes in your Hadoop cluster.
-Failing to consistently use FQDNs can have unexpected consequences in how Accumulo uses the FileSystem.
-
-A common way for this problem can be observed is via applications that use Bulk Ingest. The Accumulo
-Master coordinates moving the input files to Bulk Ingest to an Accumulo-managed directory. However,
-Accumulo cannot safely move files across different Hadoop FileSystems. This is problematic because
-Accumulo also cannot make reliable assertions across what is the same FileSystem which is specified
-with different names. Naively, while 127.0.0.1:8020 might be a valid identifier for an HDFS instance,
-Accumulo identifies `localhost:8020` as a different HDFS instance than `127.0.0.1:8020`.
-
-### Deploy Configuration
-
-Copy accumulo-env.sh and accumulo-site.xml from the `conf/` directory on the master to all Accumulo
-tablet servers. The "host" configuration files files `accumulo-cluster` only need to be on servers
-where that command is run.
-
-### Sensitive Configuration Values
-
-Accumulo has a number of properties that can be specified via the accumulo-site.xml
-file which are sensitive in nature, instance.secret and trace.token.property.password
-are two common examples. Both of these properties, if compromised, have the ability
-to result in data being leaked to users who should not have access to that data.
-
-In Hadoop-2.6.0, a new CredentialProvider class was introduced which serves as a common
-implementation to abstract away the storage and retrieval of passwords from plaintext
-storage in configuration files. Any Property marked with the `Sensitive` annotation
-is a candidate for use with these CredentialProviders. For version of Hadoop which lack
-these classes, the feature will just be unavailable for use.
-
-A comma separated list of CredentialProviders can be configured using the Accumulo Property
-`general.security.credential.provider.paths`. Each configured URL will be consulted
-when the Configuration object for accumulo-site.xml is accessed.
-
-### Using a JavaKeyStoreCredentialProvider for storage
-
-One of the implementations provided in Hadoop-2.6.0 is a Java KeyStore CredentialProvider.
-Each entry in the KeyStore is the Accumulo Property key name. For example, to store the
-`instance.secret`, the following command can be used:
-
- hadoop credential create instance.secret --provider jceks://file/etc/accumulo/conf/accumulo.jceks
-
-The command will then prompt you to enter the secret to use and create a keystore in:
-
- /path/to/accumulo/conf/accumulo.jceks
-
-Then, accumulo-site.xml must be configured to use this KeyStore as a CredentialProvider:
-
-```xml
-<property>
- <name>general.security.credential.provider.paths</name>
- <value>jceks://file/path/to/accumulo/conf/accumulo.jceks</value>
-</property>
-```
-
-This configuration will then transparently extract the `instance.secret` from
-the configured KeyStore and alleviates a human readable storage of the sensitive
-property.
-
-A KeyStore can also be stored in HDFS, which will make the KeyStore readily available to
-all Accumulo servers. If the local filesystem is used, be aware that each Accumulo server
-will expect the KeyStore in the same location.
-
-### Client Configuration
-
-In version 1.6.0, Accumulo included a new type of configuration file known as a client
-configuration file. One problem with the traditional "site.xml" file that is prevalent
-through Hadoop is that it is a single file used by both clients and servers. This makes
-it very difficult to protect secrets that are only meant for the server processes while
-allowing the clients to connect to the servers.
-
-The client configuration file is a subset of the information stored in accumulo-site.xml
-meant only for consumption by clients of Accumulo. By default, Accumulo checks a number
-of locations for a client configuration by default:
-
-* `/path/to/accumulo/conf/client.conf`
-* `/etc/accumulo/client.conf`
-* `/etc/accumulo/conf/client.conf`
-* `~/.accumulo/config`
-
-These files are [Java Properties files](https://en.wikipedia.org/wiki/.properties). These files
-can currently contain information about ZooKeeper servers, RPC properties (such as SSL or SASL
-connectors), distributed tracing properties. Valid properties are defined by the [ClientProperty](https://github.com/apache/accumulo/blob/f1d0ec93d9f13ff84844b5ac81e4a7b383ced467/core/src/main/java/org/apache/accumulo/core/client/ClientConfiguration.java#L54)
-enum contained in the client API.
-
-#### Custom Table Tags
-
-Accumulo has the ability for users to add custom tags to tables. This allows
-applications to set application-level metadata about a table. These tags can be
-anything from a table description, administrator notes, date created, etc.
-This is done by naming and setting a property with a prefix `table.custom.*`.
-
-Currently, table properties are stored in ZooKeeper. This means that the number
-and size of custom properties should be restricted on the order of 10's of properties
-at most without any properties exceeding 1MB in size. ZooKeeper's performance can be
-very sensitive to an excessive number of nodes and the sizes of the nodes. Applications
-which leverage the user of custom properties should take these warnings into
-consideration. There is no enforcement of these warnings via the API.
-
-#### Configuring the ClassLoader
-
-Accumulo builds its Java classpath in `accumulo-env.sh`. After an Accumulo application has started, it will load classes from the locations
-specified in the deprecated `general.classpaths` property. Additionally, Accumulo will load classes from the locations specified in the
-`general.dynamic.classpaths` property and will monitor and reload them if they change. The reloading feature is useful during the development
-and testing of iterators as new or modified iterator classes can be deployed to Accumulo without having to restart the database.
-/
-Accumulo also has an alternate configuration for the classloader which will allow it to load classes from remote locations. This mechanism
-uses Apache Commons VFS which enables locations such as http and hdfs to be used. This alternate configuration also uses the
-`general.classpaths` property in the same manner described above. It differs in that you need to configure the
-`general.vfs.classpaths` property instead of the `general.dynamic.classpath` property. As in the default configuration, this alternate
-configuration will also monitor the vfs locations for changes and reload if necessary.
-
-The Accumulo classpath can be viewed in human readable format by running `accumulo classpath -d`.
-
-##### ClassLoader Contexts
-
-With the addition of the VFS based classloader, we introduced the notion of classloader contexts. A context is identified
-by a name and references a set of locations from which to load classes and can be specified in the accumulo-site.xml file or added
-using the `config` command in the shell. Below is an example for specify the app1 context in the accumulo-site.xml file:
-
-```xml
-<property>
- <name>general.vfs.context.classpath.app1</name>
- <value>hdfs://localhost:8020/applicationA/classpath/.*.jar,file:///opt/applicationA/lib/.*.jar</value>
- <description>Application A classpath, loads jars from HDFS and local file system</description>
-</property>
-```
-
-The default behavior follows the Java ClassLoader contract in that classes, if they exists, are loaded from the parent classloader first.
-You can override this behavior by delegating to the parent classloader after looking in this classloader first. An example of this
-configuration is:
-
-```xml
-<property>
- <name>general.vfs.context.classpath.app1.delegation=post</name>
- <value>hdfs://localhost:8020/applicationA/classpath/.*.jar,file:///opt/applicationA/lib/.*.jar</value>
- <description>Application A classpath, loads jars from HDFS and local file system</description>
-</property>
-```
-
-To use contexts in your application you can set the `table.classpath.context` on your tables or use the `setClassLoaderContext()` method on Scanner
-and BatchScanner passing in the name of the context, app1 in the example above. Setting the property on the table allows your minc, majc, and scan
-iterators to load classes from the locations defined by the context. Passing the context name to the scanners allows you to override the table setting
-to load only scan time iterators from a different location.
-
-## Initialization
-
-Accumulo must be initialized to create the structures it uses internally to locate
-data across the cluster. HDFS is required to be configured and running before
-Accumulo can be initialized.
-
-Once HDFS is started, initialization can be performed by executing
-`accumulo init` . This script will prompt for a name
-for this instance of Accumulo. The instance name is used to identify a set of tables
-and instance-specific settings. The script will then write some information into
-HDFS so Accumulo can start properly.
-
-The initialization script will prompt you to set a root password. Once Accumulo is
-initialized it can be started.
-
-## Running
-
-### Starting Accumulo
-
-Make sure Hadoop is configured on all of the machines in the cluster, including
-access to a shared HDFS instance. Make sure HDFS and ZooKeeper are running.
-Make sure ZooKeeper is configured and running on at least one machine in the
-cluster.
-Start Accumulo using `accumulo-cluster start`.
-
-To verify that Accumulo is running, check the [Accumulo monitor][monitor].
-In addition, the Shell can provide some information about the status of tables via reading the metadata tables.
-
-### Stopping Accumulo
-
-To shutdown cleanly, run `accumulo-cluster stop` and the master will orchestrate the
-shutdown of all the tablet servers. Shutdown waits for all minor compactions to finish, so it may
-take some time for particular configurations.
-
-### Adding a Tablet Server
-
-Update your `conf/tservers` file to account for the addition.
-
-Next, ssh to each of the hosts you want to add and run:
-
- accumulo-service tserver start
-
-Make sure the host in question has the new configuration, or else the tablet
-server won't start; at a minimum this needs to be on the host(s) being added,
-but in practice it's good to ensure consistent configuration across all nodes.
-
-### Decomissioning a Tablet Server
-
-If you need to take a node out of operation, you can trigger a graceful shutdown of a tablet
-server. Accumulo will automatically rebalance the tablets across the available tablet servers.
-
- accumulo admin stop <host(s)> {<host> ...}
-
-Alternatively, you can ssh to each of the hosts you want to remove and run:
-
- accumulo-service tserver stop
-
-Be sure to update your `conf/tservers` file to
-account for the removal of these hosts. Bear in mind that the monitor will not re-read the
-tservers file automatically, so it will report the decommissioned servers as down; it's
-recommended that you restart the monitor so that the node list is up to date.
-
-The steps described to decommission a node can also be used (without removal of the host
-from the `conf/tservers` file) to gracefully stop a node. This will
-ensure that the tabletserver is cleanly stopped and recovery will not need to be performed
-when the tablets are re-hosted.
-
-### Restarting process on a node
-
-Occasionally, it might be necessary to restart the processes on a specific node. In addition
-to the `accumulo-cluster` script, Accumulo has a `accumulo-service` script that
-can be use to start/stop processes on a node.
-
-#### A note on rolling restarts
-
-For sufficiently large Accumulo clusters, restarting multiple TabletServers within a short window can place significant
-load on the Master server. If slightly lower availability is acceptable, this load can be reduced by globally setting
-`table.suspend.duration` to a positive value.
-
-With `table.suspend.duration` set to, say, `5m`, Accumulo will wait
-for 5 minutes for any dead TabletServer to return before reassigning that TabletServer's responsibilities to other TabletServers.
-If the TabletServer returns to the cluster before the specified timeout has elapsed, Accumulo will assign the TabletServer
-its original responsibilities.
-
-It is important not to choose too large a value for `table.suspend.duration`, as during this time, all scans against the
-data that TabletServer had hosted will block (or time out).
-
-### Running multiple TabletServers on a single node
-
-With very powerful nodes, it may be beneficial to run more than one TabletServer on a given
-node. This decision should be made carefully and with much deliberation as Accumulo is designed
-to be able to scale to using 10's of GB of RAM and 10's of CPU cores.
-
-Accumulo TabletServers bind certain ports on the host to accommodate remote procedure calls to/from
-other nodes. Running more than one TabletServer on a host requires that you set the environment variable
-`ACCUMULO_SERVICE_INSTANCE` to an instance number (i.e 1, 2) for each instance that is started. Also, set
-these properties in `accumulo-site.xml`:
-
-```xml
- <property>
- <name>tserver.port.search</name>
- <value>true</value>
- </property>
- <property>
- <name>replication.receipt.service.port</name>
- <value>0</value>
- </property>
-```
-
-## Monitoring
-
-### Accumulo Monitor
-
-The Accumulo Monitor provides an interface for monitoring the status and health of
-Accumulo components. The Accumulo Monitor provides a web UI for accessing this information at
-`http://_monitorhost_:9995/`.
-
-Things highlighted in yellow may be in need of attention.
-If anything is highlighted in red on the monitor page, it is something that definitely needs attention.
-
-The Overview page contains some summary information about the Accumulo instance, including the version, instance name, and instance ID.
-There is a table labeled Accumulo Master with current status, a table listing the active Zookeeper servers, and graphs displaying various metrics over time.
-These include ingest and scan performance and other useful measurements.
-
-The Master Server, Tablet Servers, and Tables pages display metrics grouped in different ways (e.g. by tablet server or by table).
-Metrics typically include number of entries (key/value pairs), ingest and query rates.
-The number of running scans, major and minor compactions are in the form _number_running_ (_number_queued_).
-Another important metric is hold time, which is the amount of time a tablet has been waiting but unable to flush its memory in a minor compaction.
-
-The Server Activity page graphically displays tablet server status, with each server represented as a circle or square.
-Different metrics may be assigned to the nodes' color and speed of oscillation.
-The Overall Avg metric is only used on the Server Activity page, and represents the average of all the other metrics (after normalization).
-Similarly, the Overall Max metric picks the metric with the maximum normalized value.
-
-The Garbage Collector page displays a list of garbage collection cycles, the number of files found of each type (including deletion candidates in use and files actually deleted), and the length of the deletion cycle.
-The Traces page displays data for recent traces performed (see the following section for information on [tracing][tracing]).
-The Recent Logs page displays warning and error logs forwarded to the monitor from all Accumulo processes.
-Also, the XML and JSON links provide metrics in XML and JSON formats, respectively.
-
-### SSL
-
-SSL may be enabled for the monitor page by setting the following properties in the `accumulo-site.xml` file:
-
- monitor.ssl.keyStore
- monitor.ssl.keyStorePassword
- monitor.ssl.trustStore
- monitor.ssl.trustStorePassword
-
-If the Accumulo conf directory has been configured (in particular the `accumulo-env.sh` file must be set up), the
-`accumulo-util gen-monitor-cert` command can be used to create the keystore and truststore files with random passwords. The command
-will print out the properties that need to be added to the `accumulo-site.xml` file. The stores can also be generated manually with the
-Java `keytool` command, whose usage can be seen in the `accumulo-util` script.
-
-If desired, the SSL ciphers allowed for connections can be controlled via the following properties in `accumulo-site.xml`:
-
- monitor.ssl.include.ciphers
- monitor.ssl.exclude.ciphers
-
-If SSL is enabled, the monitor URL can only be accessed via https.
-This also allows you to access the Accumulo shell through the monitor page.
-The left navigation bar will have a new link to Shell.
-An Accumulo user name and password must be entered for access to the shell.
-
-## Metrics
-
-Accumulo can expose metrics through a legacy metrics library and using the Hadoop Metrics2 library.
-
-### Legacy Metrics
-
-Accumulo has a legacy metrics library that can be exposes metrics using JMX endpoints or file-based logging. These metrics can
-be enabled by setting `general.legacy.metrics` to `true` in `accumulo-site.xml` and placing the `accumulo-metrics.xml`
-configuration file on the classpath (which is typically done by placing the file in the `conf/` directory). A template for
-`accumulo-metrics.xml` can be found in `conf/templates` of the Accumulo tarball.
-
-### Hadoop Metrics2
-
-Hadoop Metrics2 is a library which allows for routing of metrics generated by registered MetricsSources to
-configured MetricsSinks. Examples of sinks that are implemented by Hadoop include file-based logging, Graphite and Ganglia.
-All metric sources are exposed via JMX when using Metrics2.
-
-Metrics2 is configured by examining the classpath for a file that matches `hadoop-metrics2*.properties`. The Accumulo tarball
-contains an example `hadoop-metrics2-accumulo.properties` file in `conf/templates` which can be copied to `conf/` to place
-on classpath. This file is used to enable file, Graphite or Ganglia sinks (some minimal configuration required for Graphite
-and Ganglia). Because the Hadoop configuration is also on the Accumulo classpath, be sure that you do not have multiple
-Metrics2 configuration files. It is recommended to consolidate metrics in a single properties file in a central location to
-remove ambiguity. The contents of `hadoop-metrics2-accumulo.properties` can be added to a central `hadoop-metrics2.properties`
-in `$HADOOP_CONF_DIR`.
-
-As a note for configuring the file sink, the provided path should be absolute. A relative path or file name will be created relative
-to the directory in which the Accumulo process was started. External tools, such as logrotate, can be used to prevent these files
-from growing without bound.
-
-Each server process should have log messages from the Metrics2 library about the sinks that were created. Be sure to check
-the Accumulo processes log files when debugging missing metrics output.
-
-For additional information on configuring Metrics2, visit the [Javadoc page for Metrics2](https://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html).
-
-## Tracing
-
-It can be difficult to determine why some operations are taking longer
-than expected. For example, you may be looking up items with very low
-latency, but sometimes the lookups take much longer. Determining the
-cause of the delay is difficult because the system is distributed, and
-the typical lookup is fast.
-
-Accumulo has been instrumented to record the time that various
-operations take when tracing is turned on. The fact that tracing is
-enabled follows all the requests made on behalf of the user throughout
-the distributed infrastructure of accumulo, and across all threads of
-execution.
-
-These time spans will be inserted into the `trace` table in
-Accumulo. You can browse recent traces from the Accumulo monitor
-page. You can also read the `trace` table directly like any
-other table.
-
-The design of Accumulo's distributed tracing follows that of [Google's Dapper](http://research.google.com/pubs/pub36356.html).
-
-### Tracers
-
-To collect traces, Accumulo needs at least one tracer server running. If you are using `accumulo-cluster` to start your cluster,
-configure your server in `conf/tracers`. The server collects traces from clients and writes them to the `trace` table. The Accumulo
-user that the tracer connects to Accumulo with can be configured with the following properties (see the [configuration management][config-mgmt]
-page for setting Accumulo server properties)
-
- trace.user
- trace.token.property.password
-
-Other tracer configuration properties include
-
- trace.port.client - port tracer listens on
- trace.table - table tracer writes to
- trace.zookeeper.path - zookeeper path where tracers register
-
-The zookeeper path is configured to /tracers by default. If
-multiple Accumulo instances are sharing the same ZooKeeper
-quorum, take care to configure Accumulo with unique values for
-this property.
-
-### Configuring Tracing
-
-Traces are collected via SpanReceivers. The default SpanReceiver
-configured is org.apache.accumulo.core.trace.ZooTraceClient, which
-sends spans to an Accumulo Tracer process, as discussed in the
-previous section. This default can be changed to a different span
-receiver, or additional span receivers can be added in a
-comma-separated list, by modifying the property
-
- trace.span.receivers
-
-Individual span receivers may require their own configuration
-parameters, which are grouped under the trace.span.receiver.*
-prefix. ZooTraceClient uses the following properties. The first
-three properties are populated from other Accumulo properties,
-while the remaining ones should be prefixed with
-trace.span.receiver. when set in the Accumulo configuration.
-
- tracer.zookeeper.host - populated from instance.zookeepers
- tracer.zookeeper.timeout - populated from instance.zookeeper.timeout
- tracer.zookeeper.path - populated from trace.zookeeper.path
- tracer.send.timer.millis - timer for flushing send queue (in ms, default 1000)
- tracer.queue.size - max queue size (default 5000)
- tracer.span.min.ms - minimum span length to store (in ms, default 1)
-
-Note that to configure an Accumulo client for tracing, including
-the Accumulo shell, the client configuration must be given the same
-trace.span.receivers, trace.span.receiver.*, and trace.zookeeper.path
-properties as the servers have.
-
-Hadoop can also be configured to send traces to Accumulo, as of
-Hadoop 2.6.0, by setting properties in Hadoop's core-site.xml
-file. Instead of using the trace.span.receiver.* prefix, Hadoop
-uses hadoop.htrace.*. The Hadoop configuration does not have
-access to Accumulo's properties, so the
-hadoop.htrace.tracer.zookeeper.host property must be specified.
-The zookeeper timeout defaults to 30000 (30 seconds), and the
-zookeeper path defaults to /tracers. An example of configuring
-Hadoop to send traces to ZooTraceClient is
-
-```xml
- <property>
- <name>hadoop.htrace.spanreceiver.classes</name>
- <value>org.apache.accumulo.core.trace.ZooTraceClient</value>
- </property>
- <property>
- <name>hadoop.htrace.tracer.zookeeper.host</name>
- <value>zookeeperHost:2181</value>
- </property>
- <property>
- <name>hadoop.htrace.tracer.zookeeper.path</name>
- <value>/tracers</value>
- </property>
- <property>
- <name>hadoop.htrace.tracer.span.min.ms</name>
- <value>1</value>
- </property>
-```
-
-The accumulo-core, accumulo-tracer, accumulo-fate and libthrift
-jars must also be placed on Hadoop's classpath.
-
-##### Adding additional SpanReceivers
-
-[Zipkin](https://github.com/openzipkin/zipkin) has a SpanReceiver supported by HTrace and popularized by Twitter
-that users looking for a more graphical trace display may opt to use.
-The following steps configure Accumulo to use `org.apache.htrace.impl.ZipkinSpanReceiver`
-in addition to the Accumulo's default ZooTraceClient, and they serve as a template
-for adding any SpanReceiver to Accumulo:
-
-1. Add the Jar containing the ZipkinSpanReceiver class file to the
-`lib/` directory. It is critical that the Jar is placed in
-`lib/` and NOT in `lib/ext/` so that the new SpanReceiver class
-is visible to the same class loader of htrace-core.
-
-2. Add the following to `accumulo-site.xml`:
-
- <property>
- <name>trace.span.receivers</name>
- <value>org.apache.accumulo.tracer.ZooTraceClient,org.apache.htrace.impl.ZipkinSpanReceiver</value>
- </property>
-
-3. Restart your Accumulo tablet servers.
-
-In order to use ZipkinSpanReceiver from a client as well as the Accumulo server,
-
-1. Ensure your client can see the ZipkinSpanReceiver class at runtime. For Maven projects,
-this is easily done by adding to your client's pom.xml (taking care to specify a good version)
-
- <dependency>
- <groupId>org.apache.htrace</groupId>
- <artifactId>htrace-zipkin</artifactId>
- <version>3.1.0-incubating</version>
- <scope>runtime</scope>
- </dependency>
-
-2. Add the following to your client configuration.
-
- trace.span.receivers=org.apache.accumulo.tracer.ZooTraceClient,org.apache.htrace.impl.ZipkinSpanReceiver
-
-3. Instrument your client as in the next section.
-
-Your SpanReceiver may require additional properties, and if so these should likewise
-be placed in the ClientConfiguration (if applicable) and Accumulo's `accumulo-site.xml`.
-Two such properties for ZipkinSpanReceiver, listed with their default values, are
-
-```xml
- <property>
- <name>trace.span.receiver.zipkin.collector-hostname</name>
- <value>localhost</value>
- </property>
- <property>
- <name>trace.span.receiver.zipkin.collector-port</name>
- <value>9410</value>
- </property>
-```
-
-#### Instrumenting a Client
-
-Tracing can be used to measure a client operation, such as a scan, as
-the operation traverses the distributed system. To enable tracing for
-your application call
-
-```java
-import org.apache.accumulo.core.trace.DistributedTrace;
-...
-DistributedTrace.enable(hostname, "myApplication");
-// do some tracing
-...
-DistributedTrace.disable();
-```
-
-Once tracing has been enabled, a client can wrap an operation in a trace.
-
-```java
-import org.apache.htrace.Sampler;
-import org.apache.htrace.Trace;
-import org.apache.htrace.TraceScope;
-...
-TraceScope scope = Trace.startSpan("Client Scan", Sampler.ALWAYS);
-BatchScanner scanner = conn.createBatchScanner(...);
-// Configure your scanner
-for (Entry entry : scanner) {
-}
-scope.close();
-```
-
-The user can create additional Spans within a Trace.
-
-The sampler (such as `Sampler.ALWAYS`) for the trace should only be specified with a top-level span,
-and subsequent spans will be collected depending on whether that first span was sampled.
-Don't forget to specify a Sampler at the top-level span
-because the default Sampler only samples when part of a pre-existing trace,
-which will never occur in a client that never specifies a Sampler.
-
-```java
-TraceScope scope = Trace.startSpan("Client Update", Sampler.ALWAYS);
-...
-TraceScope readScope = Trace.startSpan("Read");
-...
-readScope.close();
-...
-TraceScope writeScope = Trace.startSpan("Write");
-...
-writeScope.close();
-scope.close();
-```
-
-Like Dapper, Accumulo tracing supports user defined annotations to associate additional data with a Trace.
-Checking whether currently tracing is necessary when using a sampler other than Sampler.ALWAYS.
-
-```java
-...
-int numberOfEntriesRead = 0;
-TraceScope readScope = Trace.startSpan("Read");
-// Do the read, update the counter
-...
-if (Trace.isTracing)
- readScope.getSpan().addKVAnnotation("Number of Entries Read".getBytes(StandardCharsets.UTF_8),
- String.valueOf(numberOfEntriesRead).getBytes(StandardCharsets.UTF_8));
-```
-
-It is also possible to add timeline annotations to your spans.
-This associates a string with a given timestamp between the start and stop times for a span.
-
-```java
-...
-writeScope.getSpan().addTimelineAnnotation("Initiating Flush");
-```
-
-Some client operations may have a high volume within your
-application. As such, you may wish to only sample a percentage of
-operations for tracing. As seen below, the CountSampler can be used to
-help enable tracing for 1-in-1000 operations
-
-```java
-import org.apache.htrace.impl.CountSampler;
-...
-Sampler sampler = new CountSampler(HTraceConfiguration.fromMap(
- Collections.singletonMap(CountSampler.SAMPLER_FREQUENCY_CONF_KEY, "1000")));
-...
-TraceScope readScope = Trace.startSpan("Read", sampler);
-...
-readScope.close();
-```
-
-Remember to close all spans and disable tracing when finished.
-
-```java
-DistributedTrace.disable();
-```
-
-### Viewing Collected Traces
-
-To view collected traces, use the "Recent Traces" link on the Monitor
-UI. You can also programmatically access and print traces using the
-`TraceDump` class.
-
-#### Trace Table Format
-
-This section is for developers looking to use data recorded in the trace table
-directly, above and beyond the default services of the Accumulo monitor.
-Please note the trace table format and its supporting classes
-are not in the public API and may be subject to change in future versions.
-
-Each span received by a tracer's ZooTraceClient is recorded in the trace table
-in the form of three entries: span entries, index entries, and start time entries.
-Span and start time entries record full span information,
-whereas index entries provide indexing into span information
-useful for quickly finding spans by type or start time.
-
-Each entry is illustrated by a description and sample of data.
-In the description, a token in quotes is a String literal,
-whereas other other tokens are span variables.
-Parentheses group parts together, to distinguish colon characters inside the
-column family or qualifier from the colon that separates column family and qualifier.
-We use the format `row columnFamily:columnQualifier columnVisibility value`
-(omitting timestamp which records the time an entry is written to the trace table).
-
-Span entries take the following form:
-
- traceId "span":(parentSpanId:spanId) [] spanBinaryEncoding
- 63b318de80de96d1 span:4b8f66077df89de1:3778c6739afe4e1 [] %18;%09;...
-
-The parentSpanId is "" for the root span of a trace.
-The spanBinaryEncoding is a compact Apache Thrift encoding of the original Span object.
-This allows clients (and the Accumulo monitor) to recover all the details of the original Span
-at a later time, by scanning the trace table and decoding the value of span entries
-via `TraceFormatter.getRemoteSpan(entry)`.
-
-The trace table has a formatter class by default (org.apache.accumulo.tracer.TraceFormatter)
-that changes how span entries appear from the Accumulo shell.
-Normal scans to the trace table do not use this formatter representation;
-it exists only to make span entries easier to view inside the Accumulo shell.
-
-Index entries take the following form:
-
- "idx":service:startTime description:sender [] traceId:elapsedTime
- idx:tserver:14f3828f58b startScan:localhost [] 63b318de80de96d1:1
-
-The service and sender are set by the first call of each Accumulo process
-(and instrumented client processes) to `DistributedTrace.enable(...)`
-(the sender is autodetected if not specified).
-The description is specified in each span.
-Start time and the elapsed time (start - stop, 1 millisecond in the example above)
-are recorded in milliseconds as long values serialized to a string in hex.
-
-Start time entries take the following form:
-
- "start":startTime "id":traceId [] spanBinaryEncoding
- start:14f3828a351 id:63b318de80de96d1 [] %18;%09;...
-
-The following classes may be run while Accumulo is running to provide insight into trace statistics. These require
-accumulo-trace-VERSION.jar to be provided on the Accumulo classpath (`lib/ext` is fine).
-
- $ accumulo org.apache.accumulo.tracer.TraceTableStats -u username -p password -i instancename
- $ accumulo org.apache.accumulo.tracer.TraceDump -u username -p password -i instancename -r
-
-### Tracing from the Shell
-You can enable tracing for operations run from the shell by using the
-`trace on` and `trace off` commands.
-
-```
-root@test test> trace on
-
-root@test test> scan
-a b:c [] d
-
-root@test test> trace off
-Waiting for trace information
-Waiting for trace information
-Trace started at 2013/08/26 13:24:08.332
-Time Start Service@Location Name
- 3628+0 shell@localhost shell:root
- 8+1690 shell@localhost scan
- 7+1691 shell@localhost scan:location
- 6+1692 tserver@localhost startScan
- 5+1692 tserver@localhost tablet read ahead 6
-```
-
-## Logging
-
-Accumulo processes each write to a set of log files. By default, these logs are found at directory
-set by `ACCUMULO_LOG_DIR` in `accumulo-env.sh`.
-
-## Recovery
-
-In the event of TabletServer failure or error on shutting Accumulo down, some
-mutations may not have been minor compacted to HDFS properly. In this case,
-Accumulo will automatically reapply such mutations from the write-ahead log
-either when the tablets from the failed server are reassigned by the Master (in the
-case of a single TabletServer failure) or the next time Accumulo starts (in the event of
-failure during shutdown).
-
-Recovery is performed by asking a tablet server to sort the logs so that tablets can easily find their missing
-updates. The sort status of each file is displayed on
-Accumulo monitor status page. Once the recovery is complete any
-tablets involved should return to an ``online'' state. Until then those tablets will be
-unavailable to clients.
-
-The Accumulo client library is configured to retry failed mutations and in many
-cases clients will be able to continue processing after the recovery process without
-throwing an exception.
-
-## Migrating Accumulo from non-HA Namenode to HA Namenode
-
-The following steps will allow a non-HA instance to be migrated to an HA instance. Consider an HDFS URL
-`hdfs://namenode.example.com:8020` which is going to be moved to `hdfs://nameservice1`.
-
-Before moving HDFS over to the HA namenode, use `accumulo admin volumes` to confirm
-that the only volume displayed is the volume from the current namenode's HDFS URL.
-
- Listing volumes referenced in zookeeper
- Volume : hdfs://namenode.example.com:8020/accumulo
-
- Listing volumes referenced in accumulo.root tablets section
- Volume : hdfs://namenode.example.com:8020/accumulo
- Listing volumes referenced in accumulo.root deletes section (volume replacement occurrs at deletion time)
-
- Listing volumes referenced in accumulo.metadata tablets section
- Volume : hdfs://namenode.example.com:8020/accumulo
-
- Listing volumes referenced in accumulo.metadata deletes section (volume replacement occurrs at deletion time)
-
-After verifying the current volume is correct, shut down the cluster and transition HDFS to the HA nameservice.
-
-Edit `accumulo-site.xml` to notify accumulo that a volume is being replaced. First,
-add the new nameservice volume to the `instance.volumes` property. Next, add the
-`instance.volumes.replacements` property in the form of `old new`. It's important to not include
-the volume that's being replaced in `instance.volumes`, otherwise it's possible accumulo could continue
-to write to the volume.
-
-```xml
-<!-- instance.dfs.uri and instance.dfs.dir should not be set-->
-<property>
- <name>instance.volumes</name>
- <value>hdfs://nameservice1/accumulo</value>
-</property>
-<property>
- <name>instance.volumes.replacements</name>
- <value>hdfs://namenode.example.com:8020/accumulo hdfs://nameservice1/accumulo</value>
-</property>
-```
-
-Run `accumulo init --add-volumes` and start up the accumulo cluster. Verify that the
-new nameservice volume shows up with `accumulo admin volumes`.
-
- Listing volumes referenced in zookeeper
- Volume : hdfs://namenode.example.com:8020/accumulo
- Volume : hdfs://nameservice1/accumulo
-
- Listing volumes referenced in accumulo.root tablets section
- Volume : hdfs://namenode.example.com:8020/accumulo
- Volume : hdfs://nameservice1/accumulo
- Listing volumes referenced in accumulo.root deletes section (volume replacement occurrs at deletion time)
-
- Listing volumes referenced in accumulo.metadata tablets section
- Volume : hdfs://namenode.example.com:8020/accumulo
- Volume : hdfs://nameservice1/accumulo
- Listing volumes referenced in accumulo.metadata deletes section (volume replacement occurrs at deletion time)
-
-Some erroneous GarbageCollector messages may still be seen for a small period while data is transitioning to
-the new volumes. This is expected and can usually be ignored.
-
-## Achieving Stability in a VM Environment
-
-For testing, demonstration, and even operation uses, Accumulo is often
-installed and run in a virtual machine (VM) environment. The majority of
-long-term operational uses of Accumulo are on bare-metal cluster. However, the
-core design of Accumulo and its dependencies do not preclude running stably for
-long periods within a VM. Many of Accumulo’s operational robustness features to
-handle failures like periodic network partitioning in a large cluster carry
-over well to VM environments. This guide covers general recommendations for
-maximizing stability in a VM environment, including some of the common failure
-modes that are more common when running in VMs.
-
-### Known failure modes: Setup and Troubleshooting
-
-In addition to the general failure modes of running Accumulo, VMs can introduce a
-couple of environmental challenges that can affect process stability. Clock
-drift is something that is more common in VMs, especially when VMs are
-suspended and resumed. Clock drift can cause Accumulo servers to assume that
-they have lost connectivity to the other Accumulo processes and/or lose their
-locks in Zookeeper. VM environments also frequently have constrained resources,
-such as CPU, RAM, network, and disk throughput and capacity. Accumulo generally
-deals well with constrained resources from a stability perspective (optimizing
-performance will require additional tuning, which is not covered in this
-section), however there are some limits.
-
-#### Physical Memory
-
-One of those limits has to do with the Linux out of memory killer. A common
-failure mode in VM environments (and in some bare metal installations) is when
-the Linux out of memory killer decides to kill processes in order to avoid a
-kernel panic when provisioning a memory page. This often happens in VMs due to
-the large number of processes that must run in a small memory footprint. In
-addition to the Linux core processes, a single-node Accumulo setup requires a
-Hadoop Namenode, a Hadoop Secondary Namenode a Hadoop Datanode, a Zookeeper
-server, an Accumulo Master, an Accumulo GC and an Accumulo TabletServer.
-Typical setups also include an Accumulo Monitor, an Accumulo Tracer, a Hadoop
-ResourceManager, a Hadoop NodeManager, provisioning software, and client
-applications. Between all of these processes, it is not uncommon to
-over-subscribe the available RAM in a VM. We recommend setting up VMs without
-swap enabled, so rather than performance grinding to a halt when physical
-memory is exhausted the kernel will randomly* select processes to kill in order
-to free up memory.
-
-Calculating the maximum possible memory usage is essential in creating a stable
-Accumulo VM setup. Safely engineering memory allocation for stability is a
-matter of then bringing the calculated maximum memory usage under the physical
-memory by a healthy margin. The margin is to account for operating system-level
-operations, such as managing process, maintaining virtual memory pages, and
-file system caching. When the java out-of-memory killer finds your process, you
-will probably only see evidence of that in /var/log/messages. Out-of-memory
-process kills do not show up in Accumulo or Hadoop logs.
-
-To calculate the max memory usage of all java virtual machine (JVM) processes
-add the maximum heap size (often limited by a -Xmx... argument, such as in
-accumulo-site.xml) and the off-heap memory usage. Off-heap memory usage
-includes the following:
-
-* "Permanent Space", where the JVM stores Classes, Methods, and other code elements. This can be limited by a JVM flag such as `-XX:MaxPermSize:100m`, and is typically tens of megabytes.
-* Code generation space, where the JVM stores just-in-time compiled code. This is typically small enough to ignore
-* Socket buffers, where the JVM stores send and receive buffers for each socket.
-* Thread stacks, where the JVM allocates memory to manage each thread.
-* Direct memory space and JNI code, where applications can allocate memory outside of the JVM-managed space. For Accumulo, this includes the native in-memory maps that are allocated with the memory.maps.max parameter in accumulo-site.xml.
-* Garbage collection space, where the JVM stores information used for garbage collection.
-
-You can assume that each Hadoop and Accumulo process will use ~100-150MB for
-Off-heap memory, plus the in-memory map of the Accumulo TServer process. A
-simple calculation for physical memory requirements follows:
-
-```
- Physical memory needed
- = (per-process off-heap memory) + (heap memory) + (other processes) + (margin)
- = (number of java processes * 150M + native map) + (sum of -Xmx settings for java process) + (total applications memory, provisioning memory, etc.) + (1G)
- = (11*150M +500M) + (1G +1G +1G +256M +1G +256M +512M +512M +512M +512M +512M) + (2G) + (1G)
- = (2150M) + (7G) + (2G) + (1G)
- = ~12GB
-```
-
-These calculations can add up quickly with the large number of processes,
-especially in constrained VM environments. To reduce the physical memory
-requirements, it is a good idea to reduce maximum heap limits and turn off
-unnecessary processes. If you're not using YARN in your application, you can
-turn off the ResourceManager and NodeManager. If you're not expecting to
-re-provision the cluster frequently you can turn off or reduce provisioning
-processes such as Salt Stack minions and masters.
-
-#### Disk Space
-
-Disk space is primarily used for two operations: storing data and storing logs.
-While Accumulo generally stores all of its key/value data in HDFS, Accumulo,
-Hadoop, and Zookeeper all store a significant amount of logs in a directory on
-a local file system. Care should be taken to make sure that (a) limitations to
-the amount of logs generated are in place, and (b) enough space is available to
-host the generated logs on the partitions that they are assigned. When space is
-not available to log, processes will hang. This can cause interruptions in
-availability of Accumulo, as well as cascade into failures of various
-processes.
-
-Hadoop, Accumulo, and Zookeeper use log4j as a logging mechanism, and each of
-them has a way of limiting the logs and directing them to a particular
-directory. Logs are generated independently for each process, so when
-considering the total space you need to add up the maximum logs generated by
-each process. Typically, a rolling log setup in which each process can generate
-something like 10 100MB files is instituted, resulting in a maximum file system
-usage of 1GB per process. Default setups for Hadoop and Zookeeper are often
-unbounded, so it is important to set these limits in the logging configuration
-files for each subsystem. Consult the user manual for each system for
-instructions on how to limit generated logs.
-
-#### Zookeeper Interaction
-
-Accumulo is designed to scale up to thousands of nodes. At that scale,
-intermittent interruptions in network service and other rare failures of
-compute nodes become more common. To limit the impact of node failures on
-overall service availability, Accumulo uses a heartbeat monitoring system that
-leverages Zookeeper's ephemeral locks. There are several conditions that can
-occur that cause Accumulo process to lose their Zookeeper locks, some of which
-are true interruptions to availability and some of which are false positives.
-Several of these conditions become more common in VM environments, where they
-can be exacerbated by resource constraints and clock drift.
-
-#### Tested Versions
-
-Each release of Accumulo is built with a specific version of Apache
-Hadoop, Apache ZooKeeper and Apache Thrift. We expect Accumulo to
-work with versions that are API compatible with those versions.
-However this compatibility is not guaranteed because Hadoop, ZooKeeper
-and Thift may not provide guarantees between their own versions. We
-have also found that certain versions of Accumulo and Hadoop included
-bugs that greatly affected overall stability. Thrift is particularly
-prone to compatibility changes between versions and you must use the
-same version your Accumulo is built with.
-
-Please check the release notes for your Accumulo version or use the
-mailing lists at https://accumulo.apache.org for more info.
-
-[tracing]: {{page.docs_baseurl}}/administration/overview#tracing
-[monitor]: {{page.docs_baseurl}}/administration/overview#monitoring
-[config-mgmt]: {{page.docs_baseurl}}/administration/configuration-management
-[config-props]: {{page.docs_baseurl}}/administration/configuration-properties
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/036341f2/_docs-unreleased/administration/replication.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/administration/replication.md b/_docs-unreleased/administration/replication.md
index edfa1f8..bff89aa 100644
--- a/_docs-unreleased/administration/replication.md
+++ b/_docs-unreleased/administration/replication.md
@@ -1,7 +1,7 @@
---
title: Replication
category: administration
-order: 5
+order: 10
---
## Overview
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/036341f2/_docs-unreleased/administration/tracing.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/administration/tracing.md b/_docs-unreleased/administration/tracing.md
new file mode 100644
index 0000000..bce8d0b
--- /dev/null
+++ b/_docs-unreleased/administration/tracing.md
@@ -0,0 +1,347 @@
+---
+title: Tracing
+category: administration
+order: 5
+---
+
+It can be difficult to determine why some operations are taking longer
+than expected. For example, you may be looking up items with very low
+latency, but sometimes the lookups take much longer. Determining the
+cause of the delay is difficult because the system is distributed, and
+the typical lookup is fast.
+
+Accumulo has been instrumented to record the time that various
+operations take when tracing is turned on. The fact that tracing is
+enabled follows all the requests made on behalf of the user throughout
+the distributed infrastructure of accumulo, and across all threads of
+execution.
+
+These time spans will be inserted into the `trace` table in
+Accumulo. You can browse recent traces from the Accumulo monitor
+page. You can also read the `trace` table directly like any
+other table.
+
+The design of Accumulo's distributed tracing follows that of [Google's Dapper](http://research.google.com/pubs/pub36356.html).
+
+## Tracers
+
+To collect traces, Accumulo needs at least one tracer server running. If you are using `accumulo-cluster` to start your cluster,
+configure your server in `conf/tracers`. The server collects traces from clients and writes them to the `trace` table. The Accumulo
+user that the tracer connects to Accumulo with can be configured with the following properties (see the [configuration management][config-mgmt]
+page for setting Accumulo server properties)
+
+ trace.user
+ trace.token.property.password
+
+Other tracer configuration properties include
+
+ trace.port.client - port tracer listens on
+ trace.table - table tracer writes to
+ trace.zookeeper.path - zookeeper path where tracers register
+
+The zookeeper path is configured to /tracers by default. If
+multiple Accumulo instances are sharing the same ZooKeeper
+quorum, take care to configure Accumulo with unique values for
+this property.
+
+## Configuring Tracing
+
+Traces are collected via SpanReceivers. The default SpanReceiver
+configured is org.apache.accumulo.core.trace.ZooTraceClient, which
+sends spans to an Accumulo Tracer process, as discussed in the
+previous section. This default can be changed to a different span
+receiver, or additional span receivers can be added in a
+comma-separated list, by modifying the property
+
+ trace.span.receivers
+
+Individual span receivers may require their own configuration
+parameters, which are grouped under the trace.span.receiver.*
+prefix. ZooTraceClient uses the following properties. The first
+three properties are populated from other Accumulo properties,
+while the remaining ones should be prefixed with
+trace.span.receiver. when set in the Accumulo configuration.
+
+ tracer.zookeeper.host - populated from instance.zookeepers
+ tracer.zookeeper.timeout - populated from instance.zookeeper.timeout
+ tracer.zookeeper.path - populated from trace.zookeeper.path
+ tracer.send.timer.millis - timer for flushing send queue (in ms, default 1000)
+ tracer.queue.size - max queue size (default 5000)
+ tracer.span.min.ms - minimum span length to store (in ms, default 1)
+
+Note that to configure an Accumulo client for tracing, including
+the Accumulo shell, the client configuration must be given the same
+trace.span.receivers, trace.span.receiver.*, and trace.zookeeper.path
+properties as the servers have.
+
+Hadoop can also be configured to send traces to Accumulo, as of
+Hadoop 2.6.0, by setting properties in Hadoop's core-site.xml
+file. Instead of using the trace.span.receiver.* prefix, Hadoop
+uses hadoop.htrace.*. The Hadoop configuration does not have
+access to Accumulo's properties, so the
+hadoop.htrace.tracer.zookeeper.host property must be specified.
+The zookeeper timeout defaults to 30000 (30 seconds), and the
+zookeeper path defaults to /tracers. An example of configuring
+Hadoop to send traces to ZooTraceClient is
+
+```xml
+ <property>
+ <name>hadoop.htrace.spanreceiver.classes</name>
+ <value>org.apache.accumulo.core.trace.ZooTraceClient</value>
+ </property>
+ <property>
+ <name>hadoop.htrace.tracer.zookeeper.host</name>
+ <value>zookeeperHost:2181</value>
+ </property>
+ <property>
+ <name>hadoop.htrace.tracer.zookeeper.path</name>
+ <value>/tracers</value>
+ </property>
+ <property>
+ <name>hadoop.htrace.tracer.span.min.ms</name>
+ <value>1</value>
+ </property>
+```
+
+The accumulo-core, accumulo-tracer, accumulo-fate and libthrift
+jars must also be placed on Hadoop's classpath.
+
+### Adding additional SpanReceivers
+
+[Zipkin](https://github.com/openzipkin/zipkin) has a SpanReceiver supported by HTrace and popularized by Twitter
+that users looking for a more graphical trace display may opt to use.
+The following steps configure Accumulo to use `org.apache.htrace.impl.ZipkinSpanReceiver`
+in addition to the Accumulo's default ZooTraceClient, and they serve as a template
+for adding any SpanReceiver to Accumulo:
+
+1. Add the Jar containing the ZipkinSpanReceiver class file to the
+`lib/` directory. It is critical that the Jar is placed in
+`lib/` and NOT in `lib/ext/` so that the new SpanReceiver class
+is visible to the same class loader of htrace-core.
+
+2. Add the following to `accumulo-site.xml`:
+
+ <property>
+ <name>trace.span.receivers</name>
+ <value>org.apache.accumulo.tracer.ZooTraceClient,org.apache.htrace.impl.ZipkinSpanReceiver</value>
+ </property>
+
+3. Restart your Accumulo tablet servers.
+
+In order to use ZipkinSpanReceiver from a client as well as the Accumulo server,
+
+1. Ensure your client can see the ZipkinSpanReceiver class at runtime. For Maven projects,
+this is easily done by adding to your client's pom.xml (taking care to specify a good version)
+
+ <dependency>
+ <groupId>org.apache.htrace</groupId>
+ <artifactId>htrace-zipkin</artifactId>
+ <version>3.1.0-incubating</version>
+ <scope>runtime</scope>
+ </dependency>
+
+2. Add the following to your client configuration.
+
+ trace.span.receivers=org.apache.accumulo.tracer.ZooTraceClient,org.apache.htrace.impl.ZipkinSpanReceiver
+
+3. Instrument your client as in the next section.
+
+Your SpanReceiver may require additional properties, and if so these should likewise
+be placed in the ClientConfiguration (if applicable) and Accumulo's `accumulo-site.xml`.
+Two such properties for ZipkinSpanReceiver, listed with their default values, are
+
+```xml
+ <property>
+ <name>trace.span.receiver.zipkin.collector-hostname</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>trace.span.receiver.zipkin.collector-port</name>
+ <value>9410</value>
+ </property>
+```
+
+### Instrumenting a Client
+
+Tracing can be used to measure a client operation, such as a scan, as
+the operation traverses the distributed system. To enable tracing for
+your application call
+
+```java
+import org.apache.accumulo.core.trace.DistributedTrace;
+...
+DistributedTrace.enable(hostname, "myApplication");
+// do some tracing
+...
+DistributedTrace.disable();
+```
+
+Once tracing has been enabled, a client can wrap an operation in a trace.
+
+```java
+import org.apache.htrace.Sampler;
+import org.apache.htrace.Trace;
+import org.apache.htrace.TraceScope;
+...
+TraceScope scope = Trace.startSpan("Client Scan", Sampler.ALWAYS);
+BatchScanner scanner = conn.createBatchScanner(...);
+// Configure your scanner
+for (Entry entry : scanner) {
+}
+scope.close();
+```
+
+The user can create additional Spans within a Trace.
+
+The sampler (such as `Sampler.ALWAYS`) for the trace should only be specified with a top-level span,
+and subsequent spans will be collected depending on whether that first span was sampled.
+Don't forget to specify a Sampler at the top-level span
+because the default Sampler only samples when part of a pre-existing trace,
+which will never occur in a client that never specifies a Sampler.
+
+```java
+TraceScope scope = Trace.startSpan("Client Update", Sampler.ALWAYS);
+...
+TraceScope readScope = Trace.startSpan("Read");
+...
+readScope.close();
+...
+TraceScope writeScope = Trace.startSpan("Write");
+...
+writeScope.close();
+scope.close();
+```
+
+Like Dapper, Accumulo tracing supports user defined annotations to associate additional data with a Trace.
+Checking whether currently tracing is necessary when using a sampler other than Sampler.ALWAYS.
+
+```java
+...
+int numberOfEntriesRead = 0;
+TraceScope readScope = Trace.startSpan("Read");
+// Do the read, update the counter
+...
+if (Trace.isTracing)
+ readScope.getSpan().addKVAnnotation("Number of Entries Read".getBytes(StandardCharsets.UTF_8),
+ String.valueOf(numberOfEntriesRead).getBytes(StandardCharsets.UTF_8));
+```
+
+It is also possible to add timeline annotations to your spans.
+This associates a string with a given timestamp between the start and stop times for a span.
+
+```java
+...
+writeScope.getSpan().addTimelineAnnotation("Initiating Flush");
+```
+
+Some client operations may have a high volume within your
+application. As such, you may wish to only sample a percentage of
+operations for tracing. As seen below, the CountSampler can be used to
+help enable tracing for 1-in-1000 operations
+
+```java
+import org.apache.htrace.impl.CountSampler;
+...
+Sampler sampler = new CountSampler(HTraceConfiguration.fromMap(
+ Collections.singletonMap(CountSampler.SAMPLER_FREQUENCY_CONF_KEY, "1000")));
+...
+TraceScope readScope = Trace.startSpan("Read", sampler);
+...
+readScope.close();
+```
+
+Remember to close all spans and disable tracing when finished.
+
+```java
+DistributedTrace.disable();
+```
+
+## Viewing Collected Traces
+
+To view collected traces, use the "Recent Traces" link on the Monitor
+UI. You can also programmatically access and print traces using the
+`TraceDump` class.
+
+### Trace Table Format
+
+This section is for developers looking to use data recorded in the trace table
+directly, above and beyond the default services of the Accumulo monitor.
+Please note the trace table format and its supporting classes
+are not in the public API and may be subject to change in future versions.
+
+Each span received by a tracer's ZooTraceClient is recorded in the trace table
+in the form of three entries: span entries, index entries, and start time entries.
+Span and start time entries record full span information,
+whereas index entries provide indexing into span information
+useful for quickly finding spans by type or start time.
+
+Each entry is illustrated by a description and sample of data.
+In the description, a token in quotes is a String literal,
+whereas other other tokens are span variables.
+Parentheses group parts together, to distinguish colon characters inside the
+column family or qualifier from the colon that separates column family and qualifier.
+We use the format `row columnFamily:columnQualifier columnVisibility value`
+(omitting timestamp which records the time an entry is written to the trace table).
+
+Span entries take the following form:
+
+ traceId "span":(parentSpanId:spanId) [] spanBinaryEncoding
+ 63b318de80de96d1 span:4b8f66077df89de1:3778c6739afe4e1 [] %18;%09;...
+
+The parentSpanId is "" for the root span of a trace.
+The spanBinaryEncoding is a compact Apache Thrift encoding of the original Span object.
+This allows clients (and the Accumulo monitor) to recover all the details of the original Span
+at a later time, by scanning the trace table and decoding the value of span entries
+via `TraceFormatter.getRemoteSpan(entry)`.
+
+The trace table has a formatter class by default (org.apache.accumulo.tracer.TraceFormatter)
+that changes how span entries appear from the Accumulo shell.
+Normal scans to the trace table do not use this formatter representation;
+it exists only to make span entries easier to view inside the Accumulo shell.
+
+Index entries take the following form:
+
+ "idx":service:startTime description:sender [] traceId:elapsedTime
+ idx:tserver:14f3828f58b startScan:localhost [] 63b318de80de96d1:1
+
+The service and sender are set by the first call of each Accumulo process
+(and instrumented client processes) to `DistributedTrace.enable(...)`
+(the sender is autodetected if not specified).
+The description is specified in each span.
+Start time and the elapsed time (start - stop, 1 millisecond in the example above)
+are recorded in milliseconds as long values serialized to a string in hex.
+
+Start time entries take the following form:
+
+ "start":startTime "id":traceId [] spanBinaryEncoding
+ start:14f3828a351 id:63b318de80de96d1 [] %18;%09;...
+
+The following classes may be run while Accumulo is running to provide insight into trace statistics. These require
+accumulo-trace-VERSION.jar to be provided on the Accumulo classpath (`lib/ext` is fine).
+
+ $ accumulo org.apache.accumulo.tracer.TraceTableStats -u username -p password -i instancename
+ $ accumulo org.apache.accumulo.tracer.TraceDump -u username -p password -i instancename -r
+
+### Tracing from the Shell
+You can enable tracing for operations run from the shell by using the
+`trace on` and `trace off` commands.
+
+```
+root@test test> trace on
+
+root@test test> scan
+a b:c [] d
+
+root@test test> trace off
+Waiting for trace information
+Waiting for trace information
+Trace started at 2013/08/26 13:24:08.332
+Time Start Service@Location Name
+ 3628+0 shell@localhost shell:root
+ 8+1690 shell@localhost scan
+ 7+1691 shell@localhost scan:location
+ 6+1692 tserver@localhost startScan
+ 5+1692 tserver@localhost tablet read ahead 6
+```
+
+[config-mgmt]: {{page.docs_baseurl}}/administration/configuration-management
[05/10] accumulo-website git commit: Jekyll build from master:036341f
Posted by mw...@apache.org.
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/troubleshooting/overview.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/troubleshooting/overview.html b/docs/unreleased/troubleshooting/overview.html
deleted file mode 100644
index 03df836..0000000
--- a/docs/unreleased/troubleshooting/overview.html
+++ /dev/null
@@ -1,1238 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-<meta charset="utf-8">
-<meta http-equiv="X-UA-Compatible" content="IE=edge">
-<meta name="viewport" content="width=device-width, initial-scale=1">
-<link href="https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/paper/bootstrap.min.css" rel="stylesheet" integrity="sha384-awusxf8AUojygHf2+joICySzB780jVvQaVCAt1clU3QsyAitLGul28Qxb2r1e5g+" crossorigin="anonymous">
-<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.css">
-<link href="/css/accumulo.css" rel="stylesheet" type="text/css">
-
-<title>Accumulo Documentation - Overview</title>
-
-<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
-<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
-<script type="text/javascript" src="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.js"></script>
-<script>
- // show location of canonical site if not currently on the canonical site
- $(function() {
- var host = window.location.host;
- if (typeof host !== 'undefined' && host !== 'accumulo.apache.org') {
- $('#non-canonical').show();
- }
- });
-
- $(function() {
- // decorate section headers with anchors
- return $("h2, h3, h4, h5, h6").each(function(i, el) {
- var $el, icon, id;
- $el = $(el);
- id = $el.attr('id');
- icon = '<i class="fa fa-link"></i>';
- if (id) {
- return $el.append($("<a />").addClass("header-link").attr("href", "#" + id).html(icon));
- }
- });
- });
-
- // configure Google Analytics
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
-
- if (ga.hasOwnProperty('loaded') && ga.loaded === true) {
- ga('create', 'UA-50934829-1', 'apache.org');
- ga('send', 'pageview');
- }
-</script>
-
-</head>
-<body style="padding-top: 100px">
-
- <nav class="navbar navbar-default navbar-fixed-top">
- <div class="container">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- <a href="/"><img id="nav-logo" alt="Apache Accumulo" class="img-responsive" src="/images/accumulo-logo.png" width="200"
- /></a>
- </div>
- <div class="collapse navbar-collapse" id="navbar-items">
- <ul class="nav navbar-nav">
- <li class="nav-link"><a href="/downloads">Download</a></li>
- <li class="dropdown">
- <a class="dropdown-toggle" data-toggle="dropdown" href="#">Releases<span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li><a href="/release/accumulo-1.8.1/">1.8.1 (Latest)</a></li>
- <li><a href="/release/accumulo-1.7.3/">1.7.3</a></li>
- <li><a href="/release/accumulo-1.6.6/">1.6.6</a></li>
- <li><a href="/release/">Archive</a></li>
- </ul>
- </li>
- <li class="dropdown">
- <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li><a href="/1.8/accumulo_user_manual.html">User Manual (1.8)</a></li>
- <li><a href="/1.8/apidocs">Javadocs (1.8)</a></li>
- <li><a href="/1.8/examples">Examples (1.8)</a></li>
- <li><a href="/features">Features</a></li>
- <li><a href="/glossary">Glossary</a></li>
- <li><a href="/external-docs">External Docs</a></li>
- <li><a href="/docs-archive/">Archive</a></li>
- </ul>
- </li>
- <li class="dropdown">
- <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li><a href="/get_involved">Get Involved</a></li>
- <li><a href="/mailing_list">Mailing Lists</a></li>
- <li><a href="/people">People</a></li>
- <li><a href="/related-projects">Related Projects</a></li>
- <li><a href="/contributor/">Contributor Guide</a></li>
- </ul>
- </li>
- </ul>
- <ul class="nav navbar-nav navbar-right">
- <li class="dropdown">
- <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation<span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li><a href="https://www.apache.org">Apache Homepage <i class="fa fa-external-link"></i></a></li>
- <li><a href="https://www.apache.org/licenses/LICENSE-2.0">License <i class="fa fa-external-link"></i></a></li>
- <li><a href="https://www.apache.org/foundation/sponsorship">Sponsorship <i class="fa fa-external-link"></i></a></li>
- <li><a href="https://www.apache.org/security">Security <i class="fa fa-external-link"></i></a></li>
- <li><a href="https://www.apache.org/foundation/thanks">Thanks <i class="fa fa-external-link"></i></a></li>
- <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct <i class="fa fa-external-link"></i></a></li>
- </ul>
- </li>
- </ul>
- </div>
- </div>
-</nav>
-
- <div class="container">
- <div class="row">
- <div class="col-md-12">
-
- <div id="non-canonical" style="display: none; background-color: #F0E68C; padding-left: 1em;">
- Visit the official site at: <a href="https://accumulo.apache.org">https://accumulo.apache.org</a>
- </div>
- <div id="content">
-
- <div class="alert alert-danger" role="alert">This documentation is for an unreleased version of Apache Accumulo that is currently under development! Check out the <a href="/1.8/accumulo_user_manual.html">documentation for the latest release</a>.</div>
-
-<div class="row">
- <div class="col-md-3">
- <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true">
- <div class="panel panel-default">
-
-
-
-
-
-
-
-
-
- <div class="panel-heading" role="tab" id="headingOne">
- <h4 class="panel-title">
- <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsegetting-started" aria-expanded="false" aria-controls="collapsegetting-started">
- Getting started
- </a>
- </h4>
- </div>
- <div id="collapsegetting-started" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
- <div class="panel-body">
-
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/design">Accumulo Design</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/clients">Accumulo Clients</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/shell">Accumulo Shell</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_design">Table Design</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_configuration">Table Configuration</a></div>
-
- </div>
- </div>
-
-
-
-
-
-
-
-
-
-
-
- <div class="panel-heading" role="tab" id="headingOne">
- <h4 class="panel-title">
- <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsedevelopment" aria-expanded="false" aria-controls="collapsedevelopment">
- Development
- </a>
- </h4>
- </div>
- <div id="collapsedevelopment" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
- <div class="panel-body">
-
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/sampling">Sampling</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/summaries">Summary Statistics</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/security">Security</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div>
-
- </div>
- </div>
-
-
-
-
-
-
-
-
-
-
-
- <div class="panel-heading" role="tab" id="headingOne">
- <h4 class="panel-title">
- <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapseadministration" aria-expanded="false" aria-controls="collapseadministration">
- Administration
- </a>
- </h4>
- </div>
- <div id="collapseadministration" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
- <div class="panel-body">
-
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/overview">Overview</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/multivolume">Multi-Volume Installations</a></div>
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
-
- </div>
- </div>
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <div class="panel-heading" role="tab" id="headingOne">
- <h4 class="panel-title">
- <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsetroubleshooting" aria-expanded="true" aria-controls="collapsetroubleshooting">
- Troubleshooting
- </a>
- </h4>
- </div>
- <div id="collapsetroubleshooting" class="panel-collapse collapse in" role="tabpanel" aria-labelledby="headingOne">
- <div class="panel-body">
-
-
- <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/overview">Overview</a></div>
-
- </div>
- </div>
-
-
-
- </div>
- </div>
- </div>
- <div class="col-md-9">
-
- <p><a href="/docs/unreleased/">Accumulo unreleased docs</a> >> Troubleshooting >> Overview</p>
-
-
- <h1>Overview</h1>
-
- <h2 id="logs">Logs</h2>
-
-<p><strong>The tablet server does not seem to be running!? What happened?</strong></p>
-
-<p>Accumulo is a distributed system. It is supposed to run on remote
-equipment, across hundreds of computers. Each program that runs on
-these remote computers writes down events as they occur, into a local
-file. By default, this is defined in <code class="highlighter-rouge">conf/accumulo-env.sh</code> as <code class="highlighter-rouge">ACCUMULO_LOG_DIR</code>.
-Look in the <code class="highlighter-rouge">$ACCUMULO_LOG_DIR/tserver*.log</code> file. Specifically, check the end of the file.</p>
-
-<p><strong>The tablet server did not start and the debug log does not exists! What happened?</strong></p>
-
-<p>When the individual programs are started, the stdout and stderr output
-of these programs are stored in <code class="highlighter-rouge">.out</code> and <code class="highlighter-rouge">.err</code> files in
-<code class="highlighter-rouge">$ACCUMULO_LOG_DIR</code>. Often, when there are missing configuration
-options, files or permissions, messages will be left in these files.
-Probably a start-up problem. Look in <code class="highlighter-rouge">$ACCUMULO_LOG_DIR/tserver*.err</code></p>
-
-<h2 id="monitor">Monitor</h2>
-
-<p><strong>Accumulo is not working, what’s wrong?</strong></p>
-
-<p>There’s a small web server that collects information about all the
-components that make up a running Accumulo instance. It will highlight
-unusual or unexpected conditions.</p>
-
-<p>Point your browser to the monitor (typically the master host, on port 9995). Is anything red or yellow?</p>
-
-<p><strong>My browser is reporting connection refused, and I cannot get to the monitor</strong></p>
-
-<p>The monitor program’s output is also written to .err and .out files in
-the <code class="highlighter-rouge">$ACCUMULO_LOG_DIR</code>. Look for problems in this file if the
-<code class="highlighter-rouge">$ACCUMULO_LOG_DIR/monitor*.log</code> file does not exist.</p>
-
-<p>The monitor program is probably not running. Check the log files for errors.</p>
-
-<p><strong>My browser hangs trying to talk to the monitor.</strong></p>
-
-<p>Your browser needs to be able to reach the monitor program. Often
-large clusters are firewalled, or use a VPN for internal
-communications. You can use SSH to proxy your browser to the cluster,
-or consult with your system administrator to gain access to the server
-from your browser.</p>
-
-<p>It is sometimes helpful to use a text-only browser to sanity-check the
-monitor while on the machine running the monitor:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ links http://localhost:9995
-</code></pre>
-</div>
-
-<p>Verify that you are not firewalled from the monitor if it is running on a remote host.</p>
-
-<p><strong>The monitor responds, but there are no numbers for tservers and tables. The summary page says the master is down.</strong></p>
-
-<p>The monitor program gathers all the details about the master and the
-tablet servers through the master. It will be mostly blank if the
-master is down. Check for a running master.</p>
-
-<h2 id="hdfs">HDFS</h2>
-
-<p>Accumulo reads and writes to the Hadoop Distributed File System.
-Accumulo needs this file system available at all times for normal operations.</p>
-
-<p><strong>Accumulo is having problems “getting a block blk_1234567890123”. How do I fix it?</strong></p>
-
-<p>This troubleshooting guide does not cover HDFS, but in general, you
-want to make sure that all the datanodes are running and an fsck check
-finds the file system clean:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ hadoop fsck /accumulo
-</code></pre>
-</div>
-
-<p>You can use:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ hadoop fsck /accumulo/path/to/corrupt/file -locations -blocks -files
-</code></pre>
-</div>
-
-<p>to locate the block references of individual corrupt files and use those
-references to search the name node and individual data node logs to determine which
-servers those blocks have been assigned and then try to fix any underlying file
-system issues on those nodes.</p>
-
-<p>On a larger cluster, you may need to increase the number of Xcievers for HDFS DataNodes:</p>
-
-<div class="language-xml highlighter-rouge"><pre class="highlight"><code><span class="nt"><property></span>
- <span class="nt"><name></span>dfs.datanode.max.xcievers<span class="nt"></name></span>
- <span class="nt"><value></span>4096<span class="nt"></value></span>
-<span class="nt"></property></span>
-</code></pre>
-</div>
-
-<p>Verify HDFS is healthy, check the datanode logs.</p>
-
-<h2 id="zookeeper">Zookeeper</h2>
-
-<p>** The”accumulo init” command is hanging. It says something about talking to zookeeper.**</p>
-
-<p>Zookeeper is also a distributed service. You will need to ensure that
-it is up. You can run the zookeeper command line tool to connect to
-any one of the zookeeper servers:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ zkCli.sh -server zoohost
-...
-[zk: zoohost:2181(CONNECTED) 0]
-</code></pre>
-</div>
-
-<p>It is important to see the word <code class="highlighter-rouge">CONNECTED</code>! If you only see
-<code class="highlighter-rouge">CONNECTING</code> you will need to diagnose zookeeper errors.</p>
-
-<p>Check to make sure that zookeeper is up, and that
-<code class="highlighter-rouge">accumulo-site.xml</code> has been pointed to
-your zookeeper server(s).</p>
-
-<p><strong>Zookeeper is running, but it does not say CONNECTED</strong></p>
-
-<p>Zookeeper processes talk to each other to elect a leader. All updates
-go through the leader and propagate to a majority of all the other
-nodes. If a majority of the nodes cannot be reached, zookeeper will
-not allow updates. Zookeeper also limits the number connections to a
-server from any other single host. By default, this limit can be as small as 10
-and can be reached in some everything-on-one-machine test configurations.</p>
-
-<p>You can check the election status and connection status of clients by
-asking the zookeeper nodes for their status. You connect to zookeeper
-and ask it with the four-letter <code class="highlighter-rouge">stat</code> command:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ nc zoohost 2181
-stat
-Zookeeper version: 3.4.5-1392090, built on 09/30/2012 17:52 GMT
-Clients:
- /127.0.0.1:58289[0](queued=0,recved=1,sent=0)
- /127.0.0.1:60231[1](queued=0,recved=53910,sent=53915)
-
-Latency min/avg/max: 0/5/3008
-Received: 1561459
-Sent: 1561592
-Connections: 2
-Outstanding: 0
-Zxid: 0x621a3b
-Mode: standalone
-Node count: 22524
-</code></pre>
-</div>
-
-<p>Check zookeeper status, verify that it has a quorum, and has not exceeded maxClientCnxns.</p>
-
-<p><strong>My tablet server crashed! The logs say that it lost its zookeeper lock.</strong></p>
-
-<p>Tablet servers reserve a lock in zookeeper to maintain their ownership
-over the tablets that have been assigned to them. Part of their
-responsibility for keeping the lock is to send zookeeper a keep-alive
-message periodically. If the tablet server fails to send a message in
-a timely fashion, zookeeper will remove the lock and notify the tablet
-server. If the tablet server does not receive a message from
-zookeeper, it will assume its lock has been lost, too. If a tablet
-server loses its lock, it kills itself: everything assumes it is dead
-already.</p>
-
-<p>Investigate why the tablet server did not send a timely message to
-zookeeper.</p>
-
-<h3 id="keeping-the-tablet-server-lock">Keeping the tablet server lock</h3>
-
-<p><strong>My tablet server lost its lock. Why?</strong></p>
-
-<p>The primary reason a tablet server loses its lock is that it has been pushed into swap.</p>
-
-<p>A large java program (like the tablet server) may have a large portion
-of its memory image unused. The operation system will favor pushing
-this allocated, but unused memory into swap so that the memory can be
-re-used as a disk buffer. When the java virtual machine decides to
-access this memory, the OS will begin flushing disk buffers to return that
-memory to the VM. This can cause the entire process to block long
-enough for the zookeeper lock to be lost.</p>
-
-<p>Configure your system to reduce the kernel parameter <em>swappiness</em> from the default (60) to zero.</p>
-
-<p><strong>My tablet server lost its lock, and I have already set swappiness to zero. Why?</strong></p>
-
-<p>Be careful not to over-subscribe memory. This can be easy to do if
-your accumulo processes run on the same nodes as hadoop’s map-reduce
-framework. Remember to add up:</p>
-
-<ul>
- <li>size of the JVM for the tablet server</li>
- <li>size of the in-memory map, if using the native map implementation</li>
- <li>size of the JVM for the data node</li>
- <li>size of the JVM for the task tracker</li>
- <li>size of the JVM times the maximum number of mappers and reducers</li>
- <li>size of the kernel and any support processes</li>
-</ul>
-
-<p>If a 16G node can run 2 mappers and 2 reducers, and each can be 2G,
-then there is only 8G for the data node, tserver, task tracker and OS.</p>
-
-<p>Reduce the memory footprint of each component until it fits comfortably.</p>
-
-<p><strong>My tablet server lost its lock, swappiness is zero, and my node has lots of unused memory!</strong></p>
-
-<p>The JVM memory garbage collector may fall behind and cause a
-“stop-the-world” garbage collection. On a large memory virtual
-machine, this collection can take a long time. This happens more
-frequently when the JVM is getting low on free memory. Check the logs
-of the tablet server. You will see lines like this:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>2013-06-20 13:43:20,607 [tabletserver.TabletServer] DEBUG: gc ParNew=0.00(+0.00) secs
- ConcurrentMarkSweep=0.00(+0.00) secs freemem=1,868,325,952(+1,868,325,952) totalmem=2,040,135,680
-</code></pre>
-</div>
-
-<p>When <code class="highlighter-rouge">freemem</code> becomes small relative to the amount of memory
-needed, the JVM will spend more time finding free memory than
-performing work. This can cause long delays in sending keep-alive
-messages to zookeeper.</p>
-
-<p>Ensure the tablet server JVM is not running low on memory.</p>
-
-<p><strong>I’m seeing errors in tablet server logs that include the words “MutationsRejectedException” and “# constraint violations: 1”. Moments after that the server died.</strong></p>
-
-<p>The error you are seeing is part of a failing tablet server scenario.
-This is a bit complicated, so name two of your tablet servers A and B.</p>
-
-<p>Tablet server A is hosting a tablet, let’s call it a-tablet.</p>
-
-<p>Tablet server B is hosting a metadata tablet, let’s call it m-tablet.</p>
-
-<p>m-tablet records the information about a-tablet, for example, the names of the files it is using to store data.</p>
-
-<p>When A ingests some data, it eventually flushes the updates from memory to a file.</p>
-
-<p>Tablet server A then writes this new information to m-tablet, on Tablet server B.</p>
-
-<p>Here’s a likely failure scenario:</p>
-
-<p>Tablet server A does not have enough memory for all the processes running on it.
-The operating system sees a large chunk of the tablet server being unused, and swaps it out to disk to make room for other processes.
-Tablet server A does a java memory garbage collection, which causes it to start using all the memory allocated to it.
-As the server starts pulling data from swap, it runs very slowly.
-It fails to send the keep-alive messages to zookeeper in a timely fashion, and it looses its zookeeper session.</p>
-
-<p>But, it’s running so slowly, that it takes a moment to realize it should no longer be hosting tablets.</p>
-
-<p>The thread that is flushing a-tablet memory attempts to update m-tablet with the new file information.</p>
-
-<p>Fortunately there’s a constraint on m-tablet.
-Mutations to the metadata table must contain a valid zookeeper session.
-This prevents tablet server A from making updates to m-tablet when it no long has the right to host the tablet.</p>
-
-<p>The “MutationsRejectedException” error is from tablet server A making an update to tablet server B’s m-tablet.
-It’s getting a constraint violation: tablet server A has lost its zookeeper session, and will fail momentarily.</p>
-
-<p>Ensure that memory is not over-allocated. Monitor swap usage, or turn swap off.</p>
-
-<p><strong>My accumulo client is getting a MutationsRejectedException. The monitor is displaying “No Such SessionID” errors.</strong></p>
-
-<p>When your client starts sending mutations to accumulo, it creates a session. Once the session is created,
-mutations are streamed to accumulo, without acknowledgement, against this session. Once the client is done,
-it will close the session, and get an acknowledgement.</p>
-
-<p>If the client fails to communicate with accumulo, it will release the session, assuming that the client has died.
-If the client then attempts to send more mutations against the session, you will see “No Such SessionID” errors on
-the server, and MutationRejectedExceptions in the client.</p>
-
-<p>The client library should be either actively using the connection to the tablet servers,
-or closing the connection and sessions. If the session times out, something is causing your client
-to pause.</p>
-
-<p>The most frequent source of these pauses are java garbage collection pauses
-due to the JVM running out of memory, or being swapped out to disk.</p>
-
-<p>Ensure your client has adequate memory and is not being swapped out to disk.</p>
-
-<h2 id="tools">Tools</h2>
-
-<p>The accumulo script can be used to run classes from the command line.
-This section shows how a few of the utilities work, but there are many
-more.</p>
-
-<p>There’s a class that will examine an accumulo storage file and print
-out basic metadata.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.core.file.rfile.PrintInfo /accumulo/tables/1/default_tablet/A000000n.rf
-2013-07-16 08:17:14,778 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
-Locality group : <DEFAULT>
- Start block : 0
- Num blocks : 1
- Index level 0 : 62 bytes 1 blocks
- First key : 288be9ab4052fe9e span:34078a86a723e5d3:3da450f02108ced5 [] 1373373521623 false
- Last key : start:13fc375709e id:615f5ee2dd822d7a [] 1373373821660 false
- Num entries : 466
- Column families : [waitForCommits, start, md major compactor 1, md major compactor 2, md major compactor 3,
- bringOnline, prep, md major compactor 4, md major compactor 5, md root major compactor 3,
- minorCompaction, wal, compactFiles, md root major compactor 4, md root major compactor 1,
- md root major compactor 2, compact, id, client:update, span, update, commit, write,
- majorCompaction]
-
-Meta block : BCFile.index
- Raw size : 4 bytes
- Compressed size : 12 bytes
- Compression type : gz
-
-Meta block : RFile.index
- Raw size : 780 bytes
- Compressed size : 344 bytes
- Compression type : gz
-</code></pre>
-</div>
-
-<p>When trying to diagnose problems related to key size, the <code class="highlighter-rouge">PrintInfo</code> tool can provide a histogram of the individual key sizes:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.core.file.rfile.PrintInfo --histogram /accumulo/tables/1/default_tablet/A000000n.rf
-...
-Up to size count %-age
- 10 : 222 28.23%
- 100 : 244 71.77%
- 1000 : 0 0.00%
- 10000 : 0 0.00%
- 100000 : 0 0.00%
- 1000000 : 0 0.00%
- 10000000 : 0 0.00%
- 100000000 : 0 0.00%
- 1000000000 : 0 0.00%
-10000000000 : 0 0.00%
-</code></pre>
-</div>
-
-<p>Likewise, <code class="highlighter-rouge">PrintInfo</code> will dump the key-value pairs and show you the contents of the RFile:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.core.file.rfile.PrintInfo --dump /accumulo/tables/1/default_tablet/A000000n.rf
-row columnFamily:columnQualifier [visibility] timestamp deleteFlag -> Value
-...
-</code></pre>
-</div>
-
-<p><strong>Accumulo is not showing me any data!</strong></p>
-
-<p>Do you have your auths set so that it matches your visibilities?</p>
-
-<p><strong>What are my visibilities?</strong></p>
-
-<p>Use <code class="highlighter-rouge">PrintInfo</code> on a representative file to get some idea of the visibilities in the underlying data.</p>
-
-<p>Note that the use of <code class="highlighter-rouge">PrintInfo</code> is an administrative tool and can only
-by used by someone who can access the underlying Accumulo data. It
-does not provide the normal access controls in Accumulo.</p>
-
-<p>If you would like to backup, or otherwise examine the contents of Zookeeper, there are commands to dump and load to/from XML.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.server.util.DumpZookeeper --root /accumulo >dump.xml
-$ accumulo org.apache.accumulo.server.util.RestoreZookeeper --overwrite < dump.xml
-</code></pre>
-</div>
-
-<p><strong>How can I get the information in the monitor page for my cluster monitoring system?</strong></p>
-
-<p>Use GetMasterStats:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.test.GetMasterStats | grep Load
- OS Load Average: 0.27
-</code></pre>
-</div>
-
-<p><strong>The monitor page is showing an offline tablet. How can I find out which tablet it is?</strong></p>
-
-<p>Use FindOfflineTablets:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.server.util.FindOfflineTablets
-2<<@(null,null,localhost:9997) is UNASSIGNED #walogs:2
-</code></pre>
-</div>
-
-<p>Here’s what the output means:</p>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">2<<</code> -
- This is the tablet from (-inf, pass:[+]inf) for the
- table with id 2. The command <code class="highlighter-rouge">tables -l</code> in the shell will show table ids for
- tables.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">@(null, null, localhost:9997)</code> -
- Location information. The
- format is <code class="highlighter-rouge">@(assigned, hosted, last)</code>. In this case, the
- tablet has not been assigned, is not hosted anywhere, and was once
- hosted on localhost.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">#walogs:2</code> -
- The number of write-ahead logs that this tablet requires for recovery.</p>
- </li>
-</ul>
-
-<p>An unassigned tablet with write-ahead logs is probably waiting for
-logs to be sorted for efficient recovery.</p>
-
-<p><strong>How can I be sure that the metadata tables are up and consistent?</strong></p>
-
-<p><code class="highlighter-rouge">CheckForMetadataProblems</code> will verify the start/end of
-every tablet matches, and the start and stop for the table is empty:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.server.util.CheckForMetadataProblems -u root --password
-Enter the connection password:
-All is well for table !0
-All is well for table 1
-</code></pre>
-</div>
-
-<p><strong>My hadoop cluster has lost a file due to a NameNode failure. How can I remove the file?</strong></p>
-
-<p>There’s a utility that will check every file reference and ensure
-that the file exists in HDFS. Optionally, it will remove the
-reference:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.server.util.RemoveEntriesForMissingFiles -u root --password
-Enter the connection password:
-2013-07-16 13:10:57,293 [util.RemoveEntriesForMissingFiles] INFO : File /accumulo/tables/2/default_tablet/F0000005.rf
- is missing
-2013-07-16 13:10:57,296 [util.RemoveEntriesForMissingFiles] INFO : 1 files of 3 missing
-</code></pre>
-</div>
-
-<p><strong>I have many entries in zookeeper for old instances I no longer need. How can I remove them?</strong></p>
-
-<p>Use CleanZookeeper:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.server.util.CleanZookeeper
-</code></pre>
-</div>
-
-<p>This command will not delete the instance pointed to by the local <code class="highlighter-rouge">accumulo-site.xml</code> file.</p>
-
-<p><strong>I need to decommission a node. How do I stop the tablet server on it?</strong></p>
-
-<p>Use the admin command:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo admin stop hostname:9997
-2013-07-16 13:15:38,403 [util.Admin] INFO : Stopping server 12.34.56.78:9997
-</code></pre>
-</div>
-
-<p><strong>I cannot login to a tablet server host, and the tablet server will not shut down. How can I kill the server?</strong></p>
-
-<p>Sometimes you can kill a “stuck” tablet server by deleting its lock in zookeeper:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.server.util.TabletServerLocks --list
- 127.0.0.1:9997 TSERV_CLIENT=127.0.0.1:9997
-$ accumulo org.apache.accumulo.server.util.TabletServerLocks -delete 127.0.0.1:9997
-$ accumulo org.apache.accumulo.server.util.TabletServerLocks -list
- 127.0.0.1:9997 null
-</code></pre>
-</div>
-
-<p>You can find the master and instance id for any accumulo instances using the same zookeeper instance:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.server.util.ListInstances
-INFO : Using ZooKeepers localhost:2181
-
- Instance Name | Instance ID | Master
----------------------+--------------------------------------+-------------------------------
- "test" | 6140b72e-edd8-4126-b2f5-e74a8bbe323b | 127.0.0.1:9999
-</code></pre>
-</div>
-
-<h2 id="system-metadata-tables">System Metadata Tables</h2>
-
-<p>Accumulo tracks information about tables in metadata tables. The metadata for
-most tables is contained within the metadata table in the accumulo namespace,
-while metadata for that table is contained in the root table in the accumulo
-namespace. The root table is composed of a single tablet, which does not
-split, so it is also called the root tablet. Information about the root
-table, such as its location and write-ahead logs, are stored in ZooKeeper.</p>
-
-<p>Let’s create a table and put some data into it:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>shell> createtable test
-
-shell> tables -l
-accumulo.metadata => !0
-accumulo.root => `r
-test => 2
-trace => 1
-
-shell> insert a b c d
-
-shell> flush -w
-</code></pre>
-</div>
-
-<p>Now let’s take a look at the metadata for this table:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>shell> table accumulo.metadata
-shell> scan -b 3; -e 3<
-3< file:/default_tablet/F000009y.rf [] 186,1
-3< last:13fe86cd27101e5 [] 127.0.0.1:9997
-3< loc:13fe86cd27101e5 [] 127.0.0.1:9997
-3< log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995|6
-3< srv:dir [] /default_tablet
-3< srv:flush [] 1
-3< srv:lock [] tservers/127.0.0.1:9997/zlock-0000000001$13fe86cd27101e5
-3< srv:time [] M1373998392323
-3< ~tab:~pr [] \x00
-</code></pre>
-</div>
-
-<p>Let’s decode this little session:</p>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">scan -b 3; -e 3<</code> - Every tablet gets its own row. Every row starts with the table id followed by
- <code class="highlighter-rouge">;</code> or <code class="highlighter-rouge"><</code>, and followed by the end row split point for that tablet.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">file:/default_tablet/F000009y.rf [] 186,1</code> -
- File entry for this tablet. This tablet contains a single file reference. The
- file is <code class="highlighter-rouge">/accumulo/tables/3/default_tablet/F000009y.rf</code>. It contains 1
- key/value pair, and is 186 bytes long.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">last:13fe86cd27101e5 [] 127.0.0.1:9997</code> -
- Last location for this tablet. It was last held on 127.0.0.1:9997, and the
- unique tablet server lock data was `13fe86cd27101e5+. The default balancer
- will tend to put tablets back on their last location.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">loc:13fe86cd27101e5 [] 127.0.0.1:9997</code> -
- The current location of this tablet.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0. ...</code> -
- This tablet has a reference to a single write-ahead log. This file can be found in
- <code class="highlighter-rouge">/accumulo/wal/127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995</code>. The value
- of this entry could refer to multiple files. This tablet’s data is encoded as
- <code class="highlighter-rouge">6</code> within the log.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">srv:dir [] /default_tablet</code> -
- Files written for this tablet will be placed into
- <code class="highlighter-rouge">/accumulo/tables/3/default_tablet</code>.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">srv:flush [] 1</code> -
- Flush id. This table has successfully completed the flush with the id of +1+.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">srv:lock [] tservers/127.0.0.1:9997/zlock-0000000001\$13fe86cd27101e5</code> -
- This is the lock information for the tablet holding the present lock. This
- information is checked against zookeeper whenever this is updated, which
- prevents a metadata update from a tablet server that no longer holds its
- lock.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">srv:time [] M1373998392323</code> -
- This indicates the time time type (+M+ for milliseconds or +L+ for logical) and the timestamp of the most recently written key in this tablet. It is used to ensure automatically assigned key timestamps are strictly increasing for the tablet, regardless of the tablet server’s system time.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">~tab:~pr [] \x00</code> -
- The end-row marker for the previous tablet (prev-row). The first byte
- indicates the presence of a prev-row. This tablet has the range (-inf, `inf),
- so it has no prev-row (or end row).</p>
- </li>
-</ul>
-
-<p>Besides these columns, you may see:</p>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">rowId future:zooKeeperID location</code> -
- Tablet has been assigned to a tablet, but not yet loaded.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">~del:filename</code> -
- When a tablet server is done use a file, it will create a delete marker in the appropriate metadata table, unassociated with any tablet. The garbage collector will remove the marker, and the file, when no other reference to the file exists.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">~blip:txid</code> -
- Bulk-Load In Progress marker.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">rowId loaded:filename</code> -
- A file has been bulk-loaded into this tablet, however the bulk load has not yet completed on other tablets, so this marker prevents the file from being loaded multiple times.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">rowId !cloned</code> -
- A marker that indicates that this tablet has been successfully cloned.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">rowId splitRatio:ratio</code> -
- A marker that indicates a split is in progress, and the files are being split at the given ratio.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">rowId chopped</code> -
- A marker that indicates that the files in the tablet do not contain keys outside the range of the tablet.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">rowId scan</code> -
- A marker that prevents a file from being removed while there are still active scans using it.</p>
- </li>
-</ul>
-
-<h2 id="simple-system-recovery">Simple System Recovery</h2>
-
-<p><strong>One of my Accumulo processes died. How do I bring it back?</strong></p>
-
-<p>The easiest way to bring all services online for an Accumulo instance is to run the <code class="highlighter-rouge">accumulo-cluster</code> script.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo-cluster start
-</code></pre>
-</div>
-
-<p>This process will check the process listing, using <code class="highlighter-rouge">jps</code> on each host before attempting to restart a service on the given host.
-Typically, this check is sufficient except in the face of a hung/zombie process. For large clusters, it may be
-undesirable to ssh to every node in the cluster to ensure that all hosts are running the appropriate processes and <code class="highlighter-rouge">accumulo-service</code> may be of use.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ ssh host_with_dead_process
-$ accumulo-service tserver start
-</code></pre>
-</div>
-
-<p><strong>My process died again. Should I restart it via <code class="highlighter-rouge">cron</code> or tools like <code class="highlighter-rouge">supervisord</code>?</strong></p>
-
-<p>A repeatedly dying Accumulo process is a sign of a larger problem. Typically these problems are due to a
-misconfiguration of Accumulo or over-saturation of resources. Blind automation of any service restart inside of Accumulo
-is generally an undesirable situation as it is indicative of a problem that is being masked and ignored. Accumulo
-processes should be stable on the order of months and not require frequent restart.</p>
-
-<h2 id="advanced-system-recovery">Advanced System Recovery</h2>
-
-<h3 id="hdfs-failure">HDFS Failure</h3>
-
-<p><strong>I had disasterous HDFS failure. After bringing everything back up, several tablets refuse to go online.</strong></p>
-
-<p>Data written to tablets is written into memory before being written into indexed files. In case the server
-is lost before the data is saved into a an indexed file, all data stored in memory is first written into a
-write-ahead log (WAL). When a tablet is re-assigned to a new tablet server, the write-ahead logs are read to
-recover any mutations that were in memory when the tablet was last hosted.</p>
-
-<p>If a write-ahead log cannot be read, then the tablet is not re-assigned. All it takes is for one of
-the blocks in the write-ahead log to be missing. This is unlikely unless multiple data nodes in HDFS have been
-lost.</p>
-
-<p>Get the WAL files online and healthy. Restore any data nodes that may be down.</p>
-
-<p><strong>How do find out which tablets are offline?</strong></p>
-
-<p>Use <code class="highlighter-rouge">accumulo admin checkTablets</code></p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo admin checkTablets
-</code></pre>
-</div>
-
-<p><strong>I lost three data nodes, and I’m missing blocks in a WAL. I don’t care about data loss, how
-can I get those tablets online?</strong></p>
-
-<p>See the discussion in «metadata», which shows a typical metadata table listing.
-The entries with a column family of <code class="highlighter-rouge">log</code> are references to the WAL for that tablet.
-If you know what WAL is bad, you can find all the references with a grep in the shell:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>shell> grep 0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995
-3< log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995|6
-</code></pre>
-</div>
-
-<p>You can remove the WAL references in the metadata table.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>shell> grant -u root Table.WRITE -t accumulo.metadata
-shell> delete 3< log 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995
-</code></pre>
-</div>
-
-<p>Note: the colon (<code class="highlighter-rouge">:</code>) is omitted when specifying the <em>row cf cq</em> for the delete command.</p>
-
-<p>The master will automatically discover the tablet no longer has a bad WAL reference and will
-assign the tablet. You will need to remove the reference from all the tablets to get them
-online.</p>
-
-<p><strong>The metadata (or root) table has references to a corrupt WAL.</strong></p>
-
-<p>This is a much more serious state, since losing updates to the metadata table will result
-in references to old files which may not exist, or lost references to new files, resulting
-in tablets that cannot be read, or large amounts of data loss.</p>
-
-<p>The best hope is to restore the WAL by fixing HDFS data nodes and bringing the data back online.
-If this is not possible, the best approach is to re-create the instance and bulk import all files from
-the old instance into a new tables.</p>
-
-<p>A complete set of instructions for doing this is outside the scope of this guide,
-but the basic approach is:</p>
-
-<ul>
- <li>Use <code class="highlighter-rouge">tables -l</code> in the shell to discover the table name to table id mapping</li>
- <li>Stop all accumulo processes on all nodes</li>
- <li>Move the accumulo directory in HDFS out of the way:
- $ hadoop fs -mv /accumulo /corrupt</li>
- <li>Re-initalize accumulo</li>
- <li>Recreate tables, users and permissions</li>
- <li>Import the directories under <code class="highlighter-rouge">/corrupt/tables/<id></code> into the new instance</li>
-</ul>
-
-<p><strong>One or more HDFS Files under /accumulo/tables are corrupt</strong></p>
-
-<p>Accumulo maintains multiple references into the tablet files in the metadata
-tables and within the tablet server hosting the file, this makes it difficult to
-reliably just remove those references.</p>
-
-<p>The directory structure in HDFS for tables will follow the general structure:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>/accumulo
-/accumulo/tables/
-/accumulo/tables/!0
-/accumulo/tables/!0/default_tablet/A000001.rf
-/accumulo/tables/!0/t-00001/A000002.rf
-/accumulo/tables/1
-/accumulo/tables/1/default_tablet/A000003.rf
-/accumulo/tables/1/t-00001/A000004.rf
-/accumulo/tables/1/t-00001/A000005.rf
-/accumulo/tables/2/default_tablet/A000006.rf
-/accumulo/tables/2/t-00001/A000007.rf
-</code></pre>
-</div>
-
-<p>If files under <code class="highlighter-rouge">/accumulo/tables</code> are corrupt, the best course of action is to
-recover those files in hdsf see the section on HDFS. Once these recovery efforts
-have been exhausted, the next step depends on where the missing file(s) are
-located. Different actions are required when the bad files are in Accumulo data
-table files or if they are metadata table files.</p>
-
-<p><em>Data File Corruption</em></p>
-
-<p>When an Accumulo data file is corrupt, the most reliable way to restore Accumulo
-operations is to replace the missing file with an ``empty’’ file so that
-references to the file in the METADATA table and within the tablet server
-hosting the file can be resolved by Accumulo. An empty file can be created using
-the CreateEmpty utiity:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ accumulo org.apache.accumulo.core.file.rfile.CreateEmpty /path/to/empty/file/empty.rf
-</code></pre>
-</div>
-
-<p>The process is to delete the corrupt file and then move the empty file into its
-place (The generated empty file can be copied and used multiple times if necessary and does not need
-to be regenerated each time)</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ hadoop fs –rm /accumulo/tables/corrupt/file/thename.rf; \
-hadoop fs -mv /path/to/empty/file/empty.rf /accumulo/tables/corrupt/file/thename.rf
-</code></pre>
-</div>
-
-<p><em>Metadata File Corruption</em></p>
-
-<p>If the corrupt files are metadata files, see «metadata» (under the path
-<code class="highlighter-rouge">/accumulo/tables/!0</code>) then you will need to rebuild
-the metadata table by initializing a new instance of Accumulo and then importing
-all of the existing data into the new instance. This is the same procedure as
-recovering from a zookeeper failure (see «zookeeper_failure»), except that
-you will have the benefit of having the existing user and table authorizations
-that are maintained in zookeeper.</p>
-
-<p>You can use the DumpZookeeper utility to save this information for reference
-before creating the new instance. You will not be able to use RestoreZookeeper
-because the table names and references are likely to be different between the
-original and the new instances, but it can serve as a reference.</p>
-
-<p>If the files cannot be recovered, replace corrupt data files with a empty
-rfiles to allow references in the metadata table and in the tablet servers to be
-resolved. Rebuild the metadata table if the corrupt files are metadata files.</p>
-
-<p><em>Write-Ahead Log(WAL) File Corruption</em></p>
-
-<p>In certain versions of Accumulo, a corrupt WAL file (caused by HDFS corruption
-or a bug in Accumulo that created the file) can block the successful recovery
-of one to many Tablets. Accumulo can be stuck in a loop trying to recover the
-WAL file, never being able to succeed.</p>
-
-<p>In the cases where the WAL file’s original contents are unrecoverable or some degree
-of data loss is acceptable (beware if the WAL file contains updates to the Accumulo
-metadat table!), the following process can be followed to create an valid, empty
-WAL file. Run the following commands as the Accumulo unix user (to ensure that
-the proper file permissions in HDFS)</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ echo -n -e '--- Log File Header (v2) ---\x00\x00\x00\x00' > empty.wal
-</code></pre>
-</div>
-
-<p>The above creates a file with the text “— Log File Header (v2) —” and then
-four bytes. You should verify the contents of the file with a hexdump tool.</p>
-
-<p>Then, place this empty WAL in HDFS and then replace the corrupt WAL file in HDFS
-with the empty WAL.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ hdfs dfs -moveFromLocal empty.wal /user/accumulo/empty.wal
-$ hdfs dfs -mv /user/accumulo/empty.wal /accumulo/wal/tserver-4.example.com+10011/26abec5b-63e7-40dd-9fa1-b8ad2436606e
-</code></pre>
-</div>
-
-<p>After the corrupt WAL file has been replaced, the system should automatically recover.
-It may be necessary to restart the Accumulo Master process as an exponential
-backup policy is used which could lead to a long wait before Accumulo will
-try to re-load the WAL file.</p>
-
-<h3 id="zookeeper-failure">ZooKeeper Failure</h3>
-
-<p><strong>I lost my ZooKeeper quorum (hardware failure), but HDFS is still intact. How can I recover my Accumulo instance?</strong></p>
-
-<p>ZooKeeper, in addition to its lock-service capabilities, also serves to bootstrap an Accumulo
-instance from some location in HDFS. It contains the pointers to the root tablet in HDFS which
-is then used to load the Accumulo metadata tablets, which then loads all user tables. ZooKeeper
-also stores all namespace and table configuration, the user database, the mapping of table IDs to
-table names, and more across Accumulo restarts.</p>
-
-<p>Presently, the only way to recover such an instance is to initialize a new instance and import all
-of the old data into the new instance. The easiest way to tackle this problem is to first recreate
-the mapping of table ID to table name and then recreate each of those tables in the new instance.
-Set any necessary configuration on the new tables and add some split points to the tables to close
-the gap between how many splits the old table had and no splits.</p>
-
-<p>The directory structure in HDFS for tables will follow the general structure:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>/accumulo
-/accumulo/tables/
-/accumulo/tables/1
-/accumulo/tables/1/default_tablet/A000001.rf
-/accumulo/tables/1/t-00001/A000002.rf
-/accumulo/tables/1/t-00001/A000003.rf
-/accumulo/tables/2/default_tablet/A000004.rf
-/accumulo/tables/2/t-00001/A000005.rf
-</code></pre>
-</div>
-
-<p>For each table, make a new directory that you can move (or copy if you have the HDFS space to do so)
-all of the rfiles for a given table into. For example, to process the table with an ID of <code class="highlighter-rouge">1</code>, make a new directory,
-say <code class="highlighter-rouge">/new-table-1</code> and then copy all files from <code class="highlighter-rouge">/accumulo/tables/1/\*/*.rf</code> into that directory. Additionally,
-make a directory, <code class="highlighter-rouge">/new-table-1-failures</code>, for any failures during the import process. Then, issue the import
-command using the Accumulo shell into the new table, telling Accumulo to not re-set the timestamp:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>user@instance new_table> importdirectory /new-table-1 /new-table-1-failures false
-</code></pre>
-</div>
-
-<p>Any RFiles which were failed to be loaded will be placed in <code class="highlighter-rouge">/new-table-1-failures</code>. Rfiles that were successfully
-imported will no longer exist in <code class="highlighter-rouge">/new-table-1</code>. For failures, move them back to the import directory and retry
-the <code class="highlighter-rouge">importdirectory</code> command.</p>
-
-<p>It is <em>extremely</em> important to note that this approach may introduce stale data back into
-the tables. For a few reasons, RFiles may exist in the table directory which are candidates for deletion but have
-not yet been deleted. Additionally, deleted data which was not compacted away, but still exists in write-ahead logs if
-the original instance was somehow recoverable, will be re-introduced in the new instance. Table splits and merges
-(which also include the deleteRows API call on TableOperations, are also vulnerable to this problem. This process should
-<em>not</em> be used if these are unacceptable risks. It is possible to try to re-create a view of the <code class="highlighter-rouge">accumulo.metadata</code>
-table to prune out files that are candidates for deletion, but this is a difficult task that also may not be entirely accurate.</p>
-
-<p>Likewise, it is also possible that data loss may occur from write-ahead log (WAL) files which existed on the old table but
-were not minor-compacted into an RFile. Again, it may be possible to reconstruct the state of these WAL files to
-replay data not yet in an RFile; however, this is a difficult task and is not implemented in any automated fashion.</p>
-
-<p>The <code class="highlighter-rouge">importdirectory</code> shell command can be used to import RFiles from the old instance into a newly created instance,
-but extreme care should go into the decision to do this as it may result in reintroduction of stale data or the
-omission of new data.</p>
-
-<h2 id="upgrade-issues">Upgrade Issues</h2>
-
-<p><strong>I upgraded from 1.4 to 1.5 to 1.6 but still have some WAL files on local disk. Do I have any way to recover them?</strong></p>
-
-<p>Yes, you can recover them by running the LocalWALRecovery utility (not available in 1.8 and later) on each node that needs recovery performed. The utility
-will default to using the directory specified by <code class="highlighter-rouge">logger.dir.walog</code> in your configuration, or can be
-overriden by using the <code class="highlighter-rouge">--local-wal-directories</code> option on the tool. It can be invoked as follows:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>accumulo org.apache.accumulo.tserver.log.LocalWALRecovery
-</code></pre>
-</div>
-
-<h2 id="file-naming-conventions">File Naming Conventions</h2>
-
-<p><strong>Why are files named like they are? Why do some start with <code class="highlighter-rouge">C</code> and others with <code class="highlighter-rouge">F</code>?</strong></p>
-
-<p>The file names give you a basic idea for the source of the file.</p>
-
-<p>The base of the filename is a base-36 unique number. All filenames in accumulo are coordinated
-with a counter in zookeeper, so they are always unique, which is useful for debugging.</p>
-
-<p>The leading letter gives you an idea of how the file was created:</p>
-
-<ul>
- <li><code class="highlighter-rouge">F</code> - Flush: entries in memory were written to a file (Minor Compaction)</li>
- <li><code class="highlighter-rouge">M</code> - Merging compaction: entries in memory were combined with the smallest file to create one new file</li>
- <li><code class="highlighter-rouge">C</code> - Several files, but not all files, were combined to produce this file (Major Compaction)</li>
- <li><code class="highlighter-rouge">A</code> - All files were compacted, delete entries were dropped</li>
- <li><code class="highlighter-rouge">I</code> - Bulk import, complete, sorted index files. Always in a directory starting with <code class="highlighter-rouge">b-</code></li>
-</ul>
-
-<p>This simple file naming convention allows you to see the basic structure of the files from just
-their filenames, and reason about what should be happening to them next, just
-by scanning their entries in the metadata tables.</p>
-
-<p>For example, if you see multiple files with <code class="highlighter-rouge">M</code> prefixes, the tablet is, or was, up against its
-maximum file limit, so it began merging memory updates with files to keep the file count reasonable. This
-slows down ingest performance, so knowing there are many files like this tells you that the system
-is struggling to keep up with ingest vs the compaction strategy which reduces the number of files.</p>
-
-<h2 id="hdfs-decommissioning-issues">HDFS Decommissioning Issues</h2>
-
-<p><strong>My Hadoop DataNode is hung for hours trying to decommission.</strong></p>
-
-<p>Write Ahead Logs stay open until they hit the size threshold, which could be many hours or days in some cases. These open files will prevent a DN from finishing its decommissioning process (HDFS-3599) in some versions of Hadoop 2. If you stop the DN, then the WALog file will not be closed and you could lose data. To work around this issue, we now close WALogs on a time period specified by the property <code class="highlighter-rouge">tserver.walog.max.age</code> with a default period of 24 hours.</p>
-
- </div>
-</div>
-
- </div>
-
-
-<footer>
-
- <p><a href="https://www.apache.org/foundation/contributing"><img src="https://www.apache.org/images/SupportApache-small.png" alt="Support the ASF" id="asf-logo" height="100" /></a></p>
-
- <p>Copyright © 2011-2017 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.</p>
-
-</footer>
-
-
- </div>
- </div>
- </div>
-</body>
-</html>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/3e4443c1/docs/unreleased/troubleshooting/system-metadata-tables.html
----------------------------------------------------------------------
diff --git a/docs/unreleased/troubleshooting/system-metadata-tables.html b/docs/unreleased/troubleshooting/system-metadata-tables.html
new file mode 100644
index 0000000..92d7a5a
--- /dev/null
+++ b/docs/unreleased/troubleshooting/system-metadata-tables.html
@@ -0,0 +1,474 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<meta charset="utf-8">
+<meta http-equiv="X-UA-Compatible" content="IE=edge">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<link href="https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/paper/bootstrap.min.css" rel="stylesheet" integrity="sha384-awusxf8AUojygHf2+joICySzB780jVvQaVCAt1clU3QsyAitLGul28Qxb2r1e5g+" crossorigin="anonymous">
+<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
+<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.css">
+<link href="/css/accumulo.css" rel="stylesheet" type="text/css">
+
+<title>Accumulo Documentation - System Metadata Tables</title>
+
+<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
+<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
+<script type="text/javascript" src="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.js"></script>
+<script>
+ // show location of canonical site if not currently on the canonical site
+ $(function() {
+ var host = window.location.host;
+ if (typeof host !== 'undefined' && host !== 'accumulo.apache.org') {
+ $('#non-canonical').show();
+ }
+ });
+
+ $(function() {
+ // decorate section headers with anchors
+ return $("h2, h3, h4, h5, h6").each(function(i, el) {
+ var $el, icon, id;
+ $el = $(el);
+ id = $el.attr('id');
+ icon = '<i class="fa fa-link"></i>';
+ if (id) {
+ return $el.append($("<a />").addClass("header-link").attr("href", "#" + id).html(icon));
+ }
+ });
+ });
+
+ // configure Google Analytics
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ if (ga.hasOwnProperty('loaded') && ga.loaded === true) {
+ ga('create', 'UA-50934829-1', 'apache.org');
+ ga('send', 'pageview');
+ }
+</script>
+
+</head>
+<body style="padding-top: 100px">
+
+ <nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="/"><img id="nav-logo" alt="Apache Accumulo" class="img-responsive" src="/images/accumulo-logo.png" width="200"
+ /></a>
+ </div>
+ <div class="collapse navbar-collapse" id="navbar-items">
+ <ul class="nav navbar-nav">
+ <li class="nav-link"><a href="/downloads">Download</a></li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Releases<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/release/accumulo-1.8.1/">1.8.1 (Latest)</a></li>
+ <li><a href="/release/accumulo-1.7.3/">1.7.3</a></li>
+ <li><a href="/release/accumulo-1.6.6/">1.6.6</a></li>
+ <li><a href="/release/">Archive</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/1.8/accumulo_user_manual.html">User Manual (1.8)</a></li>
+ <li><a href="/1.8/apidocs">Javadocs (1.8)</a></li>
+ <li><a href="/1.8/examples">Examples (1.8)</a></li>
+ <li><a href="/features">Features</a></li>
+ <li><a href="/glossary">Glossary</a></li>
+ <li><a href="/external-docs">External Docs</a></li>
+ <li><a href="/docs-archive/">Archive</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/get_involved">Get Involved</a></li>
+ <li><a href="/mailing_list">Mailing Lists</a></li>
+ <li><a href="/people">People</a></li>
+ <li><a href="/related-projects">Related Projects</a></li>
+ <li><a href="/contributor/">Contributor Guide</a></li>
+ </ul>
+ </li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://www.apache.org">Apache Homepage <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/licenses/LICENSE-2.0">License <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/sponsorship">Sponsorship <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/security">Security <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/thanks">Thanks <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct <i class="fa fa-external-link"></i></a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12">
+
+ <div id="non-canonical" style="display: none; background-color: #F0E68C; padding-left: 1em;">
+ Visit the official site at: <a href="https://accumulo.apache.org">https://accumulo.apache.org</a>
+ </div>
+ <div id="content">
+
+ <div class="alert alert-danger" role="alert">This documentation is for an unreleased version of Apache Accumulo that is currently under development! Check out the <a href="/1.8/accumulo_user_manual.html">documentation for the latest release</a>.</div>
+
+<div class="row">
+ <div class="col-md-3">
+ <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true">
+ <div class="panel panel-default">
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsegetting-started" aria-expanded="false" aria-controls="collapsegetting-started">
+ Getting started
+ </a>
+ </h4>
+ </div>
+ <div id="collapsegetting-started" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/design">Accumulo Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/clients">Accumulo Clients</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/shell">Accumulo Shell</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_design">Table Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_configuration">Table Configuration</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsedevelopment" aria-expanded="false" aria-controls="collapsedevelopment">
+ Development
+ </a>
+ </h4>
+ </div>
+ <div id="collapsedevelopment" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/sampling">Sampling</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/summaries">Summary Statistics</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/security">Security</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapseadministration" aria-expanded="false" aria-controls="collapseadministration">
+ Administration
+ </a>
+ </h4>
+ </div>
+ <div id="collapseadministration" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/multivolume">Multi-Volume Installations</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div>
+
+ </div>
+ </div>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <div class="panel-heading" role="tab" id="headingOne">
+ <h4 class="panel-title">
+ <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsetroubleshooting" aria-expanded="true" aria-controls="collapsetroubleshooting">
+ Troubleshooting
+ </a>
+ </h4>
+ </div>
+ <div id="collapsetroubleshooting" class="panel-collapse collapse in" role="tabpanel" aria-labelledby="headingOne">
+ <div class="panel-body">
+
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div>
+
+ <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div>
+
+ </div>
+ </div>
+
+
+
+ </div>
+ </div>
+ </div>
+ <div class="col-md-9">
+
+ <p><a href="/docs/unreleased/">Accumulo unreleased docs</a> >> Troubleshooting >> System Metadata Tables</p>
+
+
+ <h1>System Metadata Tables</h1>
+
+ <p>Accumulo tracks information about tables in metadata tables. The metadata for
+most tables is contained within the metadata table in the accumulo namespace,
+while metadata for that table is contained in the root table in the accumulo
+namespace. The root table is composed of a single tablet, which does not
+split, so it is also called the root tablet. Information about the root
+table, such as its location and write-ahead logs, are stored in ZooKeeper.</p>
+
+<p>Let’s create a table and put some data into it:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>shell> createtable test
+
+shell> tables -l
+accumulo.metadata => !0
+accumulo.root => `r
+test => 2
+trace => 1
+
+shell> insert a b c d
+
+shell> flush -w
+</code></pre>
+</div>
+
+<p>Now let’s take a look at the metadata for this table:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>shell> table accumulo.metadata
+shell> scan -b 3; -e 3<
+3< file:/default_tablet/F000009y.rf [] 186,1
+3< last:13fe86cd27101e5 [] 127.0.0.1:9997
+3< loc:13fe86cd27101e5 [] 127.0.0.1:9997
+3< log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995|6
+3< srv:dir [] /default_tablet
+3< srv:flush [] 1
+3< srv:lock [] tservers/127.0.0.1:9997/zlock-0000000001$13fe86cd27101e5
+3< srv:time [] M1373998392323
+3< ~tab:~pr [] \x00
+</code></pre>
+</div>
+
+<p>Let’s decode this little session:</p>
+
+<ul>
+ <li>
+ <p><code class="highlighter-rouge">scan -b 3; -e 3<</code> - Every tablet gets its own row. Every row starts with the table id followed by
+ <code class="highlighter-rouge">;</code> or <code class="highlighter-rouge"><</code>, and followed by the end row split point for that tablet.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">file:/default_tablet/F000009y.rf [] 186,1</code> -
+ File entry for this tablet. This tablet contains a single file reference. The
+ file is <code class="highlighter-rouge">/accumulo/tables/3/default_tablet/F000009y.rf</code>. It contains 1
+ key/value pair, and is 186 bytes long.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">last:13fe86cd27101e5 [] 127.0.0.1:9997</code> -
+ Last location for this tablet. It was last held on 127.0.0.1:9997, and the
+ unique tablet server lock data was `13fe86cd27101e5+. The default balancer
+ will tend to put tablets back on their last location.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">loc:13fe86cd27101e5 [] 127.0.0.1:9997</code> -
+ The current location of this tablet.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0. ...</code> -
+ This tablet has a reference to a single write-ahead log. This file can be found in
+ <code class="highlighter-rouge">/accumulo/wal/127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995</code>. The value
+ of this entry could refer to multiple files. This tablet’s data is encoded as
+ <code class="highlighter-rouge">6</code> within the log.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">srv:dir [] /default_tablet</code> -
+ Files written for this tablet will be placed into
+ <code class="highlighter-rouge">/accumulo/tables/3/default_tablet</code>.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">srv:flush [] 1</code> -
+ Flush id. This table has successfully completed the flush with the id of +1+.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">srv:lock [] tservers/127.0.0.1:9997/zlock-0000000001\$13fe86cd27101e5</code> -
+ This is the lock information for the tablet holding the present lock. This
+ information is checked against zookeeper whenever this is updated, which
+ prevents a metadata update from a tablet server that no longer holds its
+ lock.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">srv:time [] M1373998392323</code> -
+ This indicates the time time type (+M+ for milliseconds or +L+ for logical) and the timestamp of the most recently written key in this tablet. It is used to ensure automatically assigned key timestamps are strictly increasing for the tablet, regardless of the tablet server’s system time.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">~tab:~pr [] \x00</code> -
+ The end-row marker for the previous tablet (prev-row). The first byte
+ indicates the presence of a prev-row. This tablet has the range (-inf, `inf),
+ so it has no prev-row (or end row).</p>
+ </li>
+</ul>
+
+<p>Besides these columns, you may see:</p>
+
+<ul>
+ <li>
+ <p><code class="highlighter-rouge">rowId future:zooKeeperID location</code> -
+ Tablet has been assigned to a tablet, but not yet loaded.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">~del:filename</code> -
+ When a tablet server is done use a file, it will create a delete marker in the appropriate metadata table, unassociated with any tablet. The garbage collector will remove the marker, and the file, when no other reference to the file exists.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">~blip:txid</code> -
+ Bulk-Load In Progress marker.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">rowId loaded:filename</code> -
+ A file has been bulk-loaded into this tablet, however the bulk load has not yet completed on other tablets, so this marker prevents the file from being loaded multiple times.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">rowId !cloned</code> -
+ A marker that indicates that this tablet has been successfully cloned.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">rowId splitRatio:ratio</code> -
+ A marker that indicates a split is in progress, and the files are being split at the given ratio.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">rowId chopped</code> -
+ A marker that indicates that the files in the tablet do not contain keys outside the range of the tablet.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">rowId scan</code> -
+ A marker that prevents a file from being removed while there are still active scans using it.</p>
+ </li>
+</ul>
+
+
+ </div>
+</div>
+
+ </div>
+
+
+<footer>
+
+ <p><a href="https://www.apache.org/foundation/contributing"><img src="https://www.apache.org/images/SupportApache-small.png" alt="Support the ASF" id="asf-logo" height="100" /></a></p>
+
+ <p>Copyright © 2011-2017 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.</p>
+
+</footer>
+
+
+ </div>
+ </div>
+ </div>
+</body>
+</html>