You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by bi...@apache.org on 2014/05/08 04:16:20 UTC
[3/5] ACCUMULO-1327 converted latex manual to asciidoc
http://git-wip-us.apache.org/repos/asf/accumulo/blob/900d6abb/docs/src/main/asciidoc/chapters/troubleshooting.txt
----------------------------------------------------------------------
diff --git a/docs/src/main/asciidoc/chapters/troubleshooting.txt b/docs/src/main/asciidoc/chapters/troubleshooting.txt
new file mode 100644
index 0000000..eec75fa
--- /dev/null
+++ b/docs/src/main/asciidoc/chapters/troubleshooting.txt
@@ -0,0 +1,758 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+## Troubleshooting
+
+### Logs
+
+*Q*: The tablet server does not seem to be running!? What happened?
+
+Accumulo is a distributed system. It is supposed to run on remote
+equipment, across hundreds of computers. Each program that runs on
+these remote computers writes down events as they occur, into a local
+file. By default, this is defined in
++$ACCUMULO_HOME/conf/accumule-env.sh+ as +ACCUMULO_LOG_DIR+.
+
+*A*: Look in the +$ACCUMULO_LOG_DIR/tserver*.log+ file. Specifically, check the end of the file.
+
+*Q*: The tablet server did not start and the debug log does not exists! What happened?
+
+When the individual programs are started, the stdout and stderr output
+of these programs are stored in +.out+ and +.err+ files in
++$ACCUMULO_LOG_DIR+. Often, when there are missing configuration
+options, files or permissions, messages will be left in these files.
+
+*A*: Probably a start-up problem. Look in +$ACCUMULO_LOG_DIR/tserver*.err+
+
+### Monitor
+
+*Q*: Accumulo is not working, what's wrong?
+
+There's a small web server that collects information about all the
+components that make up a running Accumulo instance. It will highlight
+unusual or unexpected conditions.
+
+*A*: Point your browser to the monitor (typically the master host, on port 50095). Is anything red or yellow?
+
+*Q*: My browser is reporting connection refused, and I cannot get to the monitor
+
+The monitor program's output is also written to .err and .out files in
+the +$ACCUMULO_LOG_DIR+. Look for problems in this file if the
++$ACCUMULO_LOG_DIR/monitor*.log+ file does not exist.
+
+*A*: The monitor program is probably not running. Check the log files for errors.
+
+*Q*: My browser hangs trying to talk to the monitor.
+
+Your browser needs to be able to reach the monitor program. Often
+large clusters are firewalled, or use a VPN for internal
+communications. You can use SSH to proxy your browser to the cluster,
+or consult with your system administrator to gain access to the server
+from your browser.
+
+It is sometimes helpful to use a text-only browser to sanity-check the
+monitor while on the machine running the monitor:
+
+ $ links http://localhost:50095
+
+*A*: Verify that you are not firewalled from the monitor if it is running on a remote host.
+
+*Q*: The monitor responds, but there are no numbers for tservers and tables. The summary page says the master is down.
+
+The monitor program gathers all the details about the master and the
+tablet servers through the master. It will be mostly blank if the
+master is down.
+
+*A*: Check for a running master.
+
+### HDFS
+
+Accumulo reads and writes to the Hadoop Distributed File System.
+Accumulo needs this file system available at all times for normal operations.
+
+*Q*: Accumulo is having problems ``getting a block blk_1234567890123.'' How do I fix it?
+
+This troubleshooting guide does not cover HDFS, but in general, you
+want to make sure that all the datanodes are running and an fsck check
+finds the file system clean:
+
+ $ hadoop fsck /accumulo
+
+You can use:
+
+ $ hadoop fsck /accumulo/path/to/corrupt/file -locations -blocks -files
+
+to locate the block references of individual corrupt files and use those
+references to search the name node and individual data node logs to determine which
+servers those blocks have been assigned and then try to fix any underlying file
+system issues on those nodes.
+
+On a larger cluster, you may need to increase the number of Xcievers for HDFS DataNodes:
+
+[source,xml]
+<property>
+ <name>dfs.datanode.max.xcievers</name>
+ <value>4096</value>
+</property>
+
+*A*: Verify HDFS is healthy, check the datanode logs.
+
+### Zookeeper
+
+*Q*: +accumulo init+ is hanging. It says something about talking to zookeeper.
+
+Zookeeper is also a distributed service. You will need to ensure that
+it is up. You can run the zookeeper command line tool to connect to
+any one of the zookeeper servers:
+
+ $ zkCli.sh -server zoohost
+ ...
+ [zk: zoohost:2181(CONNECTED) 0]
+
+It is important to see the word +CONNECTED+! If you only see
++CONNECTING+ you will need to diagnose zookeeper errors.
+
+*A*: Check to make sure that zookeeper is up, and that
++$ACCUMULO_HOME/conf/accumulo-site.xml+ has been pointed to
+your zookeeper server(s).
+
+*Q*: Zookeeper is running, but it does not say +CONNECTED+
+
+Zookeeper processes talk to each other to elect a leader. All updates
+go through the leader and propagate to a majority of all the other
+nodes. If a majority of the nodes cannot be reached, zookeeper will
+not allow updates. Zookeeper also limits the number connections to a
+server from any other single host. By default, this limit can be as small as 10
+and can be reached in some everything-on-one-machine test configurations.
+
+You can check the election status and connection status of clients by
+asking the zookeeper nodes for their status. You connect to zookeeper
+and ask it with the four-letter +stat+ command:
+
+----
+$ nc zoohost 2181
+stat
+Zookeeper version: 3.4.5-1392090, built on 09/30/2012 17:52 GMT
+Clients:
+ /127.0.0.1:58289[0](queued=0,recved=1,sent=0)
+ /127.0.0.1:60231[1](queued=0,recved=53910,sent=53915)
+
+Latency min/avg/max: 0/5/3008
+Received: 1561459
+Sent: 1561592
+Connections: 2
+Outstanding: 0
+Zxid: 0x621a3b
+Mode: standalone
+Node count: 22524
+----
+
+*A*: Check zookeeper status, verify that it has a quorum, and has not exceeded maxClientCnxns.
+
+*Q*: My tablet server crashed! The logs say that it lost its zookeeper lock.
+
+Tablet servers reserve a lock in zookeeper to maintain their ownership
+over the tablets that have been assigned to them. Part of their
+responsibility for keeping the lock is to send zookeeper a keep-alive
+message periodically. If the tablet server fails to send a message in
+a timely fashion, zookeeper will remove the lock and notify the tablet
+server. If the tablet server does not receive a message from
+zookeeper, it will assume its lock has been lost, too. If a tablet
+server loses its lock, it kills itself: everything assumes it is dead
+already.
+
+*A*: Investigate why the tablet server did not send a timely message to
+zookeeper.
+
+#### Keeping the tablet server lock
+
+*Q*: My tablet server lost its lock. Why?
+
+The primary reason a tablet server loses its lock is that it has been pushed into swap.
+
+A large java program (like the tablet server) may have a large portion
+of its memory image unused. The operation system will favor pushing
+this allocated, but unused memory into swap so that the memory can be
+re-used as a disk buffer. When the java virtual machine decides to
+access this memory, the OS will begin flushing disk buffers to return that
+memory to the VM. This can cause the entire process to block long
+enough for the zookeeper lock to be lost.
+
+*A*: Configure your system to reduce the kernel parameter _swappiness_ from the default (60) to zero.
+
+*Q*: My tablet server lost its lock, and I have already set swappiness to
+zero. Why?
+
+Be careful not to over-subscribe memory. This can be easy to do if
+your accumulo processes run on the same nodes as hadoop's map-reduce
+framework. Remember to add up:
+
+* size of the JVM for the tablet server
+* size of the in-memory map, if using the native map implementation
+* size of the JVM for the data node
+* size of the JVM for the task tracker
+* size of the JVM times the maximum number of mappers and reducers
+* size of the kernel and any support processes
+
+If a 16G node can run 2 mappers and 2 reducers, and each can be 2G,
+then there is only 8G for the data node, tserver, task tracker and OS.
+
+*A*: Reduce the memory footprint of each component until it fits comfortably.
+
+*Q*: My tablet server lost its lock, swappiness is zero, and my node has lots of unused memory!
+
+The JVM memory garbage collector may fall behind and cause a
+"stop-the-world" garbage collection. On a large memory virtual
+machine, this collection can take a long time. This happens more
+frequently when the JVM is getting low on free memory. Check the logs
+of the tablet server. You will see lines like this:
+
+ 2013-06-20 13:43:20,607 [tabletserver.TabletServer] DEBUG: gc ParNew=0.00(+0.00) secs
+ ConcurrentMarkSweep=0.00(+0.00) secs freemem=1,868,325,952(+1,868,325,952) totalmem=2,040,135,680
+
+When +freemem+ becomes small relative to the amount of memory
+needed, the JVM will spend more time finding free memory than
+performing work. This can cause long delays in sending keep-alive
+messages to zookeeper.
+
+*A*: Ensure the tablet server JVM is not running low on memory.
+
+### Tools
+
+The accumulo script can be used to run classes from the command line.
+This section shows how a few of the utilities work, but there are many
+more.
+
+There's a class that will examine an accumulo storage file and print
+out basic metadata.
+
+----
+$ ./bin/accumulo org.apache.accumulo.core.file.rfile.PrintInfo /accumulo/tables/1/default_tablet/A000000n.rf
+2013-07-16 08:17:14,778 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
+Locality group : <DEFAULT>
+ Start block : 0
+ Num blocks : 1
+ Index level 0 : 62 bytes 1 blocks
+ First key : 288be9ab4052fe9e span:34078a86a723e5d3:3da450f02108ced5 [] 1373373521623 false
+ Last key : start:13fc375709e id:615f5ee2dd822d7a [] 1373373821660 false
+ Num entries : 466
+ Column families : [waitForCommits, start, md major compactor 1, md major compactor 2, md major compactor 3,
+ bringOnline, prep, md major compactor 4, md major compactor 5, md root major compactor 3,
+ minorCompaction, wal, compactFiles, md root major compactor 4, md root major compactor 1,
+ md root major compactor 2, compact, id, client:update, span, update, commit, write,
+ majorCompaction]
+
+Meta block : BCFile.index
+ Raw size : 4 bytes
+ Compressed size : 12 bytes
+ Compression type : gz
+
+Meta block : RFile.index
+ Raw size : 780 bytes
+ Compressed size : 344 bytes
+ Compression type : gz
+----
+
+When trying to diagnose problems related to key size, the +PrintInfo+ tool can provide a histogram of the individual key sizes:
+
+ $ ./bin/accumulo org.apache.accumulo.core.file.rfile.PrintInfo --histogram /accumulo/tables/1/default_tablet/A000000n.rf
+ ...
+ Up to size count %-age
+ 10 : 222 28.23%
+ 100 : 244 71.77%
+ 1000 : 0 0.00%
+ 10000 : 0 0.00%
+ 100000 : 0 0.00%
+ 1000000 : 0 0.00%
+ 10000000 : 0 0.00%
+ 100000000 : 0 0.00%
+ 1000000000 : 0 0.00%
+ 10000000000 : 0 0.00%
+
+Likewise, +PrintInfo+ will dump the key-value pairs and show you the contents of the RFile:
+
+ $ ./bin/accumulo org.apache.accumulo.core.file.rfile.PrintInfo --dump /accumulo/tables/1/default_tablet/A000000n.rf
+ row columnFamily:columnQualifier [visibility] timestamp deleteFlag -> Value
+ ...
+
+*Q*: Accumulo is not showing me any data!
+
+*A*: Do you have your auths set so that it matches your visibilities?
+
+*Q*: What are my visibilities?
+
+*A*: Use +PrintInfo+ on a representative file to get some idea of the visibilities in the underlying data.
+
+Note that the use of +PrintInfo+ is an administrative tool and can only
+by used by someone who can access the underlying Accumulo data. It
+does not provide the normal access controls in Accumulo.
+
+If you would like to backup, or otherwise examine the contents of Zookeeper, there are commands to dump and load to/from XML.
+
+ $ ./bin/accumulo org.apache.accumulo.server.util.DumpZookeeper --root /accumulo >dump.xml
+ $ ./bin/accumulo org.apache.accumulo.server.util.RestoreZookeeper --overwrite < dump.xml
+
+*Q*: How can I get the information in the monitor page for my cluster monitoring system?
+
+*A*: Use GetMasterStats:
+
+ $ ./bin/accumulo org.apache.accumulo.test.GetMasterStats | grep Load
+ OS Load Average: 0.27
+
+*Q*: The monitor page is showing an offline tablet. How can I find out which tablet it is?
+
+*A*: Use FindOfflineTablets:
+
+ $ ./bin/accumulo org.apache.accumulo.server.util.FindOfflineTablets
+ 2<<@(null,null,localhost:9997) is UNASSIGNED #walogs:2
+
+Here's what the output means:
+
++2<<+::
+ This is the tablet from (-inf, pass:[+]inf) for the
+ table with id 2. The command +tables -l+ in the shell will show table ids for
+ tables.
+
++@(null, null, localhost:9997)+::
+ Location information. The
+ format is +@(assigned, hosted, last)+. In this case, the
+ tablet has not been assigned, is not hosted anywhere, and was once
+ hosted on localhost.
+
++#walogs:2+::
+ The number of write-ahead logs that this tablet requires for recovery.
+
+An unassigned tablet with write-ahead logs is probably waiting for
+logs to be sorted for efficient recovery.
+
+*Q*: How can I be sure that the metadata tables are up and consistent?
+
+*A*: +CheckForMetadataProblems+ will verify the start/end of
+every tablet matches, and the start and stop for the table is empty:
+
+ $ ./bin/accumulo org.apache.accumulo.server.util.CheckForMetadataProblems -u root --password
+ Enter the connection password:
+ All is well for table !0
+ All is well for table 1
+
+*Q*: My hadoop cluster has lost a file due to a NameNode failure. How can I remove the file?
+
+*A*: There's a utility that will check every file reference and ensure
+that the file exists in HDFS. Optionally, it will remove the
+reference:
+
+ $ ./bin/accumulo org.apache.accumulo.server.util.RemoveEntriesForMissingFiles -u root --password
+ Enter the connection password:
+ 2013-07-16 13:10:57,293 [util.RemoveEntriesForMissingFiles] INFO : File /accumulo/tables/2/default_tablet/F0000005.rf
+ is missing
+ 2013-07-16 13:10:57,296 [util.RemoveEntriesForMissingFiles] INFO : 1 files of 3 missing
+
+*Q*: I have many entries in zookeeper for old instances I no longer need. How can I remove them?
+
+*A*: Use CleanZookeeper:
+
+ $ ./bin/accumulo org.apache.accumulo.server.util.CleanZookeeper
+
+This command will not delete the instance pointed to by the local +conf/accumulo-site.xml+ file.
+
+*Q*: I need to decommission a node. How do I stop the tablet server on it?
+
+*A*: Use the admin command:
+
+ $ ./bin/accumulo admin stop hostname:9997
+ 2013-07-16 13:15:38,403 [util.Admin] INFO : Stopping server 12.34.56.78:9997
+
+*Q*: I cannot login to a tablet server host, and the tablet server will not shut down. How can I kill the server?
+
+*A*: Sometimes you can kill a "stuck" tablet server by deleting its lock in zookeeper:
+
+ $ ./bin/accumulo org.apache.accumulo.server.util.TabletServerLocks --list
+ 127.0.0.1:9997 TSERV_CLIENT=127.0.0.1:9997
+ $ ./bin/accumulo org.apache.accumulo.server.util.TabletServerLocks -delete 127.0.0.1:9997
+ $ ./bin/accumulo org.apache.accumulo.server.util.TabletServerLocks -list
+ 127.0.0.1:9997 null
+
+You can find the master and instance id for any accumulo instances using the same zookeeper instance:
+
+----
+$ ./bin/accumulo org.apache.accumulo.server.util.ListInstances
+INFO : Using ZooKeepers localhost:2181
+
+ Instance Name | Instance ID | Master
+---------------------+--------------------------------------+-------------------------------
+ "test" | 6140b72e-edd8-4126-b2f5-e74a8bbe323b | 127.0.0.1:9999
+----
+
+[[metadata]]
+### System Metadata Tables
+
+Accumulo tracks information about tables in metadata tables. The metadata for
+most tables is contained within the metadata table in the accumulo namespace,
+while metadata for that table is contained in the root table in the accumulo
+namespace. The root table is composed of a single tablet, which does not
+split, so it is also called the root tablet. Information about the root
+table, such as its location and write-ahead logs, are stored in ZooKeeper.
+
+Let's create a table and put some data into it:
+
+----
+shell> createtable test
+
+shell> tables -l
+accumulo.metadata => !0
+accumulo.root => +r
+test => 2
+trace => 1
+
+shell> insert a b c d
+
+shell> flush -w
+----
+
+Now let's take a look at the metadata for this table:
+
+ shell> table accumulo.metadata
+ shell> scan -b 3; -e 3<
+ 3< file:/default_tablet/F000009y.rf [] 186,1
+ 3< last:13fe86cd27101e5 [] 127.0.0.1:9997
+ 3< loc:13fe86cd27101e5 [] 127.0.0.1:9997
+ 3< log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995|6
+ 3< srv:dir [] /default_tablet
+ 3< srv:flush [] 1
+ 3< srv:lock [] tservers/127.0.0.1:9997/zlock-0000000001$13fe86cd27101e5
+ 3< srv:time [] M1373998392323
+ 3< ~tab:~pr [] \x00
+
+Let's decode this little session:
+
++scan -b 3; -e 3<+:: Every tablet gets its own row. Every row starts with the table id followed by
+ +;+ or +<+, and followed by the end row split point for that tablet.
+
++file:/default_tablet/F000009y.rf [] 186,1+::
+ File entry for this tablet. This tablet contains a single file reference. The
+ file is +/accumulo/tables/3/default_tablet/F000009y.rf+. It contains 1
+ key/value pair, and is 186 bytes long.
+
++last:13fe86cd27101e5 [] 127.0.0.1:9997+::
+ Last location for this tablet. It was last held on 127.0.0.1:9997, and the
+ unique tablet server lock data was +13fe86cd27101e5+. The default balancer
+ will tend to put tablets back on their last location.
+
++loc:13fe86cd27101e5 [] 127.0.0.1:9997+::
+ The current location of this tablet.
+
++log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0. ...+::
+ This tablet has a reference to a single write-ahead log. This file can be found in
+ +/accumulo/wal/127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995+. The value
+ of this entry could refer to multiple files. This tablet's data is encoded as
+ +6+ within the log.
+
++srv:dir [] /default_tablet+::
+ Files written for this tablet will be placed into
+ +/accumulo/tables/3/default_tablet+.
+
++srv:flush [] 1+::
+ Flush id. This table has successfully completed the flush with the id of +1+.
+
++srv:lock [] tservers/127.0.0.1:9997/zlock-0000000001\$13fe86cd27101e5+::
+ This is the lock information for the tablet holding the present lock. This
+ information is checked against zookeeper whenever this is updated, which
+ prevents a metadata update from a tablet server that no longer holds its
+ lock.
+
++srv:time [] M1373998392323+::
+ This indicates the time time type (+M+ for milliseconds or +L+ for logical) and the timestamp of the most recently written key in this tablet. It is used to ensure automatically assigned key timestamps are strictly increasing for the tablet, regardless of the tablet server's system time.
+
+`~tab:~pr [] \x00`::
+ The end-row marker for the previous tablet (prev-row). The first byte
+ indicates the presence of a prev-row. This tablet has the range (-inf, +inf),
+ so it has no prev-row (or end row).
+
+Besides these columns, you may see:
+
++rowId future:zooKeeperID location+::
+ Tablet has been assigned to a tablet, but not yet loaded.
+
++~del:filename+::
+ When a tablet server is done use a file, it will create a delete marker in the appropriate metadata table, unassociated with any tablet. The garbage collector will remove the marker, and the file, when no other reference to the file exists.
+
++~blip:txid+::
+ Bulk-Load In Progress marker.
+
++rowId loaded:filename+::
+ A file has been bulk-loaded into this tablet, however the bulk load has not yet completed on other tablets, so this marker prevents the file from being loaded multiple times.
+
++rowId !cloned+::
+ A marker that indicates that this tablet has been successfully cloned.
+
++rowId splitRatio:ratio+::
+ A marker that indicates a split is in progress, and the files are being split at the given ratio.
+
++rowId chopped+::
+ A marker that indicates that the files in the tablet do not contain keys outside the range of the tablet.
+
++rowId scan+::
+ A marker that prevents a file from being removed while there are still active scans using it.
+
+### Simple System Recovery
+
+*Q*: One of my Accumulo processes died. How do I bring it back?
+
+The easiest way to bring all services online for an Accumulo instance is to run the +start-all.sh+ script.
+
+ $ bin/start-all.sh
+
+This process will check the process listing, using +jps+ on each host before attempting to restart a service on the given host.
+Typically, this check is sufficient except in the face of a hung/zombie process. For large clusters, it may be
+undesirable to ssh to every node in the cluster to ensure that all hosts are running the appropriate processes and +start-here.sh+ may be of use.
+
+ $ ssh host_with_dead_process
+ $ bin/start-here.sh
+
++start-here.sh+ should be invoked on the host which is missing a given process. Like start-all.sh, it will start all
+necessary processes that are not currently running, but only on the current host and not cluster-wide. Tools such as +pssh+ or
++pdsh+ can be used to automate this process.
+
++start-server.sh+ can also be used to start a process on a given host; however, it is not generally recommended for
+users to issue this directly as the +start-all.sh+ and +start-here.sh+ scripts provide the same functionality with
+more automation and are less prone to user error.
+
+*A*: Use +start-all.sh+ or +start-here.sh+.
+
+*Q*: My process died again. Should I restart it via +cron+ or tools like +supervisord+?
+
+*A*: A repeatedly dying Accumulo process is a sign of a larger problem. Typically these problems are due to a
+misconfiguration of Accumulo or over-saturation of resources. Blind automation of any service restart inside of Accumulo
+is generally an undesirable situation as it is indicative of a problem that is being masked and ignored. Accumulo
+processes should be stable on the order of months and not require frequent restart.
+
+
+### Advanced System Recovery
+
+#### HDFS Failure
+*Q*: I had disasterous HDFS failure. After bringing everything back up, several tablets refuse to go online.
+
+Data written to tablets is written into memory before being written into indexed files. In case the server
+is lost before the data is saved into a an indexed file, all data stored in memory is first written into a
+write-ahead log (WAL). When a tablet is re-assigned to a new tablet server, the write-ahead logs are read to
+recover any mutations that were in memory when the tablet was last hosted.
+
+If a write-ahead log cannot be read, then the tablet is not re-assigned. All it takes is for one of
+the blocks in the write-ahead log to be missing. This is unlikely unless multiple data nodes in HDFS have been
+lost.
+
+*A*: Get the WAL files online and healthy. Restore any data nodes that may be down.
+
+*Q*: How do find out which tablets are offline?
+
+*A*: Use +accumulo admin checkTablets+
+
+ $ bin/accumulo admin checkTablets
+
+*Q*: I lost three data nodes, and I'm missing blocks in a WAL. I don't care about data loss, how
+can I get those tablets online?
+
+See the discussion in <<metadata>>, which shows a typical metadata table listing.
+The entries with a column family of +log+ are references to the WAL for that tablet.
+If you know what WAL is bad, you can find all the references with a grep in the shell:
+
+ shell> grep 0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995
+ 3< log:127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995 [] 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995|6
+
+*A*: You can remove the WAL references in the metadata table.
+
+ shell> grant -u root Table.WRITE -t accumulo.metadata
+ shell> delete 3< log 127.0.0.1+9997/0cb7ce52-ac46-4bf7-ae1d-acdcfaa97995
+
+Note: the colon (+:+) is omitted when specifying the _row cf cq_ for the delete command.
+
+The master will automatically discover the tablet no longer has a bad WAL reference and will
+assign the tablet. You will need to remove the reference from all the tablets to get them
+online.
+
+
+*Q*: The metadata (or root) table has references to a corrupt WAL.
+
+This is a much more serious state, since losing updates to the metadata table will result
+in references to old files which may not exist, or lost references to new files, resulting
+in tablets that cannot be read, or large amounts of data loss.
+
+The best hope is to restore the WAL by fixing HDFS data nodes and bringing the data back online.
+If this is not possible, the best approach is to re-create the instance and bulk import all files from
+the old instance into a new tables.
+
+A complete set of instructions for doing this is outside the scope of this guide,
+but the basic approach is:
+
+* Use +tables -l+ in the shell to discover the table name to table id mapping
+* Stop all accumulo processes on all nodes
+* Move the accumulo directory in HDFS out of the way:
+ $ hadoop fs -mv /accumulo /corrupt
+* Re-initalize accumulo
+* Recreate tables, users and permissions
+* Import the directories under +/corrupt/tables/<id>+ into the new instance
+
+*Q*: One or more HDFS Files under /accumulo/tables are corrupt
+
+Accumulo maintains multiple references into the tablet files in the metadata
+tables and within the tablet server hosting the file, this makes it difficult to
+reliably just remove those references.
+
+The directory structure in HDFS for tables will follow the general structure:
+
+ /accumulo
+ /accumulo/tables/
+ /accumulo/tables/!0
+ /accumulo/tables/!0/default_tablet/A000001.rf
+ /accumulo/tables/!0/t-00001/A000002.rf
+ /accumulo/tables/1
+ /accumulo/tables/1/default_tablet/A000003.rf
+ /accumulo/tables/1/t-00001/A000004.rf
+ /accumulo/tables/1/t-00001/A000005.rf
+ /accumulo/tables/2/default_tablet/A000006.rf
+ /accumulo/tables/2/t-00001/A000007.rf
+
+If files under +/accumulo/tables+ are corrupt, the best course of action is to
+recover those files in hdsf see the section on HDFS. Once these recovery efforts
+have been exhausted, the next step depends on where the missing file(s) are
+located. Different actions are required when the bad files are in Accumulo data
+table files or if they are metadata table files.
+
+*Data File Corruption*
+
+When an Accumulo data file is corrupt, the most reliable way to restore Accumulo
+operations is to replace the missing file with an ``empty'' file so that
+references to the file in the METADATA table and within the tablet server
+hosting the file can be resolved by Accumulo. An empty file can be created using
+the CreateEmpty utiity:
+
+ $ accumulo org.apache.accumulo.core.file.rfile.CreateEmpty /path/to/empty/file/empty.rf
+
+The process is to delete the corrupt file and then move the empty file into its
+place (The generated empty file can be copied and used multiple times if necessary and does not need
+to be regenerated each time)
+
+ $ hadoop fs –rm /accumulo/tables/corrupt/file/thename.rf; \
+ hadoop fs -mv /path/to/empty/file/empty.rf /accumulo/tables/corrupt/file/thename.rf
+
+*Metadata File Corruption*
+
+If the corrupt files are metadata files, see <<metadata>> (under the path
++/accumulo/tables/!0+) then you will need to rebuild
+the metadata table by initializing a new instance of Accumulo and then importing
+all of the existing data into the new instance. This is the same procedure as
+recovering from a zookeeper failure (see <<zookeeper_failure>>), except that
+you will have the benefit of having the existing user and table authorizations
+that are maintained in zookeeper.
+
+You can use the DumpZookeeper utility to save this information for reference
+before creating the new instance. You will not be able to use RestoreZookeeper
+because the table names and references are likely to be different between the
+original and the new instances, but it can serve as a reference.
+
+*A*: If the files cannot be recovered, replace corrupt data files with a empty
+rfiles to allow references in the metadata table and in the tablet servers to be
+resolved. Rebuild the metadata table if the corrupt files are metadata files.
+
+[[zookeeper_failure]]
+#### ZooKeeper Failure
+*Q*: I lost my ZooKeeper quorum (hardware failure), but HDFS is still intact. How can I recover my Accumulo instance?
+
+ZooKeeper, in addition to its lock-service capabilities, also serves to bootstrap an Accumulo
+instance from some location in HDFS. It contains the pointers to the root tablet in HDFS which
+is then used to load the Accumulo metadata tablets, which then loads all user tables. ZooKeeper
+also stores all namespace and table configuration, the user database, the mapping of table IDs to
+table names, and more across Accumulo restarts.
+
+Presently, the only way to recover such an instance is to initialize a new instance and import all
+of the old data into the new instance. The easiest way to tackle this problem is to first recreate
+the mapping of table ID to table name and then recreate each of those tables in the new instance.
+Set any necessary configuration on the new tables and add some split points to the tables to close
+the gap between how many splits the old table had and no splits.
+
+The directory structure in HDFS for tables will follow the general structure:
+
+ /accumulo
+ /accumulo/tables/
+ /accumulo/tables/1
+ /accumulo/tables/1/default_tablet/A000001.rf
+ /accumulo/tables/1/t-00001/A000002.rf
+ /accumulo/tables/1/t-00001/A000003.rf
+ /accumulo/tables/2/default_tablet/A000004.rf
+ /accumulo/tables/2/t-00001/A000005.rf
+
+For each table, make a new directory that you can move (or copy if you have the HDFS space to do so)
+all of the rfiles for a given table into. For example, to process the table with an ID of +1+, make a new directory,
+say +/new-table-1+ and then copy all files from +/accumulo/tables/1/*/*.rf+ into that directory. Additionally,
+make a directory, +/new-table-1-failures+, for any failures during the import process. Then, issue the import
+command using the Accumulo shell into the new table, telling Accumulo to not re-set the timestamp:
+
+ user@instance new_table> importdirectory /new-table-1 /new-table-1-failures false
+
+Any RFiles which were failed to be loaded will be placed in +/new-table-1-failures+. Rfiles that were successfully
+imported will no longer exist in +/new-table-1+. For failures, move them back to the import directory and retry
+the +importdirectory+ command.
+
+It is *extremely* important to note that this approach may introduce stale data back into
+the tables. For a few reasons, RFiles may exist in the table directory which are candidates for deletion but have
+not yet been deleted. Additionally, deleted data which was not compacted away, but still exists in write-ahead logs if
+the original instance was somehow recoverable, will be re-introduced in the new instance. Table splits and merges
+(which also include the deleteRows API call on TableOperations, are also vulnerable to this problem. This process should
+*not* be used if these are unacceptable risks. It is possible to try to re-create a view of the +accumulo.metadata+
+table to prune out files that are candidates for deletion, but this is a difficult task that also may not be entirely accurate.
+
+Likewise, it is also possible that data loss may occur from write-ahead log (WAL) files which existed on the old table but
+were not minor-compacted into an RFile. Again, it may be possible to reconstruct the state of these WAL files to
+replay data not yet in an RFile; however, this is a difficult task and is not implemented in any automated fashion.
+
+*A*: The +importdirectory+ shell command can be used to import RFiles from the old instance into a newly created instance,
+but extreme care should go into the decision to do this as it may result in reintroduction of stale data or the
+omission of new data.
+
+### File Naming Conventions
+
+*Q*: Why are files named like they are? Why do some start with +C+ and others with +F+?
+
+*A*: The file names give you a basic idea for the source of the file.
+
+The base of the filename is a base-36 unique number. All filenames in accumulo are coordinated
+with a counter in zookeeper, so they are always unique, which is useful for debugging.
+
+The leading letter gives you an idea of how the file was created:
+
++F+::
+ Flush: entries in memory were written to a file (Minor Compaction)
+
++M+::
+ Merging compaction: entries in memory were combined with the smallest file to create one new file
+
++C+::
+ Several files, but not all files, were combined to produce this file (Major Compaction)
+
++A+::
+ All files were compacted, delete entries were dropped
+
++I+::
+ Bulk import, complete, sorted index files. Always in a directory starting with +b-+
+
+This simple file naming convention allows you to see the basic structure of the files from just
+their filenames, and reason about what should be happening to them next, just
+by scanning their entries in the metadata tables.
+
+For example, if you see multiple files with +M+ prefixes, the tablet is, or was, up against its
+maximum file limit, so it began merging memory updates with files to keep the file count reasonable. This
+slows down ingest performance, so knowing there are many files like this tells you that the system
+is struggling to keep up with ingest vs the compaction strategy which reduces the number of files.
+
http://git-wip-us.apache.org/repos/asf/accumulo/blob/900d6abb/docs/src/main/asciidoc/images/accumulo-logo.png
----------------------------------------------------------------------
diff --git a/docs/src/main/asciidoc/images/accumulo-logo.png b/docs/src/main/asciidoc/images/accumulo-logo.png
new file mode 100644
index 0000000..5b0f6b4
Binary files /dev/null and b/docs/src/main/asciidoc/images/accumulo-logo.png differ
http://git-wip-us.apache.org/repos/asf/accumulo/blob/900d6abb/docs/src/main/asciidoc/images/data_distribution.png
----------------------------------------------------------------------
diff --git a/docs/src/main/asciidoc/images/data_distribution.png b/docs/src/main/asciidoc/images/data_distribution.png
new file mode 100644
index 0000000..7f18d3f
Binary files /dev/null and b/docs/src/main/asciidoc/images/data_distribution.png differ
http://git-wip-us.apache.org/repos/asf/accumulo/blob/900d6abb/docs/src/main/asciidoc/images/failure_handling.png
----------------------------------------------------------------------
diff --git a/docs/src/main/asciidoc/images/failure_handling.png b/docs/src/main/asciidoc/images/failure_handling.png
new file mode 100644
index 0000000..c131de6
Binary files /dev/null and b/docs/src/main/asciidoc/images/failure_handling.png differ
http://git-wip-us.apache.org/repos/asf/accumulo/blob/900d6abb/docs/src/main/latex/accumulo_user_manual/accumulo_user_manual.tex
----------------------------------------------------------------------
diff --git a/docs/src/main/latex/accumulo_user_manual/accumulo_user_manual.tex b/docs/src/main/latex/accumulo_user_manual/accumulo_user_manual.tex
deleted file mode 100644
index ae2a440..0000000
--- a/docs/src/main/latex/accumulo_user_manual/accumulo_user_manual.tex
+++ /dev/null
@@ -1,55 +0,0 @@
-
-% Licensed to the Apache Software Foundation (ASF) under one or more
-% contributor license agreements. See the NOTICE file distributed with
-% this work for additional information regarding copyright ownership.
-% The ASF licenses this file to You under the Apache License, Version 2.0
-% (the "License"); you may not use this file except in compliance with
-% the License. You may obtain a copy of the License at
-%
-% http://www.apache.org/licenses/LICENSE-2.0
-%
-% Unless required by applicable law or agreed to in writing, software
-% distributed under the License is distributed on an "AS IS" BASIS,
-% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-% See the License for the specific language governing permissions and
-% limitations under the License.
-
-\documentclass[11pt]{report}
-\title{Apache Accumulo User Manual\\
-Version 1.6}
-\usepackage{alltt}
-\usepackage{multirow}
-\usepackage{graphicx}
-\usepackage[T1]{fontenc}
-\usepackage{ulem}
-\renewcommand{\ttdefault}{txtt}
-
-\setlength{\textwidth}{6.25in}
-\evensidemargin=0in
-\oddsidemargin=0in
-
-\usepackage{parskip}
-\usepackage{hyperref}
-\newenvironment{shellcommands}[0]
-{\ttfamily \begin{flushleft} \setlength{\parindent}{0cm} \setlength{\parskip}{12mm} \hangindent=1cm}
-{\end{flushleft} \setlength{\parindent}{7mm} \hangindent=0cm \normalfont}
-
-\begin{document}
-\maketitle
-\tableofcontents
-\include{chapters/introduction}
-\include{chapters/design}
-\include{chapters/shell}
-\include{chapters/clients}
-\include{chapters/development_clients}
-\include{chapters/table_configuration}
-\include{chapters/table_design}
-\include{chapters/high_speed_ingest}
-\include{chapters/analytics}
-\include{chapters/security}
-\include{chapters/administration}
-\include{chapters/multivolume}
-\include{chapters/troubleshooting}
-\appendix
-\include{appendices/config}
-\end{document}
http://git-wip-us.apache.org/repos/asf/accumulo/blob/900d6abb/docs/src/main/latex/accumulo_user_manual/chapters/administration.tex
----------------------------------------------------------------------
diff --git a/docs/src/main/latex/accumulo_user_manual/chapters/administration.tex b/docs/src/main/latex/accumulo_user_manual/chapters/administration.tex
deleted file mode 100644
index 743b496..0000000
--- a/docs/src/main/latex/accumulo_user_manual/chapters/administration.tex
+++ /dev/null
@@ -1,431 +0,0 @@
-
-% Licensed to the Apache Software Foundation (ASF) under one or more
-% contributor license agreements. See the NOTICE file distributed with
-% this work for additional information regarding copyright ownership.
-% The ASF licenses this file to You under the Apache License, Version 2.0
-% (the "License"); you may not use this file except in compliance with
-% the License. You may obtain a copy of the License at
-%
-% http://www.apache.org/licenses/LICENSE-2.0
-%
-% Unless required by applicable law or agreed to in writing, software
-% distributed under the License is distributed on an "AS IS" BASIS,
-% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-% See the License for the specific language governing permissions and
-% limitations under the License.
-
-\chapter{Administration}
-
-\section{Hardware}
-
-Because we are running essentially two or three systems simultaneously layered
-across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to
-consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have
-at least one core and 2 - 4 GB each.
-
-One core running HDFS can typically keep 2 to 4 disks busy, so each machine may
-typically have as little as 2 x 300GB disks and as much as 4 x 1TB or 2TB disks.
-
-It is possible to do with less than this, such as with 1u servers with 2 cores and 4GB
-each, but in this case it is recommended to only run up to two processes per
-machine - i.e. DataNode and TabletServer or DataNode and MapReduce worker but
-not all three. The constraint here is having enough available heap space for all the
-processes on a machine.
-
-\section{Network}
-
-Accumulo communicates via remote procedure calls over TCP/IP for both passing
-data and control messages. In addition, Accumulo uses HDFS clients to
-communicate with HDFS. To achieve good ingest and query performance, sufficient
-network bandwidth must be available between any two machines.
-
-In addition to needing access to ports associated with HDFS and ZooKeeper, Accumulo will
-use the following default ports. Please make sure that they are open, or change
-their value in conf/accumulo-site.xml.
-
-\begin{table}[t]
-\centering
-\begin{tabular}{|l|c|c|}
-\hline
-Port & Description & Property Name\\
-\hline
-4445 & Shutdown Port (Accumulo MiniCluster) & n/a\\
-\hline
-4560 & Accumulo monitor (for centralized log display) & monitor.port.log4j\\
-\hline
-9997 & Tablet Server & tserver.port.client\\
-\hline
-9999 & Master Server & master.port.client\\
-\hline
-12234 & Accumulo Tracer & trace.port.client\\
-\hline
-42424 & Accumulo Proxy Server & n/a\\
-\hline
-50091 & Accumulo GC & gc.port.client\\
-\hline
-50095 & Accumulo HTTP monitor & monitor.port.client\\
-\hline
-\end{tabular}
-\caption{Accumulo default ports}
-\label{tab:template}
-\end{table}
-
-In addition, the user can provide '0' and an ephemeral port will be chosen instead. This
-ephemeral port is likely to be unique and not already bound. Thus, configuring ports to
-use '0' instead of an explicit value, should, in most cases, work around any issues of
-running multiple distinct Accumulo instances (or any other process which tries to use the
-same default ports) on the same hardware.
-
-\section{Installation}
-Choose a directory for the Accumulo installation. This directory will be referenced
-by the environment variable \texttt{\$ACCUMULO\_HOME}. Run the following:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-$ tar xzf accumulo-1.5.0-bin.tar.gz # unpack to subdirectory
-$ mv accumulo-1.5.0 $ACCUMULO_HOME # move to desired location
-\end{verbatim}\endgroup
-
-Repeat this step at each machine within the cluster. Usually all machines have the
-same \texttt{\$ACCUMULO\_HOME}.
-
-\section{Dependencies}
-Accumulo requires HDFS and ZooKeeper to be configured and running
-before starting. Password-less SSH should be configured between at least the
-Accumulo master and TabletServer machines. It is also a good idea to run Network
-Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of
-sync, which can cause problems with automatically timestamped data.
-
-\section{Configuration}
-
-Accumulo is configured by editing several Shell and XML files found in
-\texttt{\$ACCUMULO\_HOME/conf}. The structure closely resembles Hadoop's configuration
-files.
-
-\subsection{Edit conf/accumulo-env.sh}
-
-Accumulo needs to know where to find the software it depends on. Edit accumulo-env.sh
-and specify the following:
-
-\begin{enumerate}
-\item{Enter the location of the installation directory of Accumulo for \texttt{\$ACCUMULO\_HOME}}
-\item{Enter your system's Java home for \texttt{\$JAVA\_HOME}}
-\item{Enter the location of Hadoop for \texttt{\$HADOOP\_PREFIX}}
-\item{Choose a location for Accumulo logs and enter it for \texttt{\$ACCUMULO\_LOG\_DIR}}
-\item{Enter the location of ZooKeeper for \texttt{\$ZOOKEEPER\_HOME}}
-\end{enumerate}
-
-By default Accumulo TabletServers are set to use 1GB of memory. You may change
-this by altering the value of \texttt{\$ACCUMULO\_TSERVER\_OPTS}. Note the syntax is that of
-the Java JVM command line options. This value should be less than the physical
-memory of the machines running TabletServers.
-
-There are similar options for the master's memory usage and the garbage collector
-process. Reduce these if they exceed the physical RAM of your hardware and
-increase them, within the bounds of the physical RAM, if a process fails because of
-insufficient memory.
-
-Note that you will be specifying the Java heap space in accumulo-env.sh. You should
-make sure that the total heap space used for the Accumulo tserver and the Hadoop
-DataNode and TaskTracker is less than the available memory on each slave node in
-the cluster. On large clusters, it is recommended that the Accumulo master, Hadoop
-NameNode, secondary NameNode, and Hadoop JobTracker all be run on separate
-machines to allow them to use more heap space. If you are running these on the
-same machine on a small cluster, likewise make sure their heap space settings fit
-within the available memory.
-
-\subsection{Native Map}
-
-The tablet server uses a data structure called a MemTable to store sorted key/value
-pairs in memory when they are first received from the client. When a minor compaction
-occurs, this data structure is written to HDFS. The MemTable will default to using
-memory in the JVM but a JNI version, called the native map, can be used to significantly
-speed up performance by utilizing the memory space of the native operating system. The
-native map also avoids the performance implications brought on by garbage collection
-in the JVM by causing it to pause much less frequently.
-
-32-bit and 64-bit Linux versions of the native map ship with the Accumulo dist package.
-For other operating systems, the native map can be built from the codebase in two ways-
-from maven or from the Makefile.
-
-\begin{enumerate}
-\item{Build from maven using the following command: \texttt{mvn clean package -Pnative.}}
-\item{Build from the c++ source by running \texttt{make} in the \texttt{\$ACCUMULO\_HOME/server/src/main/c++} directory.}
-\end{enumerate}
-
-After building the native map from the source, you will find the artifact in
-\texttt{\$ACCUMULO\_HOME/lib/native.} Upon starting up, the tablet server will look
-in this directory for the map library. If the file is renamed or moved from its
-target directory, the tablet server may not be able to find it.
-
-\subsection{Cluster Specification}
-
-On the machine that will serve as the Accumulo master:
-
-\begin{enumerate}
-\item{Write the IP address or domain name of the Accumulo Master to the\\\texttt{\$ACCUMULO\_HOME/conf/masters} file.}
-\item{Write the IP addresses or domain name of the machines that will be TabletServers in\\\texttt{\$ACCUMULO\_HOME/conf/slaves}, one per line.}
-\end{enumerate}
-
-Note that if using domain names rather than IP addresses, DNS must be configured
-properly for all machines participating in the cluster. DNS can be a confusing source
-of errors.
-
-\subsection{Accumulo Settings}
-Specify appropriate values for the following settings in\\
-\texttt{\$ACCUMULO\_HOME/conf/accumulo-site.xml} :
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-<property>
- <name>instance.zookeeper.host</name>
- <value>zooserver-one:2181,zooserver-two:2181</value>
- <description>list of zookeeper servers</description>
-</property>
-\end{verbatim}\endgroup
-
-This enables Accumulo to find ZooKeeper. Accumulo uses ZooKeeper to coordinate
-settings between processes and helps finalize TabletServer failure.
-
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-<property>
- <name>instance.secret</name>
- <value>DEFAULT</value>
-</property>
-\end{verbatim}\endgroup
-
-The instance needs a secret to enable secure communication between servers. Configure your
-secret and make sure that the \texttt{accumulo-site.xml} file is not readable to other users.
-
-Some settings can be modified via the Accumulo shell and take effect immediately, but
-some settings require a process restart to take effect. See the configuration documentation
-(available in the docs directory of the tarball and in Appendix~\ref{app:config}) for details.
-
-\subsection{Deploy Configuration}
-
-Copy the masters, slaves, accumulo-env.sh, and if necessary, accumulo-site.xml
-from the\\\texttt{\$ACCUMULO\_HOME/conf/} directory on the master to all the machines
-specified in the slaves file.
-
-\section{Initialization}
-
-Accumulo must be initialized to create the structures it uses internally to locate
-data across the cluster. HDFS is required to be configured and running before
-Accumulo can be initialized.
-
-Once HDFS is started, initialization can be performed by executing\\
-\texttt{\$ACCUMULO\_HOME/bin/accumulo init} . This script will prompt for a name
-for this instance of Accumulo. The instance name is used to identify a set of tables
-and instance-specific settings. The script will then write some information into
-HDFS so Accumulo can start properly.
-
-The initialization script will prompt you to set a root password. Once Accumulo is
-initialized it can be started.
-
-\section{Running}
-
-\subsection{Starting Accumulo}
-
-Make sure Hadoop is configured on all of the machines in the cluster, including
-access to a shared HDFS instance. Make sure HDFS and ZooKeeper are running.
-Make sure ZooKeeper is configured and running on at least one machine in the
-cluster.
-Start Accumulo using the \texttt{bin/start-all.sh} script.
-
-To verify that Accumulo is running, check the Status page as described under
-\emph{Monitoring}. In addition, the Shell can provide some information about the status of
-tables via reading the metadata tables.
-
-\subsection{Stopping Accumulo}
-
-To shutdown cleanly, run \texttt{bin/stop-all.sh} and the master will orchestrate the
-shutdown of all the tablet servers. Shutdown waits for all minor compactions to finish, so it may
-take some time for particular configurations.
-
-\subsection{Adding a Node}
-
-Update your \texttt{\$ACCUMULO\_HOME/conf/slaves} (or \texttt{\$ACCUMULO\_CONF\_DIR/slaves}) file to account for the addition.
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-$ACCUMULO_HOME/bin/accumulo admin start <host(s)> {<host> ...}
-\end{verbatim}\endgroup
-
-Alternatively, you can ssh to each of the hosts you want to add and run:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-$ACCUMULO\_HOME/bin/start-here.sh
-\end{verbatim}\endgroup
-
-Make sure the host in question has the new configuration, or else the tablet
-server won't start; at a minimum this needs to be on the host(s) being added,
-but in practice it's good to ensure consistent configuration across all nodes.
-
-\subsection{Decomissioning a Node}
-
-If you need to take a node out of operation, you can trigger a graceful shutdown of a tablet
-server. Accumulo will automatically rebalance the tablets across the available tablet servers.
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-$ACCUMULO_HOME/bin/accumulo admin stop <host(s)> {<host> ...}
-\end{verbatim}\endgroup
-
-Alternatively, you can ssh to each of the hosts you want to remove and run:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-$ACCUMULO\_HOME/bin/stop-here.sh
-\end{verbatim}\endgroup
-
-Be sure to update your \texttt{\$ACCUMUL\_HOME/conf/slaves} (or \texttt{\$ACCUMULO\_CONF\_DIR/slaves}) file to
-account for the removal of these hosts. Bear in mind that the monitor will not re-read the
-slaves file automatically, so it will report the decomissioned servers as down; it's
-recommended that you restart the monitor so that the node list is up to date.
-
-\section{Monitoring}
-
-The Accumulo Master provides an interface for monitoring the status and health of
-Accumulo components. This interface can be accessed by pointing a web browser to\\
-\texttt{http://accumulomaster:50095/status}
-
-\section{Tracing}
-It can be difficult to determine why some operations are taking longer
-than expected. For example, you may be looking up items with very low
-latency, but sometimes the lookups take much longer. Determining the
-cause of the delay is difficult because the system is distributed, and
-the typical lookup is fast.
-
-Accumulo has been instrumented to record the time that various
-operations take when tracing is turned on. The fact that tracing is
-enabled follows all the requests made on behalf of the user throughout
-the distributed infrastructure of accumulo, and across all threads of
-execution.
-
-These time spans will be inserted into the \texttt{trace} table in
-Accumulo. You can browse recent traces from the Accumulo monitor
-page. You can also read the \texttt{trace} table directly like any
-other table.
-
-The design of Accumulo's distributed tracing follows that of
-\href{http://research.google.com/pubs/pub36356.html}{Google's Dapper}.
-
-\subsection{Tracers}
-To collect traces, Accumulo needs at least one server listed in
-\\\texttt{\$ACCUMULO\_HOME/conf/tracers}. The server collects traces
-from clients and writes them to the \texttt{trace} table. The Accumulo
-user that the tracer connects to Accumulo with can be configured with
-the following properties
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-trace.user
-trace.token.property.password
-\end{verbatim}\endgroup
-
-\subsection{Instrumenting a Client}
-Tracing can be used to measure a client operation, such as a scan, as
-the operation traverses the distributed system. To enable tracing for
-your application call
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-DistributedTrace.enable(instance, new ZooReader(instance), hostname, "myApplication");
-\end{verbatim}\endgroup
-
-Once tracing has been enabled, a client can wrap an operation in a trace.
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-Trace.on("Client Scan");
-BatchScanner scanner = conn.createBatchScanner(...);
-// Configure your scanner
-for (Entry entry : scanner) {
-}
-Trace.off();
-\end{verbatim}\endgroup
-
-Additionally, the user can create additional Spans within a Trace.
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-Trace.on("Client Update");
-...
-Span readSpan = Trace.start("Read");
-...
-readSpan.stop();
-...
-Span writeSpan = Trace.start("Write");
-...
-writeSpan.stop();
-Trace.off();
-\end{verbatim}\endgroup
-
-Like Dapper, Accumulo tracing supports user defined annotations to associate additional data with a Trace.
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-...
-int numberOfEntriesRead = 0;
-Span readSpan = Trace.start("Read");
-// Do the read, update the counter
-...
-readSpan.data("Number of Entries Read", String.valueOf(numberOfEntriesRead));
-\end{verbatim}\endgroup
-
-Some client operations may have a high volume within your
-application. As such, you may wish to only sample a percentage of
-operations for tracing. As seen below, the CountSampler can be used to
-help enable tracing for 1-in-1000 operations
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-Sampler sampler = new CountSampler(1000);
-...
-if (sampler.next()) {
- Trace.on("Read");
-}
-...
-Trace.offNoFlush();
-\end{verbatim}\endgroup
-
-It should be noted that it is safe to turn off tracing even if it
-isn't currently active. The Trace.offNoFlush() should be used if the
-user does not wish to have Trace.off() block while flushing trace
-data.
-
-\subsection{Viewing Collected Traces}
-To view collected traces, use the "Recent Traces" link on the Monitor
-UI. You can also programmatically access and print traces using the
-\texttt{TraceDump} class.
-
-\subsection{Tracing from the Shell}
-You can enable tracing for operations run from the shell by using the
-\texttt{trace on} and \texttt{trace off} commands.
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-root@test test> trace on
-root@test test> scan
-a b:c [] d
-root@test test> trace off
-Waiting for trace information
-Waiting for trace information
-Trace started at 2013/08/26 13:24:08.332
-Time Start Service@Location Name
- 3628+0 shell@localhost shell:root
- 8+1690 shell@localhost scan
- 7+1691 shell@localhost scan:location
- 6+1692 tserver@localhost startScan
- 5+1692 tserver@localhost tablet read ahead 6
-\end{verbatim}\endgroup
-
-\section{Logging}
-Accumulo processes each write to a set of log files. By default these are found under\\
-\texttt{\$ACCUMULO/logs/}.
-
-\section{Recovery}
-
-In the event of TabletServer failure or error on shutting Accumulo down, some
-mutations may not have been minor compacted to HDFS properly. In this case,
-Accumulo will automatically reapply such mutations from the write-ahead log
-either when the tablets from the failed server are reassigned by the Master (in the
-case of a single TabletServer failure) or the next time Accumulo starts (in the event of
-failure during shutdown).
-
-Recovery is performed by asking a tablet server to sort the logs so that tablets can easily find their missing
-updates. The sort status of each file is displayed on
-Accumulo monitor status page. Once the recovery is complete any
-tablets involved should return to an ``online" state. Until then those tablets will be
-unavailable to clients.
-
-The Accumulo client library is configured to retry failed mutations and in many
-cases clients will be able to continue processing after the recovery process without
-throwing an exception.
-
http://git-wip-us.apache.org/repos/asf/accumulo/blob/900d6abb/docs/src/main/latex/accumulo_user_manual/chapters/analytics.tex
----------------------------------------------------------------------
diff --git a/docs/src/main/latex/accumulo_user_manual/chapters/analytics.tex b/docs/src/main/latex/accumulo_user_manual/chapters/analytics.tex
deleted file mode 100644
index 301120b..0000000
--- a/docs/src/main/latex/accumulo_user_manual/chapters/analytics.tex
+++ /dev/null
@@ -1,248 +0,0 @@
-
-% Licensed to the Apache Software Foundation (ASF) under one or more
-% contributor license agreements. See the NOTICE file distributed with
-% this work for additional information regarding copyright ownership.
-% The ASF licenses this file to You under the Apache License, Version 2.0
-% (the "License"); you may not use this file except in compliance with
-% the License. You may obtain a copy of the License at
-%
-% http://www.apache.org/licenses/LICENSE-2.0
-%
-% Unless required by applicable law or agreed to in writing, software
-% distributed under the License is distributed on an "AS IS" BASIS,
-% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-% See the License for the specific language governing permissions and
-% limitations under the License.
-
-\chapter{Analytics}
-
-Accumulo supports more advanced data processing than simply keeping keys
-sorted and performing efficient lookups. Analytics can be developed by using
-MapReduce and Iterators in conjunction with Accumulo tables.
-
-\section{MapReduce}
-
-Accumulo tables can be used as the source and destination of MapReduce jobs. To
-use an Accumulo table with a MapReduce job (specifically with the new Hadoop API
-as of version 0.20), configure the job parameters to use the AccumuloInputFormat
-and AccumuloOutputFormat. Accumulo specific parameters can be set via these
-two format classes to do the following:
-\begin{itemize}
-\item{Authenticate and provide user credentials for the input}
-\item{Restrict the scan to a range of rows}
-\item{Restrict the input to a subset of available columns}
-\end{itemize}
-
-\subsection{Mapper and Reducer classes}
-
-To read from an Accumulo table create a Mapper with the following class
-parameterization and be sure to configure the AccumuloInputFormat.
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-class MyMapper extends Mapper<Key,Value,WritableComparable,Writable> {
- public void map(Key k, Value v, Context c) {
- // transform key and value data here
- }
-}
-\end{verbatim}\endgroup
-
-To write to an Accumulo table, create a Reducer with the following class
-parameterization and be sure to configure the AccumuloOutputFormat. The key
-emitted from the Reducer identifies the table to which the mutation is sent. This
-allows a single Reducer to write to more than one table if desired. A default table
-can be configured using the AccumuloOutputFormat, in which case the output table
-name does not have to be passed to the Context object within the Reducer.
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-class MyReducer extends Reducer<WritableComparable, Writable, Text, Mutation> {
- public void reduce(WritableComparable key, Iterable<Text> values, Context c) {
- Mutation m;
- // create the mutation based on input key and value
- c.write(new Text("output-table"), m);
- }
-}
-\end{verbatim}\endgroup
-
-The Text object passed as the output should contain the name of the table to which
-this mutation should be applied. The Text can be null in which case the mutation
-will be applied to the default table name specified in the AccumuloOutputFormat
-options.
-
-\subsection{AccumuloInputFormat options}
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-Job job = new Job(getConf());
-AccumuloInputFormat.setInputInfo(job,
- "user",
- "passwd".getBytes(),
- "table",
- new Authorizations());
-
-AccumuloInputFormat.setZooKeeperInstance(job, "myinstance",
- "zooserver-one,zooserver-two");
-\end{verbatim}\endgroup
-
-\Large
-\textbf{Optional Settings:}
-\normalsize
-
-To restrict Accumulo to a set of row ranges:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-ArrayList<Range> ranges = new ArrayList<Range>();
-// populate array list of row ranges ...
-AccumuloInputFormat.setRanges(job, ranges);
-\end{verbatim}\endgroup
-
-To restrict Accumulo to a list of columns:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-ArrayList<Pair<Text,Text>> columns = new ArrayList<Pair<Text,Text>>();
-// populate list of columns
-AccumuloInputFormat.fetchColumns(job, columns);
-\end{verbatim}\endgroup
-
-To use a regular expression to match row IDs:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-IteratorSetting is = new IteratorSetting(30, RexExFilter.class);
-RegExFilter.setRegexs(is, ".*suffix", null, null, null, true);
-AccumuloInputFormat.addIterator(job, is);
-\end{verbatim}\endgroup
-
-\subsection{AccumuloMultiTableInputFormat options}
-
-The AccumuloMultiTableInputFormat allows the scanning over multiple tables
-in a single MapReduce job. Separate ranges, columns, and iterators can be
-used for each table.
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-InputTableConfig tableOneConfig = new InputTableConfig();
-InputTableConfig tableTwoConfig = new InputTableConfig();
-\end{verbatim}\endgroup
-
-To set the configuration objects on the job:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-Map<String, InputTableConfig> configs = new HashMap<String,InputTableConfig>();
-configs.put("table1", tableOneConfig);
-configs.put("table2", tableTwoConfig);
-AccumuloMultiTableInputFormat.setInputTableConfigs(job, configs);
-\end{verbatim}\endgroup
-
-\Large
-\textbf{Optional settings:}
-\normalsize
-
-To restrict to a set of ranges:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-ArrayList<Range> tableOneRanges = new ArrayList<Range>();
-ArrayList<Range> tableTwoRanges = new ArrayList<Range>();
-// populate array lists of row ranges for tables...
-tableOneConfig.setRanges(tableOneRanges);
-tableTwoConfig.setRanges(tableTwoRanges);
-\end{verbatim}\endgroup
-
-To restrict Accumulo to a list of columns:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-ArrayList<Pair<Text,Text>> tableOneColumns = new ArrayList<Pair<Text,Text>>();
-ArrayList<Pair<Text,Text>> tableTwoColumns = new ArrayList<Pair<Text,Text>>();
-// populate lists of columns for each of the tables ...
-tableOneConfig.fetchColumns(tableOneColumns);
-tableTwoConfig.fetchColumns(tableTwoColumns);
-\end{verbatim}\endgroup
-
-To set scan iterators:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-List<IteratorSetting> tableOneIterators = new ArrayList<IteratorSetting>();
-List<IteratorSetting> tableTwoIterators = new ArrayList<IteratorSetting>();
-// populate the lists of iterator settings for each of the tables ...
-tableOneConfig.setIterators(tableOneIterators);
-tableTwoConfig.setIterators(tableTwoIterators);
-\end{verbatim}\endgroup
-
-
-The name of the table can be retrieved from the input split:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-class MyMapper extends Mapper<Key,Value,WritableComparable,Writable> {
- public void map(Key k, Value v, Context c) {
- RangeInputSplit split = (RangeInputSplit)c.getInputSplit();
- String tableName = split.getTableName();
- // do something with table name
- }
-}
-\end{verbatim}\endgroup
-
-
-\subsection{AccumuloOutputFormat options}
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-boolean createTables = true;
-String defaultTable = "mytable";
-
-AccumuloOutputFormat.setOutputInfo(job,
- "user",
- "passwd".getBytes(),
- createTables,
- defaultTable);
-
-AccumuloOutputFormat.setZooKeeperInstance(job, "myinstance",
- "zooserver-one,zooserver-two");
-\end{verbatim}\endgroup
-
-\Large
-\textbf{Optional Settings:}
-\normalsize
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-AccumuloOutputFormat.setMaxLatency(job, 300000); // milliseconds
-AccumuloOutputFormat.setMaxMutationBufferSize(job, 50000000); // bytes
-\end{verbatim}\endgroup
-
-An example of using MapReduce with Accumulo can be found at\\
-accumulo/docs/examples/README.mapred
-
-\section{Combiners}
-
-Many applications can benefit from the ability to aggregate values across common
-keys. This can be done via Combiner iterators and is similar to the Reduce step in
-MapReduce. This provides the ability to define online, incrementally updated
-analytics without the overhead or latency associated with batch-oriented
-MapReduce jobs.
-
-All that is needed to aggregate values of a table is to identify the fields over which
-values will be grouped, insert mutations with those fields as the key, and configure
-the table with a combining iterator that supports the summarizing operation
-desired.
-
-The only restriction on an combining iterator is that the combiner developer
-should not assume that all values for a given key have been seen, since new
-mutations can be inserted at anytime. This precludes using the total number of
-values in the aggregation such as when calculating an average, for example.
-
-\subsection{Feature Vectors}
-
-An interesting use of combining iterators within an Accumulo table is to store
-feature vectors for use in machine learning algorithms. For example, many
-algorithms such as k-means clustering, support vector machines, anomaly detection,
-etc. use the concept of a feature vector and the calculation of distance metrics to
-learn a particular model. The columns in an Accumulo table can be used to efficiently
-store sparse features and their weights to be incrementally updated via the use of an
-combining iterator.
-
-\section{Statistical Modeling}
-
-Statistical models that need to be updated by many machines in parallel could be
-similarly stored within an Accumulo table. For example, a MapReduce job that is
-iteratively updating a global statistical model could have each map or reduce worker
-reference the parts of the model to be read and updated through an embedded
-Accumulo client.
-
-Using Accumulo this way enables efficient and fast lookups and updates of small
-pieces of information in a random access pattern, which is complementary to
-MapReduce's sequential access model.
-
http://git-wip-us.apache.org/repos/asf/accumulo/blob/900d6abb/docs/src/main/latex/accumulo_user_manual/chapters/clients.tex
----------------------------------------------------------------------
diff --git a/docs/src/main/latex/accumulo_user_manual/chapters/clients.tex b/docs/src/main/latex/accumulo_user_manual/chapters/clients.tex
deleted file mode 100644
index 925c352..0000000
--- a/docs/src/main/latex/accumulo_user_manual/chapters/clients.tex
+++ /dev/null
@@ -1,331 +0,0 @@
-
-% Licensed to the Apache Software Foundation (ASF) under one or more
-% contributor license agreements. See the NOTICE file distributed with
-% this work for additional information regarding copyright ownership.
-% The ASF licenses this file to You under the Apache License, Version 2.0
-% (the "License"); you may not use this file except in compliance with
-% the License. You may obtain a copy of the License at
-%
-% http://www.apache.org/licenses/LICENSE-2.0
-%
-% Unless required by applicable law or agreed to in writing, software
-% distributed under the License is distributed on an "AS IS" BASIS,
-% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-% See the License for the specific language governing permissions and
-% limitations under the License.
-
-\chapter{Writing Accumulo Clients}
-
-\section{Running Client Code}
-
-There are multiple ways to run Java code that uses Accumulo. Below is a list
-of the different ways to execute client code.
-
-\begin{itemize}
- \item using java executable
- \item using the accumulo script
- \item using the tool script
-\end{itemize}
-
-In order to run client code written to run against Accumulo, you will need to
-include the jars that Accumulo depends on in your classpath. Accumulo client
-code depends on Hadoop and Zookeeper. For Hadoop add the hadoop client jar, all
-of the jars in the Hadoop lib directory, and the conf directory to the
-classpath. For Zookeeper 3.3 you only need to add the Zookeeper jar, and not
-what is in the Zookeeper lib directory. You can run the following command on a
-configured Accumulo system to see what its using for its classpath.
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-$ACCUMULO_HOME/bin/accumulo classpath
-\end{verbatim}\endgroup
-
-Another option for running your code is to put a jar file in
-\texttt{\$ACCUMULO\_HOME/lib/ext}. After doing this you can use the accumulo
-script to execute your code. For example if you create a jar containing the
-class com.foo.Client and placed that in lib/ext, then you could use the command
-\texttt{\$ACCUMULO\_HOME/bin/accumulo com.foo.Client} to execute your code.
-
-If you are writing map reduce job that access Accumulo, then you can use the
-bin/tool.sh script to run those jobs. See the map reduce example.
-
-\section{Connecting}
-
-All clients must first identify the Accumulo instance to which they will be
-communicating. Code to do this is as follows:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-String instanceName = "myinstance";
-String zooServers = "zooserver-one,zooserver-two"
-Instance inst = new ZooKeeperInstance(instanceName, zooServers);
-
-Connector conn = inst.getConnector("user", new PasswordToken("passwd"));
-\end{verbatim}\endgroup
-
-\section{Writing Data}
-
-Data are written to Accumulo by creating Mutation objects that represent all the
-changes to the columns of a single row. The changes are made atomically in the
-TabletServer. Clients then add Mutations to a BatchWriter which submits them to
-the appropriate TabletServers.
-
-Mutations can be created thus:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-Text rowID = new Text("row1");
-Text colFam = new Text("myColFam");
-Text colQual = new Text("myColQual");
-ColumnVisibility colVis = new ColumnVisibility("public");
-long timestamp = System.currentTimeMillis();
-
-Value value = new Value("myValue".getBytes());
-
-Mutation mutation = new Mutation(rowID);
-mutation.put(colFam, colQual, colVis, timestamp, value);
-\end{verbatim}\endgroup
-
-\subsection{BatchWriter}
-The BatchWriter is highly optimized to send Mutations to multiple TabletServers
-and automatically batches Mutations destined for the same TabletServer to
-amortize network overhead. Care must be taken to avoid changing the contents of
-any Object passed to the BatchWriter since it keeps objects in memory while
-batching.
-
-Mutations are added to a BatchWriter thus:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-//BatchWriterConfig has reasonable defaults
-BatchWriterConfig config = new BatchWriterConfig();
-config.setMaxMemory(10000000L); // bytes available to batchwriter for buffering mutations
-
-BatchWriter writer = conn.createBatchWriter("table", config)
-
-writer.add(mutation);
-
-writer.close();
-\end{verbatim}\endgroup
-
-An example of using the batch writer can be found at\\
-accumulo/docs/examples/README.batch
-
-\subsection{ConditionalWriter}
-The ConditionalWriter enables efficient, atomic read-modify-write operations on
-rows. The ConditionalWriter writes special Mutations which have a list of per
-column conditions that must all be met before the mutation is applied. The
-conditions are checked in the tablet server while a row lock is
-held\footnote{Mutations written by the BatchWriter will not obtain a row
-lock.}. The conditions that can be checked for a column are equality and
-absence. For example a conditional mutation can require that column A is
-absent inorder to be applied. Iterators can be applied when checking
-conditions. Using iterators, many other operations besides equality and
-absence can be checked. For example, using an iterator that converts values
-less than 5 to 0 and everything else to 1, its possible to only apply a
-mutation when a column is less than 5.
-
-In the case when a tablet server dies after a client sent a conditional
-mutation, its not known if the mutation was applied or not. When this happens
-the ConditionalWriter reports a status of UNKNOWN for the ConditionalMutation.
-In many cases this situation can be dealt with by simply reading the row again
-and possibly sending another conditional mutation. If this is not sufficient,
-then a higher level of abstraction can be built by storing transactional
-information within a row.
-
-An example of using the batch writer can be found at\\
-accumulo/docs/examples/README.reservations
-
-\section{Reading Data}
-
-Accumulo is optimized to quickly retrieve the value associated with a given key, and
-to efficiently return ranges of consecutive keys and their associated values.
-
-\subsection{Scanner}
-
-To retrieve data, Clients use a Scanner, which acts like an Iterator over
-keys and values. Scanners can be configured to start and stop at particular keys, and
-to return a subset of the columns available.
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-// specify which visibilities we are allowed to see
-Authorizations auths = new Authorizations("public");
-
-Scanner scan =
- conn.createScanner("table", auths);
-
-scan.setRange(new Range("harry","john"));
-scan.fetchFamily("attributes");
-
-for(Entry<Key,Value> entry : scan) {
- String row = entry.getKey().getRow();
- Value value = entry.getValue();
-}
-\end{verbatim}\endgroup
-
-\subsection{Isolated Scanner}
-
-Accumulo supports the ability to present an isolated view of rows when
-scanning. There are three possible ways that a row could change in Accumulo :
-
-\begin{itemize}
- \item a mutation applied to a table
- \item iterators executed as part of a minor or major compaction
- \item bulk import of new files
-\end{itemize}
-
-Isolation guarantees that either all or none of the changes made by these
-operations on a row are seen. Use the IsolatedScanner to obtain an isolated
-view of an Accumulo table. When using the regular scanner it is possible to see
-a non isolated view of a row. For example if a mutation modifies three
-columns, it is possible that you will only see two of those modifications.
-With the isolated scanner either all three of the changes are seen or none.
-
-The IsolatedScanner buffers rows on the client side so a large row will not
-crash a tablet server. By default rows are buffered in memory, but the user
-can easily supply their own buffer if they wish to buffer to disk when rows are
-large.
-
-For an example, look at the following
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-examples/simple/src/main/java/org/apache/accumulo/examples/simple/isolation/InterferenceTest.java
-\end{verbatim}\endgroup
-
-\subsection{BatchScanner}
-
-For some types of access, it is more efficient to retrieve several ranges
-simultaneously. This arises when accessing a set of rows that are not consecutive
-whose IDs have been retrieved from a secondary index, for example.
-
-The BatchScanner is configured similarly to the Scanner; it can be configured to
-retrieve a subset of the columns available, but rather than passing a single Range,
-BatchScanners accept a set of Ranges. It is important to note that the keys returned
-by a BatchScanner are not in sorted order since the keys streamed are from multiple
-TabletServers in parallel.
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-ArrayList<Range> ranges = new ArrayList<Range>();
-// populate list of ranges ...
-
-BatchScanner bscan =
- conn.createBatchScanner("table", auths, 10);
-
-bscan.setRanges(ranges);
-bscan.fetchFamily("attributes");
-for(Entry<Key,Value> entry : scan)
- System.out.println(entry.getValue());
-\end{verbatim}\endgroup
-
-An example of the BatchScanner can be found at\\
-accumulo/docs/examples/README.batch
-
-\section{Proxy}
-
-The proxy API allows the interaction with Accumulo with languages other than Java.
-A proxy server is provided in the codebase and a client can further be generated.
-
-\subsection{Prequisites}
-
-The proxy server can live on any node in which the basic client API would work. That
-means it must be able to communicate with the Master, ZooKeepers, NameNode, and the
-Data nodes. A proxy client only needs the ability to communicate with the proxy server.
-
-
-\subsection{Configuration}
-
-The configuration options for the proxy server live inside of a properties file. At
-the very least, you need to supply the following properties:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-protocolFactory=org.apache.thrift.protocol.TCompactProtocol$Factory
-tokenClass=org.apache.accumulo.core.client.security.tokens.PasswordToken
-port=42424
-instance=test
-zookeepers=localhost:2181
-\end{verbatim}\endgroup
-
-You can find a sample configuration file in your distribution:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-$ACCUMULO_HOME/proxy/proxy.properties.
-\end{verbatim}\endgroup
-
-This sample configuration file further demonstrates an abilty to back the proxy server
-by MockAccumulo or the MiniAccumuloCluster.
-
-\subsection{Running the Proxy Server}
-
-After the properties file holding the configuration is created, the proxy server
-can be started using the following command in the Accumulo distribution (assuming
-you your properties file is named config.properties):
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-$ACCUMULO_HOME/bin/accumulo proxy -p config.properties
-\end{verbatim}\endgroup
-
-\subsection{Creating a Proxy Client}
-
-Aside from installing the Thrift compiler, you will also need the language-specific library
-for Thrift installed to generate client code in that language. Typically, your operating
-system's package manager will be able to automatically install these for you in an expected
-location such as /usr/lib/python/site-packages/thrift.
-
-You can find the thrift file for generating the client:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-$ACCUMULO_HOME/proxy/proxy.thrift.
-\end{verbatim}\endgroup
-
-After a client is generated, the port specified in the configuration properties above will be
-used to connect to the server.
-
-\subsection{Using a Proxy Client}
-
-The following examples have been written in Java and the method signatures may be
-slightly different depending on the language specified when generating client with
-the Thrift compiler. After initiating a connection to the Proxy (see Apache Thrift's
-documentation for examples of connecting to a Thrift service), the methods on the
-proxy client will be available. The first thing to do is log in:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-Map password = new HashMap<String,String>();
-password.put("password", "secret");
-ByteBuffer token = client.login("root", password);
-\end{verbatim}\endgroup
-
-Once logged in, the token returned will be used for most subsequent calls to the client.
-Let's create a table, add some data, scan the table, and delete it.
-
-
-First, create a table.
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-client.createTable(token, "myTable", true, TimeType.MILLIS);
-\end{verbatim}\endgroup
-
-
-Next, add some data:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-// first, create a writer on the server
-String writer = client.createWriter(token, "myTable", new WriterOptions());
-
-// build column updates
-Map<ByteBuffer, List<ColumnUpdate> cells> cellsToUpdate = //...
-
-// send updates to the server
-client.updateAndFlush(writer, "myTable", cellsToUpdate);
-
-client.closeWriter(writer);
-\end{verbatim}\endgroup
-
-
-Scan for the data and batch the return of the results on the server:
-
-\begingroup\fontsize{8pt}{8pt}\selectfont\begin{verbatim}
-String scanner = client.createScanner(token, "myTable", new ScanOptions());
-ScanResult results = client.nextK(scanner, 100);
-
-for(KeyValue keyValue : results.getResultsIterator()) {
- // do something with results
-}
-
-client.closeScanner(scanner);
-\end{verbatim}\endgroup