You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by bi...@apache.org on 2011/11/01 15:49:31 UTC
svn commit: r1196033 [1/2] - in /incubator/accumulo:
branches/1.3/docs/examples/
site/trunk/content/accumulo/user_manual_1.3-incubating/
site/trunk/content/accumulo/user_manual_1.4-incubating/
Author: billie
Date: Tue Nov 1 14:49:30 2011
New Revision: 1196033
URL: http://svn.apache.org/viewvc?rev=1196033&view=rev
Log:
ACCUMULO-70 renamed section links
Modified:
incubator/accumulo/branches/1.3/docs/examples/README
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Design.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Shell.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Administration.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Analytics.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Contents.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/High_Speed_Ingest.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Introduction.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Security.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Shell_Commands.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Configuration.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Design.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Writing_Accumulo_Clients.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Accumulo_Design.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Accumulo_Shell.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Administration.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Analytics.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Contents.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/High_Speed_Ingest.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Introduction.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Security.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Shell_Commands.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Table_Configuration.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Table_Design.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Writing_Accumulo_Clients.mdtext
Modified: incubator/accumulo/branches/1.3/docs/examples/README
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.3/docs/examples/README?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/branches/1.3/docs/examples/README (original)
+++ incubator/accumulo/branches/1.3/docs/examples/README Tue Nov 1 14:49:30 2011
@@ -33,4 +33,4 @@ Commands intended to be run in bash are
Commands intended to be run in the Accumulo shell are prefixed by '>'.
-[1]: /accumulo/user_manual_1.3-incubating/Accumulo_Shell.html#SECTION00430000000000000000
+[1]: /accumulo/user_manual_1.3-incubating/Accumulo_Shell.html#User_Administration
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Design.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Design.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Design.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Design.mdtext Tue Nov 1 14:49:30 2011
@@ -28,9 +28,9 @@ Notice: Licensed to the Apache Softwa
* * *
-## <a id=SECTION00300000000000000000></a> Accumulo Design
+## <a id=Accumulo_Design></a> Accumulo Design
-## <a id=SECTION00310000000000000000></a> Data Model
+## <a id=Data_Model></a> Data Model
Accumulo provides a richer data model than simple key-value stores, but is not a fully relational database. Data is represented as key-value pairs, where the key and value are comprised of the following elements:
@@ -38,53 +38,53 @@ Accumulo provides a richer data model th
All elements of the Key and the Value are represented as byte arrays except for Timestamp, which is a Long. Accumulo sorts keys by element and lexicographically in ascending order. Timestamps are sorted in descending order so that later versions of the same Key appear first in a sequential scan. Tables consist of a set of sorted key-value pairs.
-## <a id=SECTION00320000000000000000></a> Architecture
+## <a id=Architecture></a> Architecture
Accumulo is a distributed data storage and retrieval system and as such consists of several architectural components, some of which run on many individual servers. Much of the work Accumulo does involves maintaining certain properties of the data, such as organization, availability, and integrity, across many commodity-class machines.
-## <a id=SECTION00330000000000000000></a> Components
+## <a id=Components></a> Components
An instance of Accumulo includes many TabletServers, write-ahead Logger servers, one Garbage Collector process, one Master server and many Clients.
-### <a id=SECTION00331000000000000000></a> Tablet Server
+### <a id=Tablet_Server></a> Tablet Server
The TabletServer manages some subset of all the tablets (partitions of tables). This includes receiving writes from clients, persisting writes to a writeâahead log, sorting new keyâvalue pairs in memory, periodically flushing sorted keyâvalue pairs to new files in HDFS, and responding to reads from clients, forming a mergeâsorted view of all keys and values from all the files it has created and the sorted inâmemory store.
TabletServers also perform recovery of a tablet that was previously on a server that failed, reapplying any writes found in the write-ahead log to the tablet.
-### <a id=SECTION00332000000000000000></a> Loggers
+### <a id=Loggers></a> Loggers
The Loggers accept updates to Tablet servers and write them to local on-disk storage. Each tablet server will write their updates to multiple loggers to preserve data in case of hardware failure.
-### <a id=SECTION00333000000000000000></a> Garbage Collector
+### <a id=Garbage_Collector></a> Garbage Collector
Accumulo processes will share files stored in HDFS. Periodically, the Garbage Collector will identify files that are no longer needed by any process, and delete them.
-### <a id=SECTION00334000000000000000></a> Master
+### <a id=Master></a> Master
The Accumulo Master is responsible for detecting and responding to TabletServer failure. It tries to balance the load across TabletServer by assigning tablets carefully and instructing TabletServers to migrate tablets when necessary. The Master ensures all tablets are assigned to one TabletServer each, and handles table creation, alteration, and deletion requests from clients. The Master also coordinates startup, graceful shutdown and recovery of changes in write-ahead logs when Tablet servers fail.
-### <a id=SECTION00335000000000000000></a> Client
+### <a id=Client></a> Client
Accumulo includes a client library that is linked to every application. The client library contains logic for finding servers managing a particular tablet, and communicating with TabletServers to write and retrieve key-value pairs.
-## <a id=SECTION00340000000000000000></a> Data Management
+## <a id=Data_Management></a> Data Management
Accumulo stores data in tables, which are partitioned into tablets. Tablets are partitioned on row boundaries so that all of the columns and values for a particular row are found together within the same tablet. The Master assigns Tablets to one TabletServer at a time. This enables row-level transactions to take place without using distributed locking or some other complicated synchronization mechanism. As clients insert and query data, and as machines are added and removed from the cluster, the Master migrates tablets to ensure they remain available and that the ingest and query load is balanced across the cluster.
![Image data_distribution][9]
-## <a id=SECTION00350000000000000000></a> Tablet Service
+## <a id=Tablet_Service></a> Tablet Service
When a write arrives at a TabletServer it is written to a WriteâAhead Log and then inserted into a sorted data structure in memory called a MemTable. When the MemTable reaches a certain size the TabletServer writes out the sorted key-value pairs to a file in HDFS called Indexed Sequential Access Method (ISAM) file. This process is called a minor compaction. A new MemTable is then created and the fact of the compaction is recorded in the WriteâAhead Log.
When a request to read data arrives at a TabletServer, the TabletServer does a binary search across the MemTable as well as the in-memory indexes associated with each ISAM file to find the relevant values. If clients are performing a scan, several keyâvalue pairs are returned to the client in order from the MemTable and the set of ISAM files by performing a mergeâsort as they are read.
-## <a id=SECTION00360000000000000000></a> Compactions
+## <a id=Compactions></a> Compactions
In order to manage the number of files per tablet, periodically the TabletServer performs Major Compactions of files within a tablet, in which some set of ISAM files are combined into one file. The previous files will eventually be removed by the Garbage Collector. This also provides an opportunity to permanently remove deleted keyâvalue pairs by omitting keyâvalue pairs suppressed by a delete entry when the new file is created.
-## <a id=SECTION00370000000000000000></a> Fault-Tolerance
+## <a id=Fault-Tolerance></a> Fault-Tolerance
If a TabletServer fails, the Master detects it and automatically reassigns the tablets assigned from the failed server to other servers. Any key-value pairs that were in memory at the time the TabletServer are automatically reapplied from the Write-Ahead Log to prevent any loss of data.
@@ -100,13 +100,13 @@ http://master-address:50095/monitor.
* * *
- [1]: Accumulo_Design.html#SECTION00310000000000000000
- [2]: Accumulo_Design.html#SECTION00320000000000000000
- [3]: Accumulo_Design.html#SECTION00330000000000000000
- [4]: Accumulo_Design.html#SECTION00340000000000000000
- [5]: Accumulo_Design.html#SECTION00350000000000000000
- [6]: Accumulo_Design.html#SECTION00360000000000000000
- [7]: Accumulo_Design.html#SECTION00370000000000000000
+ [1]: Accumulo_Design.html#Data_Model
+ [2]: Accumulo_Design.html#Architecture
+ [3]: Accumulo_Design.html#Components
+ [4]: Accumulo_Design.html#Data_Management
+ [5]: Accumulo_Design.html#Tablet_Service
+ [6]: Accumulo_Design.html#Compactions
+ [7]: Accumulo_Design.html#Fault-Tolerance
[8]: img1.png
[9]: ./data_distribution.png
[10]: ./failure_handling.png
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Shell.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Shell.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Shell.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Shell.mdtext Tue Nov 1 14:49:30 2011
@@ -24,7 +24,7 @@ Notice: Licensed to the Apache Softwa
* * *
-## <a id=SECTION00400000000000000000></a> Accumulo Shell
+## <a id=Accumulo_Shell></a> Accumulo Shell
Accumulo provides a simple shell that can be used to examine the contents and configuration settings of tables, apply individual mutations, and change configuration settings.
@@ -47,7 +47,7 @@ The shell will prompt for the correspond
-
-## <a id=SECTION00410000000000000000></a> Basic Administration
+## <a id=Basic_Administration></a> Basic Administration
The Accumulo shell can be used to create and delete tables, as well as to configure table and instance specific options.
@@ -84,7 +84,7 @@ The Shell can also be used to insert upd
row1 colf:colq [] value1
-## <a id=SECTION00420000000000000000></a> Table Maintenance
+## <a id=Table_Maintenance></a> Table Maintenance
The **compact** command instructs Accumulo to schedule a compaction of the table during which files are consolidated and deleted entries are removed.
@@ -102,7 +102,7 @@ The **flush** command instructs Accumulo
initiated...
-## <a id=SECTION00430000000000000000></a> User Administration
+## <a id=User_Administration></a> User Administration
The Shell can be used to add, remove, and grant privileges to users.
@@ -139,7 +139,7 @@ The Shell can be used to add, remove, an
* * *
- [1]: Accumulo_Shell.html#SECTION00410000000000000000
- [2]: Accumulo_Shell.html#SECTION00420000000000000000
- [3]: Accumulo_Shell.html#SECTION00430000000000000000
+ [1]: Accumulo_Shell.html#Basic_Administration
+ [2]: Accumulo_Shell.html#Table_Maintenance
+ [3]: Accumulo_Shell.html#User_Administration
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Administration.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Administration.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Administration.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Administration.mdtext Tue Nov 1 14:49:30 2011
@@ -31,9 +31,9 @@ Notice: Licensed to the Apache Softwa
* * *
-## <a id=SECTION001100000000000000000></a> Administration
+## <a id=Administration></a> Administration
-## <a id=SECTION001110000000000000000></a> Hardware
+## <a id=Hardware></a> Hardware
Because we are running essentially two or three systems simultaneously layered across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have at least one core and 2 - 4 GB each.
@@ -41,11 +41,11 @@ One core running HDFS can typically keep
It is possible to do with less than this, such as with 1u servers with 2 cores and 4GB each, but in this case it is recommended to only run up to two processes per machine - i.e. DataNode and TabletServer or DataNode and MapReduce worker but not all three. The constraint here is having enough available heap space for all the processes on a machine.
-## <a id=SECTION001120000000000000000></a> Network
+## <a id=Network></a> Network
Accumulo communicates via remote procedure calls over TCP/IP for both passing data and control messages. In addition, Accumulo uses HDFS clients to communicate with HDFS. To achieve good ingest and query performance, sufficient network bandwidth must be available between any two machines.
-## <a id=SECTION001130000000000000000></a> Installation
+## <a id=Installation></a> Installation
Choose a directory for the Accumulo installation. This directory will be referenced by the environment variable $ACCUMULO_HOME. Run the following:
@@ -55,15 +55,15 @@ Choose a directory for the Accumulo inst
Repeat this step at each machine within the cluster. Usually all machines have the same $ACCUMULO_HOME.
-## <a id=SECTION001140000000000000000></a> Dependencies
+## <a id=Dependencies></a> Dependencies
Accumulo requires HDFS and ZooKeeper to be configured and running before starting. Password-less SSH should be configured between at least the Accumulo master and TabletServer machines. It is also a good idea to run Network Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of sync, which can cause problems with automatically timestamped data. Accumulo will remove from the set of TabletServers those machines whose times differ too much from the master's.
-## <a id=SECTION001150000000000000000></a> Configuration
+## <a id=Configuration></a> Configuration
Accumulo is configured by editing several Shell and XML files found in $ACCUMULO_HOME/conf. The structure closely resembles Hadoop's configuration files.
-### <a id=SECTION001151000000000000000></a> Edit conf/accumulo-env.sh
+### <a id=Edit_conf/accumulo-env.sh></a> Edit conf/accumulo-env.sh
Accumulo needs to know where to find the software it depends on. Edit accumuloenv. sh and specify the following:
@@ -79,7 +79,7 @@ There are similar options for the master
Note that you will be specifying the Java heap space in accumulo-env.sh. You should make sure that the total heap space used for the Accumulo tserver and the Hadoop DataNode and TaskTracker is less than the available memory on each slave node in the cluster. On large clusters, it is recommended that the Accumulo master, Hadoop NameNode, secondary NameNode, and Hadoop JobTracker all be run on separate machines to allow them to use more heap space. If you are running these on the same machine on a small cluster, likewise make sure their heap space settings fit within the available memory.
-### <a id=SECTION001152000000000000000></a> Cluster Specification
+### <a id=Cluster_Specification></a> Cluster Specification
On the machine that will serve as the Accumulo master:
@@ -90,7 +90,7 @@ $ACCUMULO_HOME/conf/slaves, one per line
Note that if using domain names rather than IP addresses, DNS must be configured properly for all machines participating in the cluster. DNS can be a confusing source of errors.
-### <a id=SECTION001153000000000000000></a> Accumulo Settings
+### <a id=Accumulo_Settings></a> Accumulo Settings
Specify appropriate values for the following settings in
$ACCUMULO_HOME/conf/accumulo-site.xml :
@@ -114,12 +114,12 @@ Accumulo records all changes to tables t
Some settings can be modified via the Accumulo shell and take effect immediately. However, any settings that should be persisted across system restarts must be recorded in the accumulo-site.xml file.
-### <a id=SECTION001154000000000000000></a> Deploy Configuration
+### <a id=Deploy_Configuration></a> Deploy Configuration
Copy the masters, slaves, accumulo-env.sh, and if necessary, accumulo-site.xml from the
$ACCUMULO_HOME/conf/ directory on the master to all the machines specified in the slaves file.
-## <a id=SECTION001160000000000000000></a> Initialization
+## <a id=Initialization></a> Initialization
Accumulo must be initialized to create the structures it uses internally to locate data across the cluster. HDFS is required to be configured and running before Accumulo can be initialized.
@@ -128,29 +128,29 @@ $ACCUMULO_HOME/bin/accumulo init . This
The initialization script will prompt you to set a root password. Once Accumulo is initialized it can be started.
-## <a id=SECTION001170000000000000000></a> Running
+## <a id=Running></a> Running
-### <a id=SECTION001171000000000000000></a> Starting Accumulo
+### <a id=Starting_Accumulo></a> Starting Accumulo
Make sure Hadoop is configured on all of the machines in the cluster, including access to a shared HDFS instance. Make sure HDFS and ZooKeeper are running. Make sure ZooKeeper is configured and running on at least one machine in the cluster. Start Accumulo using the bin/start-all.sh script.
To verify that Accumulo is running, check the Status page as described under _Monitoring_. In addition, the Shell can provide some information about the status of tables via reading the !METADATA table.
-### <a id=SECTION001172000000000000000></a> Stopping Accumulo
+### <a id=Stopping_Accumulo></a> Stopping Accumulo
To shutdown cleanly, run bin/stop-all.sh and the master will orchestrate the shutdown of all the tablet servers. Shutdown waits for all minor compactions to finish, so it may take some time for particular configurations.
-## <a id=SECTION001180000000000000000></a> Monitoring
+## <a id=Monitoring></a> Monitoring
The Accumulo Master provides an interface for monitoring the status and health of Accumulo components. This interface can be accessed by pointing a web browser to
http://accumulomaster:50095/status
-## <a id=SECTION001190000000000000000></a> Logging
+## <a id=Logging></a> Logging
Accumulo processes each write to a set of log files. By default these are found under
$ACCUMULO/logs/.
-## <a id=SECTION0011100000000000000000></a> Recovery
+## <a id=Recovery></a> Recovery
In the event of TabletServer failure or error on shutting Accumulo down, some mutations may not have been minor compacted to HDFS properly. In this case, Accumulo will automatically reapply such mutations from the write-ahead log either when the tablets from the failed server are reassigned by the Master, in the case of a single TabletServer failure or the next time Accumulo starts, in the event of failure during shutdown.
@@ -165,14 +165,14 @@ Note that because Accumulo uses timestam
* * *
- [1]: Administration.html#SECTION001110000000000000000
- [2]: Administration.html#SECTION001120000000000000000
- [3]: Administration.html#SECTION001130000000000000000
- [4]: Administration.html#SECTION001140000000000000000
- [5]: Administration.html#SECTION001150000000000000000
- [6]: Administration.html#SECTION001160000000000000000
- [7]: Administration.html#SECTION001170000000000000000
- [8]: Administration.html#SECTION001180000000000000000
- [9]: Administration.html#SECTION001190000000000000000
- [10]: Administration.html#SECTION0011100000000000000000
+ [1]: Administration.html#Hardware
+ [2]: Administration.html#Network
+ [3]: Administration.html#Installation
+ [4]: Administration.html#Dependencies
+ [5]: Administration.html#Configuration
+ [6]: Administration.html#Initialization
+ [7]: Administration.html#Running
+ [8]: Administration.html#Monitoring
+ [9]: Administration.html#Logging
+ [10]: Administration.html#Recovery
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Analytics.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Analytics.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Analytics.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Analytics.mdtext Tue Nov 1 14:49:30 2011
@@ -24,11 +24,11 @@ Notice: Licensed to the Apache Softwa
* * *
-## <a id=SECTION00900000000000000000></a> Analytics
+## <a id=Analytics></a> Analytics
Accumulo supports more advanced data processing than simply keeping keys sorted and performing efficient lookups. Analytics can be developed by using MapReduce and Iterators in conjunction with Accumulo tables.
-## <a id=SECTION00910000000000000000></a> MapReduce
+## <a id=MapReduce></a> MapReduce
Accumulo tables can be used as the source and destination of MapReduce jobs. To use a Accumulo table with a MapReduce job (specifically with the new Hadoop API as of version 0.20), configure the job parameters to use the AccumuloInputFormat and AccumuloOutputFormat. Accumulo specific parameters can be set via these two format classes to do the following:
@@ -36,7 +36,7 @@ Accumulo tables can be used as the sourc
* Restrict the scan to a range of rows
* Restrict the input to a subset of available columns
-### <a id=SECTION00911000000000000000></a> Mapper and Reducer classes
+### <a id=Mapper_and_Reducer_classes></a> Mapper and Reducer classes
To read from a Accumulo table create a Mapper with the following class parameterization and be sure to configure the AccumuloInputFormat.
@@ -66,7 +66,7 @@ To write to a Accumulo table, create a R
The Text object passed as the output should contain the name of the table to which this mutation should be applied. The Text can be null in which case the mutation will be applied to the default table name specified in the AccumuloOutputFormat options.
-### <a id=SECTION00912000000000000000></a> AccumuloInputFormat options
+### <a id=AccumuloInputFormat_options></a> AccumuloInputFormat options
Job job = new Job(getConf());
@@ -104,7 +104,7 @@ To use a regular expression to match row
AccumuloInputFormat.setRegex(job, RegexType.ROW, "^.*");
-### <a id=SECTION00913000000000000000></a> AccumuloOutputFormat options
+### <a id=AccumuloOutputFormat_options></a> AccumuloOutputFormat options
boolean createTables = true;
@@ -130,7 +130,7 @@ To use a regular expression to match row
An example of using MapReduce with Accumulo can be found at
accumulo/docs/examples/README.mapred
-## <a id=SECTION00920000000000000000></a> Aggregating Iterators
+## <a id=Aggregating_Iterators></a> Aggregating Iterators
Many applications can benefit from the ability to aggregate values across common keys. This can be done via aggregating iterators and is similar to the Reduce step in MapReduce. This provides the ability to define online, incrementally updated analytics without the overhead or latency associated with batch-oriented MapReduce jobs.
@@ -138,11 +138,11 @@ All that is needed to aggregate values o
The only restriction on an aggregating iterator is that the aggregator developer should not assume that all values for a given key have been seen, since new mutations can be inserted at anytime. This precludes using the total number of values in the aggregation such as when calculating an average, for example.
-### <a id=SECTION00921000000000000000></a> Feature Vectors
+### <a id=Feature_Vectors></a> Feature Vectors
An interesting use of aggregating iterators within a Accumulo table is to store feature vectors for use in machine learning algorithms. For example, many algorithms such as k-means clustering, support vector machines, anomaly detection, etc. use the concept of a feature vector and the calculation of distance metrics to learn a particular model. The columns in a Accumulo table can be used to efficiently store sparse features and their weights to be incrementally updated via the use of an aggregating iterator.
-## <a id=SECTION00930000000000000000></a> Statistical Modeling
+## <a id=Statistical_Modeling></a> Statistical Modeling
Statistical models that need to be updated by many machines in parallel could be similarly stored within a Accumulo table. For example, a MapReduce job that is iteratively updating a global statistical model could have each map or reduce worker reference the parts of the model to be read and updated through an embedded Accumulo client.
@@ -153,7 +153,7 @@ Using Accumulo this way enables efficien
* * *
- [1]: Analytics.html#SECTION00910000000000000000
- [2]: Analytics.html#SECTION00920000000000000000
- [3]: Analytics.html#SECTION00930000000000000000
+ [1]: Analytics.html#MapReduce
+ [2]: Analytics.html#Aggregating_Iterators
+ [3]: Analytics.html#Statistical_Modeling
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Contents.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Contents.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Contents.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Contents.mdtext Tue Nov 1 14:49:30 2011
@@ -19,7 +19,7 @@ Notice: Licensed to the Apache Softwa
-### <a id=SECTION00100000000000000000></a> Contents
+### <a id=Contents></a> Contents
* [Introduction][1]
* [Accumulo Design][2]
@@ -165,83 +165,83 @@ Notice: Licensed to the Apache Softwa
[1]: Introduction.html
[2]: Accumulo_Design.html
- [3]: Accumulo_Design.html#SECTION00310000000000000000
- [4]: Accumulo_Design.html#SECTION00320000000000000000
- [5]: Accumulo_Design.html#SECTION00330000000000000000
- [6]: Accumulo_Design.html#SECTION00331000000000000000
- [7]: Accumulo_Design.html#SECTION00332000000000000000
- [8]: Accumulo_Design.html#SECTION00333000000000000000
- [9]: Accumulo_Design.html#SECTION00334000000000000000
- [10]: Accumulo_Design.html#SECTION00335000000000000000
- [11]: Accumulo_Design.html#SECTION00340000000000000000
- [12]: Accumulo_Design.html#SECTION00350000000000000000
- [13]: Accumulo_Design.html#SECTION00360000000000000000
- [14]: Accumulo_Design.html#SECTION00370000000000000000
+ [3]: Accumulo_Design.html#Data_Model
+ [4]: Accumulo_Design.html#Architecture
+ [5]: Accumulo_Design.html#Components
+ [6]: Accumulo_Design.html#Tablet_Server
+ [7]: Accumulo_Design.html#Loggers
+ [8]: Accumulo_Design.html#Garbage_Collector
+ [9]: Accumulo_Design.html#Master
+ [10]: Accumulo_Design.html#Client
+ [11]: Accumulo_Design.html#Data_Management
+ [12]: Accumulo_Design.html#Tablet_Service
+ [13]: Accumulo_Design.html#Compactions
+ [14]: Accumulo_Design.html#Fault-Tolerance
[15]: Accumulo_Shell.html
- [16]: Accumulo_Shell.html#SECTION00410000000000000000
- [17]: Accumulo_Shell.html#SECTION00420000000000000000
- [18]: Accumulo_Shell.html#SECTION00430000000000000000
+ [16]: Accumulo_Shell.html#Basic_Administration
+ [17]: Accumulo_Shell.html#Table_Maintenance
+ [18]: Accumulo_Shell.html#User_Administration
[19]: Writing_Accumulo_Clients.html
- [20]: Writing_Accumulo_Clients.html#SECTION00510000000000000000
- [21]: Writing_Accumulo_Clients.html#SECTION00511000000000000000
- [22]: Writing_Accumulo_Clients.html#SECTION00520000000000000000
- [23]: Writing_Accumulo_Clients.html#SECTION00521000000000000000
- [24]: Writing_Accumulo_Clients.html#SECTION00522000000000000000
+ [20]: Writing_Accumulo_Clients.html#Writing_Data
+ [21]: Writing_Accumulo_Clients.html#BatchWriter
+ [22]: Writing_Accumulo_Clients.html#Reading_Data
+ [23]: Writing_Accumulo_Clients.html#Scanner
+ [24]: Writing_Accumulo_Clients.html#BatchScanner
[25]: Table_Configuration.html
- [26]: Table_Configuration.html#SECTION00610000000000000000
- [27]: Table_Configuration.html#SECTION00611000000000000000
- [28]: Table_Configuration.html#SECTION00612000000000000000
- [29]: Table_Configuration.html#SECTION00620000000000000000
- [30]: Table_Configuration.html#SECTION00630000000000000000
- [31]: Table_Configuration.html#SECTION00640000000000000000
- [32]: Table_Configuration.html#SECTION00641000000000000000
- [33]: Table_Configuration.html#SECTION00642000000000000000
- [34]: Table_Configuration.html#SECTION00650000000000000000
- [35]: Table_Configuration.html#SECTION00651000000000000000
- [36]: Table_Configuration.html#SECTION00652000000000000000
- [37]: Table_Configuration.html#SECTION00660000000000000000
- [38]: Table_Configuration.html#SECTION00670000000000000000
+ [26]: Table_Configuration.html#Locality_Groups
+ [27]: Table_Configuration.html#Managing_Locality_Groups_via_the_Shell
+ [28]: Table_Configuration.html#Managing_Locality_Groups_via_the_Client_API
+ [29]: Table_Configuration.html#Constraints
+ [30]: Table_Configuration.html#Bloom_Filters
+ [31]: Table_Configuration.html#Iterators
+ [32]: Table_Configuration.html#Setting_Iterators_via_the_Shell
+ [33]: Table_Configuration.html#Setting_Iterators_Programmatically
+ [34]: Table_Configuration.html#Versioning_Iterators_and_Timestamps
+ [35]: Table_Configuration.html#Logical_Time
+ [36]: Table_Configuration.html#Deletes
+ [37]: Table_Configuration.html#Filtering_Iterators
+ [38]: Table_Configuration.html#Aggregating_Iterators
[39]: Table_Design.html
- [40]: Table_Design.html#SECTION00710000000000000000
- [41]: Table_Design.html#SECTION00720000000000000000
- [42]: Table_Design.html#SECTION00730000000000000000
- [43]: Table_Design.html#SECTION00740000000000000000
- [44]: Table_Design.html#SECTION00750000000000000000
+ [40]: Table_Design.html#Basic_Table
+ [41]: Table_Design.html#RowID_Design
+ [42]: Table_Design.html#Indexing
+ [43]: Table_Design.html#Entity-Attribute_and_Graph_Tables
+ [44]: Table_Design.html#Document-Partitioned_Indexing
[45]: High_Speed_Ingest.html
- [46]: High_Speed_Ingest.html#SECTION00810000000000000000
- [47]: High_Speed_Ingest.html#SECTION00820000000000000000
- [48]: High_Speed_Ingest.html#SECTION00830000000000000000
- [49]: High_Speed_Ingest.html#SECTION00840000000000000000
+ [46]: High_Speed_Ingest.html#Pre-Splitting_New_Tables
+ [47]: High_Speed_Ingest.html#Multiple_Ingester_Clients
+ [48]: High_Speed_Ingest.html#Bulk_Ingest
+ [49]: High_Speed_Ingest.html#MapReduce_Ingest
[50]: Analytics.html
- [51]: Analytics.html#SECTION00910000000000000000
- [52]: Analytics.html#SECTION00911000000000000000
- [53]: Analytics.html#SECTION00912000000000000000
- [54]: Analytics.html#SECTION00913000000000000000
- [55]: Analytics.html#SECTION00920000000000000000
- [56]: Analytics.html#SECTION00921000000000000000
- [57]: Analytics.html#SECTION00930000000000000000
+ [51]: Analytics.html#MapReduce
+ [52]: Analytics.html#Mapper_and_Reducer_classes
+ [53]: Analytics.html#AccumuloInputFormat_options
+ [54]: Analytics.html#AccumuloOutputFormat_options
+ [55]: Analytics.html#Aggregating_Iterators
+ [56]: Analytics.html#Feature_Vectors
+ [57]: Analytics.html#Statistical_Modeling
[58]: Security.html
- [59]: Security.html#SECTION001010000000000000000
- [60]: Security.html#SECTION001020000000000000000
- [61]: Security.html#SECTION001030000000000000000
- [62]: Security.html#SECTION001040000000000000000
- [63]: Security.html#SECTION001050000000000000000
+ [59]: Security.html#Security_Label_Expressions
+ [60]: Security.html#Security_Label_Expression_Syntax
+ [61]: Security.html#Authorization
+ [62]: Security.html#Secure_Authorizations_Handling
+ [63]: Security.html#Query_Services_Layer
[64]: Administration.html
- [65]: Administration.html#SECTION001110000000000000000
- [66]: Administration.html#SECTION001120000000000000000
- [67]: Administration.html#SECTION001130000000000000000
- [68]: Administration.html#SECTION001140000000000000000
- [69]: Administration.html#SECTION001150000000000000000
- [70]: Administration.html#SECTION001151000000000000000
- [71]: Administration.html#SECTION001152000000000000000
- [72]: Administration.html#SECTION001153000000000000000
- [73]: Administration.html#SECTION001154000000000000000
- [74]: Administration.html#SECTION001160000000000000000
- [75]: Administration.html#SECTION001170000000000000000
- [76]: Administration.html#SECTION001171000000000000000
- [77]: Administration.html#SECTION001172000000000000000
- [78]: Administration.html#SECTION001180000000000000000
- [79]: Administration.html#SECTION001190000000000000000
- [80]: Administration.html#SECTION0011100000000000000000
+ [65]: Administration.html#Hardware
+ [66]: Administration.html#Network
+ [67]: Administration.html#Installation
+ [68]: Administration.html#Dependencies
+ [69]: Administration.html#Configuration
+ [70]: Administration.html#Edit_conf/accumulo-env.sh
+ [71]: Administration.html#Cluster_Specification
+ [72]: Administration.html#Accumulo_Settings
+ [73]: Administration.html#Deploy_Configuration
+ [74]: Administration.html#Initialization
+ [75]: Administration.html#Running
+ [76]: Administration.html#Starting_Accumulo
+ [77]: Administration.html#Stopping_Accumulo
+ [78]: Administration.html#Monitoring
+ [79]: Administration.html#Logging
+ [80]: Administration.html#Recovery
[81]: Shell_Commands.html
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/High_Speed_Ingest.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/High_Speed_Ingest.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/High_Speed_Ingest.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/High_Speed_Ingest.mdtext Tue Nov 1 14:49:30 2011
@@ -25,11 +25,11 @@ Notice: Licensed to the Apache Softwa
* * *
-## <a id=SECTION00800000000000000000></a> High-Speed Ingest
+## <a id=High-Speed_Ingest></a> High-Speed Ingest
Accumulo is often used as part of a larger data processing and storage system. To maximize the performance of a parallel system involving Accumulo, the ingestion and query components should be designed to provide enough parallelism and concurrency to avoid creating bottlenecks for users and other systems writing to and reading from Accumulo. There are several ways to achieve high ingest performance.
-## <a id=SECTION00810000000000000000></a> Pre-Splitting New Tables
+## <a id=Pre-Splitting_New_Tables></a> Pre-Splitting New Tables
New tables consist of a single tablet by default. As mutations are applied, the table grows and splits into multiple tablets which are balanced by the Master across TabletServers. This implies that the aggregate ingest rate will be limited to fewer servers than are available within the cluster until the table has reached the point where there are tablets on every TabletServer.
@@ -41,13 +41,13 @@ Pre-splitting a table ensures that there
For the purposes of providing parallelism to ingest it is not necessary to create more tablets than there are physical machines within the cluster as the aggregate ingest rate is a function of the number of physical machines. Note that the aggregate ingest rate is still subject to the number of machines running ingest clients, and the distribution of rowIDs across the table. The aggregation ingest rate will be suboptimal if there are many inserts into a small number of rowIDs.
-## <a id=SECTION00820000000000000000></a> Multiple Ingester Clients
+## <a id=Multiple_Ingester_Clients></a> Multiple Ingester Clients
Accumulo is capable of scaling to very high rates of ingest, which is dependent upon not just the number of TabletServers in operation but also the number of ingest clients. This is because a single client, while capable of batching mutations and sending them to all TabletServers, is ultimately limited by the amount of data that can be processed on a single machine. The aggregate ingest rate will scale linearly with the number of clients up to the point at which either the aggregate I/O of TabletServers or total network bandwidth capacity is reached.
In operational settings where high rates of ingest are paramount, clusters are often configured to dedicate some number of machines solely to running Ingester Clients. The exact ratio of clients to TabletServers necessary for optimum ingestion rates will vary according to the distribution of resources per machine and by data type.
-## <a id=SECTION00830000000000000000></a> Bulk Ingest
+## <a id=Bulk_Ingest></a> Bulk Ingest
Accumulo supports the ability to import files produced by an external process such as MapReduce into an existing table. In some cases it may be faster to load data this way rather than via ingesting through clients using BatchWriters. This allows a large number of machines to format data the way Accumulo expects. The new files can then simply be introduced to Accumulo via a shell command.
@@ -75,7 +75,7 @@ Note that the paths referenced are direc
A complete example of using Bulk Ingest can be found at
accumulo/docs/examples/README.bulkIngest
-## <a id=SECTION00840000000000000000></a> MapReduce Ingest
+## <a id=MapReduce_Ingest></a> MapReduce Ingest
It is possible to efficiently write many mutations to Accumulo in parallel via a MapReduce job. In this scenario the MapReduce is written to process data that lives in HDFS and write mutations to Accumulo using the AccumuloOutputFormat. See the MapReduce section under Analytics for details.
@@ -87,8 +87,8 @@ accumulo/docs/examples/README.mapred
* * *
- [1]: High_Speed_Ingest.html#SECTION00810000000000000000
- [2]: High_Speed_Ingest.html#SECTION00820000000000000000
- [3]: High_Speed_Ingest.html#SECTION00830000000000000000
- [4]: High_Speed_Ingest.html#SECTION00840000000000000000
+ [1]: High_Speed_Ingest.html#Pre-Splitting_New_Tables
+ [2]: High_Speed_Ingest.html#Multiple_Ingester_Clients
+ [3]: High_Speed_Ingest.html#Bulk_Ingest
+ [4]: High_Speed_Ingest.html#MapReduce_Ingest
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Introduction.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Introduction.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Introduction.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Introduction.mdtext Tue Nov 1 14:49:30 2011
@@ -16,7 +16,7 @@ Notice: Licensed to the Apache Softwa
specific language governing permissions and limitations
under the License.
-## <a id=SECTION00200000000000000000></a> Introduction
+## <a id=Introduction></a> Introduction
Accumulo is a highly scalable structured store based on Google's BigTable. Accumulo is written in Java and operates over the Hadoop Distributed File System (HDFS), which is part of the popular Apache Hadoop project. Accumulo supports efficient storage and retrieval of structured data, including queries for ranges, and provides support for using Accumulo tables as input and output for MapReduce jobs.
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Security.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Security.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Security.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Security.mdtext Tue Nov 1 14:49:30 2011
@@ -26,11 +26,11 @@ Notice: Licensed to the Apache Softwa
* * *
-## <a id=SECTION001000000000000000000></a> Security
+## <a id=Security></a> Security
Accumulo extends the BigTable data model to implement a security mechanism known as cell-level security. Every key-value pair has its own security label, stored under the column visibility element of the key, which is used to determine whether a given user meets the security requirements to read the value. This enables data of various security levels to be stored within the same row, and users of varying degrees of access to query the same table, while preserving data confidentiality.
-## <a id=SECTION001010000000000000000></a> Security Label Expressions
+## <a id=Security_Label_Expressions></a> Security Label Expressions
When mutations are applied, users can specify a security label for each value. This is done as the Mutation is created by passing a ColumnVisibility object to the put() method:
@@ -47,7 +47,7 @@ When mutations are applied, users can sp
mutation.put(colFam, colQual, colVis, timestamp, value);
-## <a id=SECTION001020000000000000000></a> Security Label Expression Syntax
+## <a id=Security_Label_Expression_Syntax></a> Security Label Expression Syntax
Security labels consist of a set of user-defined tokens that are required to read the value the label is associated with. The set of tokens required can be specified using syntax that supports logical AND and OR combinations of tokens, as well as nesting groups of tokens together.
@@ -77,7 +77,7 @@ These can be specified alone or combined
When both `|` and `&` operators are used, parentheses must be used to specify precedence of the operators.
-## <a id=SECTION001030000000000000000></a> Authorization
+## <a id=Authorization></a> Authorization
When clients attempt to read data from Accumulo, any security labels present are examined against the set of authorizations passed by the client code when the Scanner or BatchScanner are created. If the authorizations are determined to be insufficient to satisfy the security label, the value is suppressed from the set of results sent back to the client.
@@ -90,13 +90,13 @@ Authorizations are specified as a comma-
Scanner s = connector.createScanner("table", auths);
-## <a id=SECTION001040000000000000000></a> Secure Authorizations Handling
+## <a id=Secure_Authorizations_Handling></a> Secure Authorizations Handling
Because the client can pass any authorization tokens to Accumulo, applications must be designed to obtain users' authorization tokens from a trusted 3rd party rather than having the users specify their authorizations directly.
Often production systems will integrate with Public-Key Infrastructure (PKI) and designate client code within the query layer to negotiate with PKI servers in order to authenticate users and retrieve their authorization tokens (credentials). This requires users to specify only the information necessary to authenticate themselves to the system. Once user identity is established, their credentials can be accessed by the client code and passed to Accumulo outside of the reach of the user.
-## <a id=SECTION001050000000000000000></a> Query Services Layer
+## <a id=Query_Services_Layer></a> Query Services Layer
Since the primary method of interaction with Accumulo is through the Java API, production environments often call for the implementation of a Query layer. This can be done using web services in containers such as Apache Tomcat, but is not a requirement. The Query Services Layer provides a mechanism for providing a platform on which user facing applications can be built. This allows the application designers to isolate potentially complex query logic, and enables a convenient point at which to perform essential security functions.
@@ -109,9 +109,9 @@ Typically, the query services layer sits
* * *
- [1]: Security.html#SECTION001010000000000000000
- [2]: Security.html#SECTION001020000000000000000
- [3]: Security.html#SECTION001030000000000000000
- [4]: Security.html#SECTION001040000000000000000
- [5]: Security.html#SECTION001050000000000000000
+ [1]: Security.html#Security_Label_Expressions
+ [2]: Security.html#Security_Label_Expression_Syntax
+ [3]: Security.html#Authorization
+ [4]: Security.html#Secure_Authorizations_Handling
+ [5]: Security.html#Query_Services_Layer
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Shell_Commands.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Shell_Commands.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Shell_Commands.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Shell_Commands.mdtext Tue Nov 1 14:49:30 2011
@@ -16,7 +16,7 @@ Notice: Licensed to the Apache Softwa
specific language governing permissions and limitations
under the License.
-## <a id=SECTION001200000000000000000></a> Shell Commands
+## <a id=Shell_Commands></a> Shell Commands
**?**
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Configuration.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Configuration.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Configuration.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Configuration.mdtext Tue Nov 1 14:49:30 2011
@@ -28,17 +28,17 @@ Notice: Licensed to the Apache Softwa
* * *
-## <a id=SECTION00600000000000000000></a> Table Configuration
+## <a id=Table_Configuration></a> Table Configuration
Accumulo tables have a few options that can be configured to alter the default behavior of Accumulo as well as improve performance based on the data stored. These include locality groups, constraints, and iterators.
-## <a id=SECTION00610000000000000000></a> Locality Groups
+## <a id=Locality_Groups></a> Locality Groups
Accumulo supports storing of sets of column families separately on disk to allow clients to scan over columns that are frequently used together efficient and to avoid scanning over column families that are not requested. After a locality group is set Scanner and BatchScanner operations will automatically take advantage of them whenever the fetchColumnFamilies() method is used.
By default tables place all column families into the same ``default" locality group. Additional locality groups can be configured anytime via the shell or programmatically as follows:
-### <a id=SECTION00611000000000000000></a> Managing Locality Groups via the Shell
+### <a id=Managing_Locality_Groups_via_the_Shell></a> Managing Locality Groups via the Shell
usage: setgroups <group>=<col fam>{,<col fam>}{ <group>=<col fam>{,<col
@@ -50,7 +50,7 @@ By default tables place all column famil
group_one=colf1,colf2
-### <a id=SECTION00612000000000000000></a> Managing Locality Groups via the Client API
+### <a id=Managing_Locality_Groups_via_the_Client_API></a> Managing Locality Groups via the Client API
Connector conn;
@@ -82,7 +82,7 @@ The assignment of Column Families to Loc
user@myinstance mytable> compact -t mytable
-## <a id=SECTION00620000000000000000></a> Constraints
+## <a id=Constraints></a> Constraints
Accumulo supports constraints applied on mutations at insert time. This can be used to disallow certain inserts according to a user defined policy. Any mutation that fails to meet the requirements of the constraint is rejected and sent back to the client.
@@ -108,7 +108,7 @@ An example of constraints can be found i
accumulo/docs/examples/README.constraints with corresponding code under
accumulo/src/examples/main/java/accumulo/examples/constraints .
-## <a id=SECTION00630000000000000000></a> Bloom Filters
+## <a id=Bloom_Filters></a> Bloom Filters
As mutations are applied to a Accumulo table, several files are created per tablet. If bloom filters are enabled, Accumulo will create and load a small data structure into memory to determine whether a file contains a given key before opening the file. This can speed up lookups considerably.
@@ -121,11 +121,11 @@ To enable bloom filters, enter the follo
An extensive example of using Bloom Filters can be found at
accumulo/docs/examples/README.bloom .
-## <a id=SECTION00640000000000000000></a> Iterators
+## <a id=Iterators></a> Iterators
Iterators provide a modular mechanism for adding functionality to be executed by TabletServers when scanning or compacting data. This allows users to efficiently summarize, filter, and aggregate data. In fact, the built-in features of cell-level security and age-off are implemented using Iterators.
-### <a id=SECTION00641000000000000000></a> Setting Iterators via the Shell
+### <a id=Setting_Iterators_via_the_Shell></a> Setting Iterators via the Shell
usage: setiter [-?] -agg | -class <name> | -filter | -nolabel |
@@ -135,7 +135,7 @@ Iterators provide a modular mechanism fo
user@myinstance mytable> setiter -t mytable -scan -p 10 -n myiter
-### <a id=SECTION00642000000000000000></a> Setting Iterators Programmatically
+### <a id=Setting_Iterators_Programmatically></a> Setting Iterators Programmatically
scanner.setScanIterators(
@@ -155,7 +155,7 @@ Some iterators take additional parameter
Tables support separate Iterator settings to be applied at scan time, upon minor compaction and upon major compaction. For most uses, tables will have identical iterator settings for all three to avoid inconsistent results.
-## <a id=SECTION00650000000000000000></a> Versioning Iterators and Timestamps
+## <a id=Versioning_Iterators_and_Timestamps></a> Versioning Iterators and Timestamps
Accumulo provides the capability to manage versioned data through the use of timestamps within the Key. If a timestamp is not specified in the key created by the client then the system will set the timestamp to the current time. Two keys with identical rowIDs and columns but different timestamps are considered two versions of the same key. If two inserts are made into accumulo with the same rowID, column, and timestamp, then the behavior is non-deterministic.
@@ -174,7 +174,7 @@ The version policy can be changed by cha
table.iterator.majc.vers.opt.maxVersions=3
-### <a id=SECTION00651000000000000000></a> Logical Time
+### <a id=Logical_Time></a> Logical Time
Accumulo 1.2 introduces the concept of logical time. This ensures that timestamps set by accumulo always move forward. This helps avoid problems caused by TabletServers that have different time settings. The per tablet counter gives unique one up time stamps on a per mutation basis. When using time in milliseconds, if two things arrive within the same millisecond then both receive the same timestamp.
@@ -184,11 +184,11 @@ A table can be configured to use logical
user@myinstance> createtable -tl logical
-### <a id=SECTION00652000000000000000></a> Deletes
+### <a id=Deletes></a> Deletes
Deletes are special keys in accumulo that get sorted along will all the other data. When a delete key is inserted, accumulo will not show anything that has a timestamp less than or equal to the delete key. During major compaction, any keys older than a delete key are omitted from the new file created, and the omitted keys are removed from disk as part of the regular garbage collection process.
-## <a id=SECTION00660000000000000000></a> Filtering Iterators
+## <a id=Filtering_Iterators></a> Filtering Iterators
When scanning over a set of key-value pairs it is possible to apply an arbitrary filtering policy through the use of a FilteringIterator. These types of iterators return only key-value pairs that satisfy the filter logic. Accumulo has two built-in filtering iterators that can be configured on any table: AgeOff and RegEx. More can be added by writing a Java class that implements the
org.apache.accumulo.core.iterators.filter.Filter interface.
@@ -257,7 +257,7 @@ To see the iterator settings for a table
---------+------------------------------------------+------------------
-## <a id=SECTION00670000000000000000></a> Aggregating Iterators
+## <a id=Aggregating_Iterators></a> Aggregating Iterators
Accumulo allows aggregating iterators to be configured on tables and column families. When an aggregating iterator is set, the iterator is applied across the values associated with any keys that share rowID, column family, and column qualifier. This is similar to the reduce step in MapReduce, which applied some function to all the values associated with a particular key.
@@ -311,11 +311,11 @@ accumulo/src/examples/main/java/accumulo
* * *
- [1]: Table_Configuration.html#SECTION00610000000000000000
- [2]: Table_Configuration.html#SECTION00620000000000000000
- [3]: Table_Configuration.html#SECTION00630000000000000000
- [4]: Table_Configuration.html#SECTION00640000000000000000
- [5]: Table_Configuration.html#SECTION00650000000000000000
- [6]: Table_Configuration.html#SECTION00660000000000000000
- [7]: Table_Configuration.html#SECTION00670000000000000000
+ [1]: Table_Configuration.html#Locality_Groups
+ [2]: Table_Configuration.html#Constraints
+ [3]: Table_Configuration.html#Bloom_Filters
+ [4]: Table_Configuration.html#Iterators
+ [5]: Table_Configuration.html#Versioning_Iterators_and_Timestamps
+ [6]: Table_Configuration.html#Filtering_Iterators
+ [7]: Table_Configuration.html#Aggregating_Iterators
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Design.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Design.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Design.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Design.mdtext Tue Nov 1 14:49:30 2011
@@ -26,9 +26,9 @@ Notice: Licensed to the Apache Softwa
* * *
-## <a id=SECTION00700000000000000000></a> Table Design
+## <a id=Table_Design></a> Table Design
-## <a id=SECTION00710000000000000000></a> Basic Table
+## <a id=Basic_Table></a> Basic Table
Since Accumulo tables are sorted by row ID, each table can be thought of as being indexed by the row ID. Lookups performed row ID can be executed quickly, by doing a binary search, first across the tablets, and then within a tablet. Clients should choose a row ID carefully in order to support their desired application. A simple rule is to select a unique identifier as the row ID for each entity to be stored and assign all the other attributes to be tracked to be columns under this row ID. For example, if we have the following data in a comma-separated file:
@@ -59,7 +59,7 @@ We could then retrieve any of the column
System.out.println(entry.getValue().toString());
-## <a id=SECTION00720000000000000000></a> RowID Design
+## <a id=RowID_Design></a> RowID Design
Often it is necessary to transform the rowID in order to have rows ordered in a way that is optimal for anticipated access patterns. A good example of this is reversing the order of components of internet domain names in order to group rows of the same parent domain together:
@@ -95,7 +95,7 @@ It could also be done by adding a string
Appending dates provides the additional capability of restricting a scan to a given date range.
-## <a id=SECTION00730000000000000000></a> Indexing
+## <a id=Indexing></a> Indexing
In order to support lookups via more than one attribute of an entity, additional indexes can be built. However, because Accumulo tables can support any number of columns without specifying them beforehand, a single additional index will often suffice for supporting lookups of records in the main table. Here, the index has, as the rowID, the Value or Term from the main table, the column families are the same, and the column qualifier of the index table contains the rowID from the main table.
@@ -133,7 +133,7 @@ To support efficient lookups of multiple
One advantage of the dynamic schema capabilities of Accumulo is that different fields may be indexed into the same physical table. However, it may be necessary to create different index tables if the terms must be formatted differently in order to maintain proper sort order. For example, real numbers must be formatted differently than their usual notation in order to be sorted correctly. In these cases, usually one index per unique data type will suffice.
-## <a id=SECTION00740000000000000000></a> Entity-Attribute and Graph Tables
+## <a id=Entity-Attribute_and_Graph_Tables></a> Entity-Attribute and Graph Tables
Accumulo is ideal for storing entities and their attributes, especially of the attributes are sparse. It is often useful to join several datasets together on common entities within the same table. This can allow for the representation of graphs, including nodes, their attributes, and connections to other nodes.
@@ -151,7 +151,7 @@ For example, to keep track of employees,
To allow efficient updating of edge weights, an aggregating iterator can be configured to add the value of all mutations applied with the same key. These types of tables can easily be created from raw events by simply extracting the entities, attributes, and relationships from individual events and inserting the keys into Accumulo each with a count of 1. The aggregating iterator will take care of maintaining the edge weights.
-## <a id=SECTION00750000000000000000></a> Document-Partitioned Indexing
+## <a id=Document-Partitioned_Indexing></a> Document-Partitioned Indexing
Using a simple index as described above works well when looking for records that match one of a set of given criteria. When looking for records that match more than one criterion simultaneously, such as when looking for documents that contain all of the words `the' and `white' and `house', there are several issues.
@@ -194,11 +194,11 @@ Of course, fast servers will return thei
* * *
- [1]: Table_Design.html#SECTION00710000000000000000
- [2]: Table_Design.html#SECTION00720000000000000000
- [3]: Table_Design.html#SECTION00730000000000000000
- [4]: Table_Design.html#SECTION00740000000000000000
- [5]: Table_Design.html#SECTION00750000000000000000
+ [1]: Table_Design.html#Basic_Table
+ [2]: Table_Design.html#RowID_Design
+ [3]: Table_Design.html#Indexing
+ [4]: Table_Design.html#Entity-Attribute_and_Graph_Tables
+ [5]: Table_Design.html#Document-Partitioned_Indexing
[6]: img2.png
[7]: img3.png
[8]: img4.png
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Writing_Accumulo_Clients.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Writing_Accumulo_Clients.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Writing_Accumulo_Clients.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Writing_Accumulo_Clients.mdtext Tue Nov 1 14:49:30 2011
@@ -23,7 +23,7 @@ Notice: Licensed to the Apache Softwa
* * *
-## <a id=SECTION00500000000000000000></a> Writing Accumulo Clients
+## <a id=Writing_Accumulo_Clients></a> Writing Accumulo Clients
All clients must first identify the Accumulo instance to which they will be communicating. Code to do this is as follows:
@@ -35,7 +35,7 @@ All clients must first identify the Accu
Connector conn = new Connector(inst, "user","passwd".getBytes());
-## <a id=SECTION00510000000000000000></a> Writing Data
+## <a id=Writing_Data></a> Writing Data
Data are written to Accumulo by creating Mutation objects that represent all the changes to the columns of a single row. The changes are made atomically in the TabletServer. Clients then add Mutations to a BatchWriter which submits them to the appropriate TabletServers.
@@ -54,7 +54,7 @@ Mutations can be created thus:
mutation.put(colFam, colQual, colVis, timestamp, value);
-### <a id=SECTION00511000000000000000></a> BatchWriter
+### <a id=BatchWriter></a> BatchWriter
The BatchWriter is highly optimized to send Mutations to multiple TabletServers and automatically batches Mutations destined for the same TabletServer to amortize network overhead. Care must be taken to avoid changing the contents of any Object passed to the BatchWriter since it keeps objects in memory while batching.
@@ -76,11 +76,11 @@ Mutations are added to a BatchWriter thu
An example of using the batch writer can be found at
accumulo/docs/examples/README.batch
-## <a id=SECTION00520000000000000000></a> Reading Data
+## <a id=Reading_Data></a> Reading Data
Accumulo is optimized to quickly retrieve the value associated with a given key, and to efficiently return ranges of consecutive keys and their associated values.
-### <a id=SECTION00521000000000000000></a> Scanner
+### <a id=Scanner></a> Scanner
To retrieve data, Clients use a Scanner, which provides acts like an Iterator over keys and values. Scanners can be configured to start and stop at particular keys, and to return a subset of the columns available.
@@ -100,7 +100,7 @@ To retrieve data, Clients use a Scanner,
}
-### <a id=SECTION00522000000000000000></a> BatchScanner
+### <a id=BatchScanner></a> BatchScanner
For some types of access, it is more efficient to retrieve several ranges simultaneously. This arises when accessing a set of rows that are not consecutive whose IDs have been retrieved from a secondary index, for example.
@@ -128,6 +128,6 @@ accumulo/docs/examples/README.batch
* * *
- [1]: Writing_Accumulo_Clients.html#SECTION00510000000000000000
- [2]: Writing_Accumulo_Clients.html#SECTION00520000000000000000
+ [1]: Writing_Accumulo_Clients.html#Writing_Data
+ [2]: Writing_Accumulo_Clients.html#Reading_Data
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples.mdtext Tue Nov 1 14:49:30 2011
@@ -33,7 +33,7 @@ Commands intended to be run in bash are
Commands intended to be run in the Accumulo shell are prefixed by '>'.
-[1]: /accumulo/user_manual_1.3-incubating/Accumulo_Shell.html#SECTION00430000000000000000
+[1]: /accumulo/user_manual_1.3-incubating/Accumulo_Shell.html#User_Administration
[aggregation](examples/aggregation.html)
[batch](examples/batch.html)
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Accumulo_Design.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Accumulo_Design.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Accumulo_Design.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Accumulo_Design.mdtext Tue Nov 1 14:49:30 2011
@@ -28,9 +28,9 @@ Notice: Licensed to the Apache Softwa
* * *
-## <a id=SECTION00300000000000000000></a> Accumulo Design
+## <a id=Accumulo_Design></a> Accumulo Design
-## <a id=SECTION00310000000000000000></a> Data Model
+## <a id=Data_Model></a> Data Model
Accumulo provides a richer data model than simple key-value stores, but is not a fully relational database. Data is represented as key-value pairs, where the key and value are comprised of the following elements:
@@ -38,53 +38,53 @@ Accumulo provides a richer data model th
All elements of the Key and the Value are represented as byte arrays except for Timestamp, which is a Long. Accumulo sorts keys by element and lexicographically in ascending order. Timestamps are sorted in descending order so that later versions of the same Key appear first in a sequential scan. Tables consist of a set of sorted key-value pairs.
-## <a id=SECTION00320000000000000000></a> Architecture
+## <a id=Architecture></a> Architecture
Accumulo is a distributed data storage and retrieval system and as such consists of several architectural components, some of which run on many individual servers. Much of the work Accumulo does involves maintaining certain properties of the data, such as organization, availability, and integrity, across many commodity-class machines.
-## <a id=SECTION00330000000000000000></a> Components
+## <a id=Components></a> Components
An instance of Accumulo includes many TabletServers, write-ahead Logger servers, one Garbage Collector process, one Master server and many Clients.
-### <a id=SECTION00331000000000000000></a> Tablet Server
+### <a id=Tablet_Server></a> Tablet Server
The TabletServer manages some subset of all the tablets (partitions of tables). This includes receiving writes from clients, persisting writes to a writeâahead log, sorting new keyâvalue pairs in memory, periodically flushing sorted keyâvalue pairs to new files in HDFS, and responding to reads from clients, forming a mergeâsorted view of all keys and values from all the files it has created and the sorted inâmemory store.
TabletServers also perform recovery of a tablet that was previously on a server that failed, reapplying any writes found in the write-ahead log to the tablet.
-### <a id=SECTION00332000000000000000></a> Loggers
+### <a id=Loggers></a> Loggers
The Loggers accept updates to Tablet servers and write them to local on-disk storage. Each tablet server will write their updates to multiple loggers to preserve data in case of hardware failure.
-### <a id=SECTION00333000000000000000></a> Garbage Collector
+### <a id=Garbage_Collector></a> Garbage Collector
Accumulo processes will share files stored in HDFS. Periodically, the Garbage Collector will identify files that are no longer needed by any process, and delete them.
-### <a id=SECTION00334000000000000000></a> Master
+### <a id=Master></a> Master
The Accumulo Master is responsible for detecting and responding to TabletServer failure. It tries to balance the load across TabletServer by assigning tablets carefully and instructing TabletServers to migrate tablets when necessary. The Master ensures all tablets are assigned to one TabletServer each, and handles table creation, alteration, and deletion requests from clients. The Master also coordinates startup, graceful shutdown and recovery of changes in write-ahead logs when Tablet servers fail.
-### <a id=SECTION00335000000000000000></a> Client
+### <a id=Client></a> Client
Accumulo includes a client library that is linked to every application. The client library contains logic for finding servers managing a particular tablet, and communicating with TabletServers to write and retrieve key-value pairs.
-## <a id=SECTION00340000000000000000></a> Data Management
+## <a id=Data_Management></a> Data Management
Accumulo stores data in tables, which are partitioned into tablets. Tablets are partitioned on row boundaries so that all of the columns and values for a particular row are found together within the same tablet. The Master assigns Tablets to one TabletServer at a time. This enables row-level transactions to take place without using distributed locking or some other complicated synchronization mechanism. As clients insert and query data, and as machines are added and removed from the cluster, the Master migrates tablets to ensure they remain available and that the ingest and query load is balanced across the cluster.
![Image data_distribution][9]
-## <a id=SECTION00350000000000000000></a> Tablet Service
+## <a id=Tablet_Service></a> Tablet Service
When a write arrives at a TabletServer it is written to a WriteâAhead Log and then inserted into a sorted data structure in memory called a MemTable. When the MemTable reaches a certain size the TabletServer writes out the sorted key-value pairs to a file in HDFS called Indexed Sequential Access Method (ISAM) file. This process is called a minor compaction. A new MemTable is then created and the fact of the compaction is recorded in the WriteâAhead Log.
When a request to read data arrives at a TabletServer, the TabletServer does a binary search across the MemTable as well as the in-memory indexes associated with each ISAM file to find the relevant values. If clients are performing a scan, several keyâvalue pairs are returned to the client in order from the MemTable and the set of ISAM files by performing a mergeâsort as they are read.
-## <a id=SECTION00360000000000000000></a> Compactions
+## <a id=Compactions></a> Compactions
In order to manage the number of files per tablet, periodically the TabletServer performs Major Compactions of files within a tablet, in which some set of ISAM files are combined into one file. The previous files will eventually be removed by the Garbage Collector. This also provides an opportunity to permanently remove deleted keyâvalue pairs by omitting keyâvalue pairs suppressed by a delete entry when the new file is created.
-## <a id=SECTION00370000000000000000></a> Fault-Tolerance
+## <a id=Fault-Tolerance></a> Fault-Tolerance
If a TabletServer fails, the Master detects it and automatically reassigns the tablets assigned from the failed server to other servers. Any key-value pairs that were in memory at the time the TabletServer are automatically reapplied from the Write-Ahead Log to prevent any loss of data.
@@ -100,13 +100,13 @@ http://master-address:50095/monitor.
* * *
- [1]: Accumulo_Design.html#SECTION00310000000000000000
- [2]: Accumulo_Design.html#SECTION00320000000000000000
- [3]: Accumulo_Design.html#SECTION00330000000000000000
- [4]: Accumulo_Design.html#SECTION00340000000000000000
- [5]: Accumulo_Design.html#SECTION00350000000000000000
- [6]: Accumulo_Design.html#SECTION00360000000000000000
- [7]: Accumulo_Design.html#SECTION00370000000000000000
+ [1]: Accumulo_Design.html#Data_Model
+ [2]: Accumulo_Design.html#Architecture
+ [3]: Accumulo_Design.html#Components
+ [4]: Accumulo_Design.html#Data_Management
+ [5]: Accumulo_Design.html#Tablet_Service
+ [6]: Accumulo_Design.html#Compactions
+ [7]: Accumulo_Design.html#Fault-Tolerance
[8]: img1.png
[9]: ./data_distribution.png
[10]: ./failure_handling.png
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Accumulo_Shell.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Accumulo_Shell.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Accumulo_Shell.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Accumulo_Shell.mdtext Tue Nov 1 14:49:30 2011
@@ -24,7 +24,7 @@ Notice: Licensed to the Apache Softwa
* * *
-## <a id=SECTION00400000000000000000></a> Accumulo Shell
+## <a id=Accumulo_Shell></a> Accumulo Shell
Accumulo provides a simple shell that can be used to examine the contents and configuration settings of tables, apply individual mutations, and change configuration settings.
@@ -47,7 +47,7 @@ The shell will prompt for the correspond
-
-## <a id=SECTION00410000000000000000></a> Basic Administration
+## <a id=Basic_Administration></a> Basic Administration
The Accumulo shell can be used to create and delete tables, as well as to configure table and instance specific options.
@@ -84,7 +84,7 @@ The Shell can also be used to insert upd
row1 colf:colq [] value1
-## <a id=SECTION00420000000000000000></a> Table Maintenance
+## <a id=Table_Maintenance></a> Table Maintenance
The **compact** command instructs Accumulo to schedule a compaction of the table during which files are consolidated and deleted entries are removed.
@@ -102,7 +102,7 @@ The **flush** command instructs Accumulo
initiated...
-## <a id=SECTION00430000000000000000></a> User Administration
+## <a id=User_Administration></a> User Administration
The Shell can be used to add, remove, and grant privileges to users.
@@ -139,7 +139,7 @@ The Shell can be used to add, remove, an
* * *
- [1]: Accumulo_Shell.html#SECTION00410000000000000000
- [2]: Accumulo_Shell.html#SECTION00420000000000000000
- [3]: Accumulo_Shell.html#SECTION00430000000000000000
+ [1]: Accumulo_Shell.html#Basic_Administration
+ [2]: Accumulo_Shell.html#Table_Maintenance
+ [3]: Accumulo_Shell.html#User_Administration
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Administration.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Administration.mdtext?rev=1196033&r1=1196032&r2=1196033&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Administration.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Administration.mdtext Tue Nov 1 14:49:30 2011
@@ -31,21 +31,21 @@ Notice: Licensed to the Apache Softwa
* * *
-## <a id=SECTION001100000000000000000></a> Administration
+## <a id=Administration></a> Administration
-## <a id=SECTION001110000000000000000></a> Hardware
+## <a id=Hardware></a> Hardware
-Because we are running essentially two or three systems simultaneously layered across the cluster: HDFS, Accumulo, and MapReduce, it is typical for hardware to consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have at least one core and 2 - 4 GB each.
+Because we are running essentially two or three systems simultaneously layered across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have at least one core and 2 - 4 GB each.
One core running HDFS can typically keep 2 to 4 disks busy, so each machine may typically have as little as 2 x 300GB disks and as much as 4 x 1TB or 2TB disks.
It is possible to do with less than this, such as with 1u servers with 2 cores and 4GB each, but in this case it is recommended to only run up to two processes per machine - i.e. DataNode and TabletServer or DataNode and MapReduce worker but not all three. The constraint here is having enough available heap space for all the processes on a machine.
-## <a id=SECTION001120000000000000000></a> Network
+## <a id=Network></a> Network
Accumulo communicates via remote procedure calls over TCP/IP for both passing data and control messages. In addition, Accumulo uses HDFS clients to communicate with HDFS. To achieve good ingest and query performance, sufficient network bandwidth must be available between any two machines.
-## <a id=SECTION001130000000000000000></a> Installation
+## <a id=Installation></a> Installation
Choose a directory for the Accumulo installation. This directory will be referenced by the environment variable $ACCUMULO_HOME. Run the following:
@@ -55,15 +55,15 @@ Choose a directory for the Accumulo inst
Repeat this step at each machine within the cluster. Usually all machines have the same $ACCUMULO_HOME.
-## <a id=SECTION001140000000000000000></a> Dependencies
+## <a id=Dependencies></a> Dependencies
-Accumulo requires HDFS, MapReduce, and ZooKeeper to be configured and running before starting. Password-less SSH should be configured between at least the Accumulo master and TabletServer machines. It is also a good idea to run Network Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of sync, which can cause problems with automatically timestamped data. Accumulo will remove from the set of TabletServers those machines whose times differ too much from the master's.
+Accumulo requires HDFS and ZooKeeper to be configured and running before starting. Password-less SSH should be configured between at least the Accumulo master and TabletServer machines. It is also a good idea to run Network Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of sync, which can cause problems with automatically timestamped data. Accumulo will remove from the set of TabletServers those machines whose times differ too much from the master's.
-## <a id=SECTION001150000000000000000></a> Configuration
+## <a id=Configuration></a> Configuration
Accumulo is configured by editing several Shell and XML files found in $ACCUMULO_HOME/conf. The structure closely resembles Hadoop's configuration files.
-### <a id=SECTION001151000000000000000></a> Edit conf/accumulo-env.sh
+### <a id=Edit_conf/accumulo-env.sh></a> Edit conf/accumulo-env.sh
Accumulo needs to know where to find the software it depends on. Edit accumuloenv. sh and specify the following:
@@ -79,7 +79,7 @@ There are similar options for the master
Note that you will be specifying the Java heap space in accumulo-env.sh. You should make sure that the total heap space used for the Accumulo tserver and the Hadoop DataNode and TaskTracker is less than the available memory on each slave node in the cluster. On large clusters, it is recommended that the Accumulo master, Hadoop NameNode, secondary NameNode, and Hadoop JobTracker all be run on separate machines to allow them to use more heap space. If you are running these on the same machine on a small cluster, likewise make sure their heap space settings fit within the available memory.
-### <a id=SECTION001152000000000000000></a> Cluster Specification
+### <a id=Cluster_Specification></a> Cluster Specification
On the machine that will serve as the Accumulo master:
@@ -90,7 +90,7 @@ $ACCUMULO_HOME/conf/slaves, one per line
Note that if using domain names rather than IP addresses, DNS must be configured properly for all machines participating in the cluster. DNS can be a confusing source of errors.
-### <a id=SECTION001153000000000000000></a> Accumulo Settings
+### <a id=Accumulo_Settings></a> Accumulo Settings
Specify appropriate values for the following settings in
$ACCUMULO_HOME/conf/accumulo-site.xml :
@@ -108,18 +108,18 @@ $ACCUMULO_HOME/conf/accumulo-site.xml :
</property>
-This enables Accumulo to find Zookeeper. Accumulo uses ZooKeeper to coordinate settings between processes and helps finalize TabletServer failure.
+This enables Accumulo to find ZooKeeper. Accumulo uses ZooKeeper to coordinate settings between processes and helps finalize TabletServer failure.
Accumulo records all changes to tables to a write-ahead log before committing them to the table. The `walog' setting specifies the local directory on each machine to which write-ahead logs are written. This directory should exist on all machines acting as TabletServers.
Some settings can be modified via the Accumulo shell and take effect immediately. However, any settings that should be persisted across system restarts must be recorded in the accumulo-site.xml file.
-### <a id=SECTION001154000000000000000></a> Deploy Configuration
+### <a id=Deploy_Configuration></a> Deploy Configuration
Copy the masters, slaves, accumulo-env.sh, and if necessary, accumulo-site.xml from the
$ACCUMULO_HOME/conf/ directory on the master to all the machines specified in the slaves file.
-## <a id=SECTION001160000000000000000></a> Initialization
+## <a id=Initialization></a> Initialization
Accumulo must be initialized to create the structures it uses internally to locate data across the cluster. HDFS is required to be configured and running before Accumulo can be initialized.
@@ -128,33 +128,33 @@ $ACCUMULO_HOME/bin/accumulo init . This
The initialization script will prompt you to set a root password. Once Accumulo is initialized it can be started.
-## <a id=SECTION001170000000000000000></a> Running
+## <a id=Running></a> Running
-### <a id=SECTION001171000000000000000></a> Starting Accumulo
+### <a id=Starting_Accumulo></a> Starting Accumulo
-Make sure Hadoop is configured on all of the machines in the cluster, including access to a shared HDFS instance. Make sure HDFS and MapReduce are running. Make sure ZooKeeper is configured and running on at least one machine in the cluster. Start Accumulo using the bin/start-all.sh script.
+Make sure Hadoop is configured on all of the machines in the cluster, including access to a shared HDFS instance. Make sure HDFS and ZooKeeper are running. Make sure ZooKeeper is configured and running on at least one machine in the cluster. Start Accumulo using the bin/start-all.sh script.
To verify that Accumulo is running, check the Status page as described under _Monitoring_. In addition, the Shell can provide some information about the status of tables via reading the !METADATA table.
-### <a id=SECTION001172000000000000000></a> Stopping Accumulo
+### <a id=Stopping_Accumulo></a> Stopping Accumulo
To shutdown cleanly, run bin/stop-all.sh and the master will orchestrate the shutdown of all the tablet servers. Shutdown waits for all minor compactions to finish, so it may take some time for particular configurations.
-## <a id=SECTION001180000000000000000></a> Monitoring
+## <a id=Monitoring></a> Monitoring
The Accumulo Master provides an interface for monitoring the status and health of Accumulo components. This interface can be accessed by pointing a web browser to
http://accumulomaster:50095/status
-## <a id=SECTION001190000000000000000></a> Logging
+## <a id=Logging></a> Logging
Accumulo processes each write to a set of log files. By default these are found under
$ACCUMULO/logs/.
-## <a id=SECTION0011100000000000000000></a> Recovery
+## <a id=Recovery></a> Recovery
In the event of TabletServer failure or error on shutting Accumulo down, some mutations may not have been minor compacted to HDFS properly. In this case, Accumulo will automatically reapply such mutations from the write-ahead log either when the tablets from the failed server are reassigned by the Master, in the case of a single TabletServer failure or the next time Accumulo starts, in the event of failure during shutdown.
-Recovery is performed using the MapReduce framework and is shown on the Accumulo monitor status page. Once the MapReduce recovery is complete any tablets involved should return to an ``online" state. Until then those tablets will be unavailable to clients.
+Recovery is performed by asking the loggers to copy their write-ahead logs into HDFS. As the logs are copied, they are also sorted, so that tablets can easily find their missing updates. The copy/sort status of each file is displayed on Accumulo monitor status page. Once the recovery is complete any tablets involved should return to an ``online" state. Until then those tablets will be unavailable to clients.
The Accumulo client library is configured to retry failed mutations and in many cases clients will be able to continue processing after the recovery process without throwing an exception.
@@ -165,14 +165,14 @@ Note that because Accumulo uses timestam
* * *
- [1]: Administration.html#SECTION001110000000000000000
- [2]: Administration.html#SECTION001120000000000000000
- [3]: Administration.html#SECTION001130000000000000000
- [4]: Administration.html#SECTION001140000000000000000
- [5]: Administration.html#SECTION001150000000000000000
- [6]: Administration.html#SECTION001160000000000000000
- [7]: Administration.html#SECTION001170000000000000000
- [8]: Administration.html#SECTION001180000000000000000
- [9]: Administration.html#SECTION001190000000000000000
- [10]: Administration.html#SECTION0011100000000000000000
+ [1]: Administration.html#Hardware
+ [2]: Administration.html#Network
+ [3]: Administration.html#Installation
+ [4]: Administration.html#Dependencies
+ [5]: Administration.html#Configuration
+ [6]: Administration.html#Initialization
+ [7]: Administration.html#Running
+ [8]: Administration.html#Monitoring
+ [9]: Administration.html#Logging
+ [10]: Administration.html#Recovery