You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by bi...@apache.org on 2011/11/15 21:53:03 UTC
svn commit: r1202418 - in /incubator/accumulo:
branches/1.4/docs/src/user_manual/
branches/1.4/docs/src/user_manual/chapters/
site/trunk/content/accumulo/user_manual_1.3-incubating/
site/trunk/content/accumulo/user_manual_1.4-incubating/ trunk/docs/src...
Author: billie
Date: Tue Nov 15 20:53:02 2011
New Revision: 1202418
URL: http://svn.apache.org/viewvc?rev=1202418&view=rev
Log:
ACCUMULO-70 updated site based on user manual changes
Added:
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img6.png (with props)
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img7.png (with props)
Modified:
incubator/accumulo/branches/1.4/docs/src/user_manual/ (props changed)
incubator/accumulo/branches/1.4/docs/src/user_manual/chapters/table_configuration.tex
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Contents.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Shell_Commands.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Configuration.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Administration.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Contents.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Shell_Commands.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Table_Configuration.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Table_Design.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img2.png
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img3.png
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img4.png
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img5.png
incubator/accumulo/trunk/docs/src/user_manual/ (props changed)
incubator/accumulo/trunk/docs/src/user_manual/chapters/table_configuration.tex
Propchange: incubator/accumulo/branches/1.4/docs/src/user_manual/
------------------------------------------------------------------------------
--- svn:ignore (original)
+++ svn:ignore Tue Nov 15 20:53:02 2011
@@ -1 +1,2 @@
accumulo_user_manual.toc
+accumulo_user_manual.out
Modified: incubator/accumulo/branches/1.4/docs/src/user_manual/chapters/table_configuration.tex
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4/docs/src/user_manual/chapters/table_configuration.tex?rev=1202418&r1=1202417&r2=1202418&view=diff
==============================================================================
--- incubator/accumulo/branches/1.4/docs/src/user_manual/chapters/table_configuration.tex (original)
+++ incubator/accumulo/branches/1.4/docs/src/user_manual/chapters/table_configuration.tex Tue Nov 15 20:53:02 2011
@@ -456,8 +456,8 @@ never need to occur and major compaction
configure the file max and compaction ratio such that only merging minor
compactions occur and major compactions never occur. This should be avoided
because doing only merging minor compactions causes $O(N^2)$ work to be done.
-The amount of work done by major compactions is $O(N*\log_R(N))$ where $R$ is
-the compaction ratio.
+The amount of work done by major compactions is $O(N*\log_R(N))$ where
+\textit{R} is the compaction ratio.
Compactions can be initiated manually for a table. To initiate a minor
compaction, use the flush command in the shell. To initiate a major compaction,
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Contents.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Contents.mdtext?rev=1202418&r1=1202417&r2=1202418&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Contents.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Contents.mdtext Tue Nov 15 20:53:02 2011
@@ -77,88 +77,85 @@ Notice: Licensed to the Apache Softwa
* [Setting Iterators via the Shell][36]
* [Setting Iterators Programmatically][37]
+ * [Versioning Iterators and Timestamps][38]
+ * [Filtering Iterators][39]
- * [Versioning Iterators and Timestamps][38]
-
- * [Logical Time][39]
- * [Deletes][40]
-
- * [Filtering Iterators][41]
- * [Aggregating Iterators][42]
+ * [Aggregating Iterators][40]
+ * [Block Cache][41]
-* [Table Design][43]
+* [Table Design][42]
- * [Basic Table][44]
- * [RowID Design][45]
- * [Indexing][46]
- * [Entity-Attribute and Graph Tables][47]
- * [Document-Partitioned Indexing][48]
+ * [Basic Table][43]
+ * [RowID Design][44]
+ * [Indexing][45]
+ * [Entity-Attribute and Graph Tables][46]
+ * [Document-Partitioned Indexing][47]
-* [High-Speed Ingest][49]
+* [High-Speed Ingest][48]
- * [Pre-Splitting New Tables][50]
- * [Multiple Ingester Clients][51]
- * [Bulk Ingest][52]
- * [MapReduce Ingest][53]
+ * [Pre-Splitting New Tables][49]
+ * [Multiple Ingester Clients][50]
+ * [Bulk Ingest][51]
+ * [MapReduce Ingest][52]
-* [Analytics][54]
+* [Analytics][53]
- * [MapReduce][55]
+ * [MapReduce][54]
- * [Mapper and Reducer classes][56]
- * [AccumuloInputFormat options][57]
- * [AccumuloOutputFormat options][58]
+ * [Mapper and Reducer classes][55]
+ * [AccumuloInputFormat options][56]
+ * [AccumuloOutputFormat options][57]
- * [Aggregating Iterators][59]
+ * [Aggregating Iterators][58]
- * [Feature Vectors][60]
+ * [Feature Vectors][59]
- * [Statistical Modeling][61]
+ * [Statistical Modeling][60]
-* [Security][62]
+* [Security][61]
- * [Security Label Expressions][63]
- * [Security Label Expression Syntax][64]
- * [Authorization][65]
- * [Secure Authorizations Handling][66]
- * [Query Services Layer][67]
+ * [Security Label Expressions][62]
+ * [Security Label Expression Syntax][63]
+ * [Authorization][64]
+ * [Secure Authorizations Handling][65]
+ * [Query Services Layer][66]
-* [Administration][68]
+* [Administration][67]
- * [Hardware][69]
- * [Network][70]
- * [Installation][71]
- * [Dependencies][72]
- * [Configuration][73]
+ * [Hardware][68]
+ * [Network][69]
+ * [Installation][70]
+ * [Dependencies][71]
+ * [Configuration][72]
- * [Edit conf/accumulo-env.sh][74]
- * [Cluster Specification][75]
- * [Accumulo Settings][76]
- * [Deploy Configuration][77]
+ * [Edit conf/accumulo-env.sh][73]
+ * [Cluster Specification][74]
+ * [Accumulo Settings][75]
+ * [Deploy Configuration][76]
- * [Initialization][78]
- * [Running][79]
+ * [Initialization][77]
+ * [Running][78]
- * [Starting Accumulo][80]
- * [Stopping Accumulo][81]
+ * [Starting Accumulo][79]
+ * [Stopping Accumulo][80]
- * [Monitoring][82]
- * [Logging][83]
- * [Recovery][84]
+ * [Monitoring][81]
+ * [Logging][82]
+ * [Recovery][83]
-* [Shell Commands][85]
+* [Shell Commands][84]
@@ -200,51 +197,50 @@ Notice: Licensed to the Apache Softwa
[36]: Table_Configuration.html#Setting_Iterators_via_the_Shell
[37]: Table_Configuration.html#Setting_Iterators_Programmatically
[38]: Table_Configuration.html#Versioning_Iterators_and_Timestamps
- [39]: Table_Configuration.html#Logical_Time
- [40]: Table_Configuration.html#Deletes
- [41]: Table_Configuration.html#Filtering_Iterators
- [42]: Table_Configuration.html#Aggregating_Iterators
- [43]: Table_Design.html
- [44]: Table_Design.html#Basic_Table
- [45]: Table_Design.html#RowID_Design
- [46]: Table_Design.html#Indexing
- [47]: Table_Design.html#Entity-Attribute_and_Graph_Tables
- [48]: Table_Design.html#Document-Partitioned_Indexing
- [49]: High_Speed_Ingest.html
- [50]: High_Speed_Ingest.html#Pre-Splitting_New_Tables
- [51]: High_Speed_Ingest.html#Multiple_Ingester_Clients
- [52]: High_Speed_Ingest.html#Bulk_Ingest
- [53]: High_Speed_Ingest.html#MapReduce_Ingest
- [54]: Analytics.html
- [55]: Analytics.html#MapReduce
- [56]: Analytics.html#Mapper_and_Reducer_classes
- [57]: Analytics.html#AccumuloInputFormat_options
- [58]: Analytics.html#AccumuloOutputFormat_options
- [59]: Analytics.html#Aggregating_Iterators
- [60]: Analytics.html#Feature_Vectors
- [61]: Analytics.html#Statistical_Modeling
- [62]: Security.html
- [63]: Security.html#Security_Label_Expressions
- [64]: Security.html#Security_Label_Expression_Syntax
- [65]: Security.html#Authorization
- [66]: Security.html#Secure_Authorizations_Handling
- [67]: Security.html#Query_Services_Layer
- [68]: Administration.html
- [69]: Administration.html#Hardware
- [70]: Administration.html#Network
- [71]: Administration.html#Installation
- [72]: Administration.html#Dependencies
- [73]: Administration.html#Configuration
- [74]: Administration.html#Edit_conf/accumulo-env.sh
- [75]: Administration.html#Cluster_Specification
- [76]: Administration.html#Accumulo_Settings
- [77]: Administration.html#Deploy_Configuration
- [78]: Administration.html#Initialization
- [79]: Administration.html#Running
- [80]: Administration.html#Starting_Accumulo
- [81]: Administration.html#Stopping_Accumulo
- [82]: Administration.html#Monitoring
- [83]: Administration.html#Logging
- [84]: Administration.html#Recovery
- [85]: Shell_Commands.html
+ [39]: Table_Configuration.html#Filtering_Iterators
+ [40]: Table_Configuration.html#Aggregating_Iterators
+ [41]: Table_Configuration.html#Block_Cache
+ [42]: Table_Design.html
+ [43]: Table_Design.html#Basic_Table
+ [44]: Table_Design.html#RowID_Design
+ [45]: Table_Design.html#Indexing
+ [46]: Table_Design.html#Entity-Attribute_and_Graph_Tables
+ [47]: Table_Design.html#Document-Partitioned_Indexing
+ [48]: High_Speed_Ingest.html
+ [49]: High_Speed_Ingest.html#Pre-Splitting_New_Tables
+ [50]: High_Speed_Ingest.html#Multiple_Ingester_Clients
+ [51]: High_Speed_Ingest.html#Bulk_Ingest
+ [52]: High_Speed_Ingest.html#MapReduce_Ingest
+ [53]: Analytics.html
+ [54]: Analytics.html#MapReduce
+ [55]: Analytics.html#Mapper_and_Reducer_classes
+ [56]: Analytics.html#AccumuloInputFormat_options
+ [57]: Analytics.html#AccumuloOutputFormat_options
+ [58]: Analytics.html#Aggregating_Iterators
+ [59]: Analytics.html#Feature_Vectors
+ [60]: Analytics.html#Statistical_Modeling
+ [61]: Security.html
+ [62]: Security.html#Security_Label_Expressions
+ [63]: Security.html#Security_Label_Expression_Syntax
+ [64]: Security.html#Authorization
+ [65]: Security.html#Secure_Authorizations_Handling
+ [66]: Security.html#Query_Services_Layer
+ [67]: Administration.html
+ [68]: Administration.html#Hardware
+ [69]: Administration.html#Network
+ [70]: Administration.html#Installation
+ [71]: Administration.html#Dependencies
+ [72]: Administration.html#Configuration
+ [73]: Administration.html#Edit_conf/accumulo-env.sh
+ [74]: Administration.html#Cluster_Specification
+ [75]: Administration.html#Accumulo_Settings
+ [76]: Administration.html#Deploy_Configuration
+ [77]: Administration.html#Initialization
+ [78]: Administration.html#Running
+ [79]: Administration.html#Starting_Accumulo
+ [80]: Administration.html#Stopping_Accumulo
+ [81]: Administration.html#Monitoring
+ [82]: Administration.html#Logging
+ [83]: Administration.html#Recovery
+ [84]: Shell_Commands.html
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Shell_Commands.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Shell_Commands.mdtext?rev=1202418&r1=1202417&r2=1202418&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Shell_Commands.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Shell_Commands.mdtext Tue Nov 15 20:53:02 2011
@@ -336,7 +336,7 @@ Notice: Licensed to the Apache Softwa
usage: listscans [-?] [-np] [-ts <tablet server>]
description: list what scans are currently running in accumulo. See the
- accumulo.core.client.admin.ActiveScan javadoc for more information
+ org.apache.accumulo.core.client.admin.ActiveScan javadoc for more information
about columns.
-?,-help display this help
-np,-no-pagination disables pagination of output
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Configuration.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Configuration.mdtext?rev=1202418&r1=1202417&r2=1202418&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Configuration.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Table_Configuration.mdtext Tue Nov 15 20:53:02 2011
@@ -24,9 +24,8 @@ Notice: Licensed to the Apache Softwa
* [Constraints][10]
* [Bloom Filters][11]
* [Iterators][12]
-* [Versioning Iterators and Timestamps][13]
-* [Filtering Iterators][14]
-* [Aggregating Iterators][15]
+* [Aggregating Iterators][13]
+* [Block Cache][14]
* * *
@@ -157,7 +156,7 @@ Some iterators take additional parameter
Tables support separate Iterator settings to be applied at scan time, upon minor compaction and upon major compaction. For most uses, tables will have identical iterator settings for all three to avoid inconsistent results.
-## <a id=Versioning_Iterators_and_Timestamps></a> Versioning Iterators and Timestamps
+### <a id=Versioning_Iterators_and_Timestamps></a> Versioning Iterators and Timestamps
Accumulo provides the capability to manage versioned data through the use of timestamps within the Key. If a timestamp is not specified in the key created by the client then the system will set the timestamp to the current time. Two keys with identical rowIDs and columns but different timestamps are considered two versions of the same key. If two inserts are made into accumulo with the same rowID, column, and timestamp, then the behavior is non-deterministic.
@@ -176,7 +175,7 @@ The version policy can be changed by cha
table.iterator.majc.vers.opt.maxVersions=3
-### <a id=Logical_Time></a> Logical Time
+#### <a id=Logical_Time></a> Logical Time
Accumulo 1.2 introduces the concept of logical time. This ensures that timestamps set by accumulo always move forward. This helps avoid problems caused by TabletServers that have different time settings. The per tablet counter gives unique one up time stamps on a per mutation basis. When using time in milliseconds, if two things arrive within the same millisecond then both receive the same timestamp.
@@ -186,11 +185,11 @@ A table can be configured to use logical
user@myinstance> createtable -tl logical
-### <a id=Deletes></a> Deletes
+#### <a id=Deletes></a> Deletes
Deletes are special keys in accumulo that get sorted along will all the other data. When a delete key is inserted, accumulo will not show anything that has a timestamp less than or equal to the delete key. During major compaction, any keys older than a delete key are omitted from the new file created, and the omitted keys are removed from disk as part of the regular garbage collection process.
-## <a id=Filtering_Iterators></a> Filtering Iterators
+### <a id=Filtering_Iterators></a> Filtering Iterators
When scanning over a set of key-value pairs it is possible to apply an arbitrary filtering policy through the use of a FilteringIterator. These types of iterators return only key-value pairs that satisfy the filter logic. Accumulo has two built-in filtering iterators that can be configured on any table: AgeOff and RegEx. More can be added by writing a Java class that implements the
org.apache.accumulo.core.iterators.filter.Filter interface.
@@ -306,7 +305,27 @@ Additional Aggregators can be added by c
**org.apache.accumulo.core.iterators.aggregation.Aggregator** and adding a jar containing that class to Accumulo's lib directory.
An example of an aggregator can be found under
-accumulo/src/examples/main/java/accumulo/examples/aggregation/SortedSetAggregator.java
+accumulo/src/examples/main/java/org/apache/accumulo/examples/aggregation/SortedSetAggregator.java
+
+## <a id=Block_Cache></a> Block Cache
+
+In order to increase throughput of commonly accessed entries, Accumulo employs a block cache. This block cache buffers data in memory so that it doesn't have to be read off of disk. The RFile format that Accumulo prefers is a mix of index blocks and data blocks, where the index blocks are used to find the appropriate data blocks. Typical queries to Accumulo result in a binary search over several index blocks followed by a linear scan of one or more data blocks.
+
+The block cache can be configured on a per-table basis, and all tablets hosted on a tablet server share a single resource pool. To configure the size of the tablet server's block cache, set the following properties:
+
+
+ tserver.cache.data.size: Specifies the size of the cache for file data blocks.
+ tserver.cache.index.size: Specifies the size of the cache for file indices.
+
+
+To enable the block cache for your table, set the following properties:
+
+
+ table.cache.block.enable: Determines whether file (data) block cache is enabled.
+ table.cache.index.enable: Determines whether index cache is enabled.
+
+
+The block cache can have a significant effect on alleviating hot spots, as well as reducing query latency. It is enabled by default for the !METADATA table.
* * *
@@ -320,7 +339,6 @@ accumulo/src/examples/main/java/accumulo
[10]: Table_Configuration.html#Constraints
[11]: Table_Configuration.html#Bloom_Filters
[12]: Table_Configuration.html#Iterators
- [13]: Table_Configuration.html#Versioning_Iterators_and_Timestamps
- [14]: Table_Configuration.html#Filtering_Iterators
- [15]: Table_Configuration.html#Aggregating_Iterators
+ [13]: Table_Configuration.html#Aggregating_Iterators
+ [14]: Table_Configuration.html#Block_Cache
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Administration.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Administration.mdtext?rev=1202418&r1=1202417&r2=1202418&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Administration.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Administration.mdtext Tue Nov 15 20:53:02 2011
@@ -136,7 +136,7 @@ The initialization script will prompt yo
Make sure Hadoop is configured on all of the machines in the cluster, including access to a shared HDFS instance. Make sure HDFS and ZooKeeper are running. Make sure ZooKeeper is configured and running on at least one machine in the cluster. Start Accumulo using the bin/start-all.sh script.
-To verify that Accumulo is running, check the Status page as described under _Monitoring_. In addition, the Shell can provide some information about the status of tables via reading the !METADATA table.
+To verify that Accumulo is running, check the Status page as described under *Monitoring*. In addition, the Shell can provide some information about the status of tables via reading the !METADATA table.
### <a id=Stopping_Accumulo></a> Stopping Accumulo
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Contents.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Contents.mdtext?rev=1202418&r1=1202417&r2=1202418&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Contents.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Contents.mdtext Tue Nov 15 20:53:02 2011
@@ -78,14 +78,12 @@ Notice: Licensed to the Apache Softwa
* [Setting Iterators via the Shell][37]
* [Setting Iterators Programmatically][38]
+ * [Versioning Iterators and Timestamps][39]
+ * [Filters][40]
+ * [Aggregating Iterators][41]
- * [Versioning Iterators and Timestamps][39]
-
- * [Logical Time][40]
- * [Deletes][41]
-
- * [Filters][42]
- * [Aggregating Iterators][43]
+ * [Block Cache][42]
+ * [Compaction][43]
* [Pre-splitting tables][44]
* [Merging tablets][45]
* [Delete Range][46]
@@ -208,10 +206,10 @@ Notice: Licensed to the Apache Softwa
[37]: Table_Configuration.html#Setting_Iterators_via_the_Shell
[38]: Table_Configuration.html#Setting_Iterators_Programmatically
[39]: Table_Configuration.html#Versioning_Iterators_and_Timestamps
- [40]: Table_Configuration.html#Logical_Time
- [41]: Table_Configuration.html#Deletes
- [42]: Table_Configuration.html#Filters
- [43]: Table_Configuration.html#Aggregating_Iterators
+ [40]: Table_Configuration.html#Filters
+ [41]: Table_Configuration.html#Aggregating_Iterators
+ [42]: Table_Configuration.html#Block_Cache
+ [43]: Table_Configuration.html#Compaction
[44]: Table_Configuration.html#Pre-splitting_tables
[45]: Table_Configuration.html#Merging_tablets
[46]: Table_Configuration.html#Delete_Range
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Shell_Commands.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Shell_Commands.mdtext?rev=1202418&r1=1202417&r2=1202418&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Shell_Commands.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Shell_Commands.mdtext Tue Nov 15 20:53:02 2011
@@ -395,7 +395,7 @@ Notice: Licensed to the Apache Softwa
usage: listscans [-?] [-np] [-ts <tablet server>]
description: list what scans are currently running in accumulo. See the
- accumulo.core.client.admin.ActiveScan javadoc for more information
+ org.apache.accumulo.core.client.admin.ActiveScan javadoc for more information
about columns.
-?,-help display this help
-np,-no-pagination disables pagination of output
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Table_Configuration.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Table_Configuration.mdtext?rev=1202418&r1=1202417&r2=1202418&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Table_Configuration.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Table_Configuration.mdtext Tue Nov 15 20:53:02 2011
@@ -24,19 +24,18 @@ Notice: Licensed to the Apache Softwa
* [Constraints][10]
* [Bloom Filters][11]
* [Iterators][12]
-* [Versioning Iterators and Timestamps][13]
-* [Filters][14]
-* [Aggregating Iterators][15]
-* [Pre-splitting tables][16]
-* [Merging tablets][17]
-* [Delete Range][18]
-* [Cloning Tables][19]
+* [Block Cache][13]
+* [Compaction][14]
+* [Pre-splitting tables][15]
+* [Merging tablets][16]
+* [Delete Range][17]
+* [Cloning Tables][18]
* * *
## <a id=Table_Configuration></a> Table Configuration
-Accumulo tables have a few options that can be configured to alter the default behavior of Accumulo as well as improve performance based on the data stored. These include locality groups, constraints, and iterators.
+Accumulo tables have a few options that can be configured to alter the default behavior of Accumulo as well as improve performance based on the data stored. These include locality groups, constraints, bloom filters, iterators, and block cache.
## <a id=Locality_Groups></a> Locality Groups
@@ -161,7 +160,7 @@ Some iterators take additional parameter
Tables support separate Iterator settings to be applied at scan time, upon minor compaction and upon major compaction. For most uses, tables will have identical iterator settings for all three to avoid inconsistent results.
-## <a id=Versioning_Iterators_and_Timestamps></a> Versioning Iterators and Timestamps
+### <a id=Versioning_Iterators_and_Timestamps></a> Versioning Iterators and Timestamps
Accumulo provides the capability to manage versioned data through the use of timestamps within the Key. If a timestamp is not specified in the key created by the client then the system will set the timestamp to the current time. Two keys with identical rowIDs and columns but different timestamps are considered two versions of the same key. If two inserts are made into accumulo with the same rowID, column, and timestamp, then the behavior is non-deterministic.
@@ -180,7 +179,7 @@ The version policy can be changed by cha
table.iterator.majc.vers.opt.maxVersions=3
-### <a id=Logical_Time></a> Logical Time
+#### <a id=Logical_Time></a> Logical Time
Accumulo 1.2 introduces the concept of logical time. This ensures that timestamps set by accumulo always move forward. This helps avoid problems caused by TabletServers that have different time settings. The per tablet counter gives unique one up time stamps on a per mutation basis. When using time in milliseconds, if two things arrive within the same millisecond then both receive the same timestamp. When using time in milliseconds, accumulo set times will still always move forward and never backwards.
@@ -190,11 +189,11 @@ A table can be configured to use logical
user@myinstance> createtable -tl logical
-### <a id=Deletes></a> Deletes
+#### <a id=Deletes></a> Deletes
Deletes are special keys in accumulo that get sorted along will all the other data. When a delete key is inserted, accumulo will not show anything that has a timestamp less than or equal to the delete key. During major compaction, any keys older than a delete key are omitted from the new file created, and the omitted keys are removed from disk as part of the regular garbage collection process.
-## <a id=Filters></a> Filters
+### <a id=Filters></a> Filters
When scanning over a set of key-value pairs it is possible to apply an arbitrary filtering policy through the use of a Filter. Filters are types of iterators that return only key-value pairs that satisfy the filter logic. Accumulo has a few built-in filters that can be configured on any table: AgeOff, ColumnAgeOff, Timestamp, NoVis, and RegEx. More can be added by writing a Java class that extends the
org.apache.accumulo.core.iterators.Filter class.
@@ -240,7 +239,7 @@ To see the iterator settings for a table
---------+------------------------------------------+------------------
-## <a id=Aggregating_Iterators></a> Aggregating Iterators
+### <a id=Aggregating_Iterators></a> Aggregating Iterators
Accumulo allows aggregating iterators to be configured on tables and column families. When an aggregating iterator is set, the iterator is applied across the values associated with any keys that share rowID, column family, and column qualifier. This is similar to the reduce step in MapReduce, which applied some function to all the values associated with a particular key.
@@ -287,7 +286,65 @@ Additional Aggregators can be added by c
**org.apache.accumulo.core.iterators.aggregation.Aggregator** and adding a jar containing that class to Accumulo's lib directory.
An example of an aggregator can be found under
-accumulo/src/examples/main/java/accumulo/examples/aggregation/SortedSetAggregator.java
+accumulo/src/examples/main/java/org/apache/accumulo/examples/aggregation/SortedSetAggregator.java
+
+## <a id=Block_Cache></a> Block Cache
+
+In order to increase throughput of commonly accessed entries, Accumulo employs a block cache. This block cache buffers data in memory so that it doesn't have to be read off of disk. The RFile format that Accumulo prefers is a mix of index blocks and data blocks, where the index blocks are used to find the appropriate data blocks. Typical queries to Accumulo result in a binary search over several index blocks followed by a linear scan of one or more data blocks.
+
+The block cache can be configured on a per-table basis, and all tablets hosted on a tablet server share a single resource pool. To configure the size of the tablet server's block cache, set the following properties:
+
+
+ tserver.cache.data.size: Specifies the size of the cache for file data blocks.
+ tserver.cache.index.size: Specifies the size of the cache for file indices.
+
+
+To enable the block cache for your table, set the following properties:
+
+
+ table.cache.block.enable: Determines whether file (data) block cache is enabled.
+ table.cache.index.enable: Determines whether index cache is enabled.
+
+
+The block cache can have a significant effect on alleviating hot spots, as well as reducing query latency. It is enabled by default for the !METADATA table.
+
+## <a id=Compaction></a> Compaction
+
+As data is written to Accumulo it is buffered in memory. The data buffered in memory is eventually written to HDFS on a per tablet basis. Files can also be added to tablets directly by bulk import. In the background tablet servers run major compactions to merge multiple files into one. The tablet server has to decide which tablets to compact and which files within a tablet to compact. This decision is made using the compaction ratio, which is configurable on a per table basis. To configure this ratio modify the following property:
+
+
+ table.compaction.major.ratio
+
+
+Increasing this ratio will result in more files per tablet and less compaction work. More files per tablet means more higher query latency. So adjusting this ratio is a trade off between ingest and query performance. The ratio defaults to 3.
+
+The way the ratio works is that a set of files is compacted into one file if the sum of the sizes of the files in the set is larger than the ratio multiplied by the size of the largest file in the set. If this is not true for the set of all files in a tablet, the largest file is removed from consideration, and the remaining files are considered for compaction. This is repeated until a compaction is triggered or there are no files left to consider.
+
+The number of background threads tablet servers use to run major compactions is configurable. To configure this modify the following property:
+
+
+ tserver.compaction.major.concurrent.max
+
+
+Also, the number of threads tablet servers use for minor compactions is configurable. To configure this modify the following property:
+
+
+ tserver.compaction.minor.concurrent.max
+
+
+The numbers of minor and major compactions running and queued is visible on the Accumulo monitor page. This allows you to see if compactions are backing up and adjustments to the above settings are needed. When adjusting the number of threads available for compactions, consider the number of cores and other tasks running on the nodes such as maps and reduces.
+
+If major compactions are not keeping up, then the number of files per tablet will grow to a point such that query performance starts to suffer. One way to handle this situation is to increase the compaction ratio. For example, if the compaction ratio were set to 1, then every new file added to a tablet by minor compaction would immediately queue the tablet for major compaction. So if a tablet has a 200M file and minor compaction writes a 1M file, then the major compaction will attempt to merge the 200M and 1M file. If the tablet server has lots of tablets trying to do this sort of thing, then major compactions will back up and the number of files per tablet will start to grow, assuming data is being continuously written. Increasing the compaction ratio will alleviate backups by lowering the amount of major compaction work that needs to be done.
+
+Another option to deal with the files per tablet growing too large is to adjust the following property:
+
+
+ table.file.max
+
+
+When a tablet reaches this number of files and needs to flush its in-memory data to disk, it will choose to do a merging minor compaction. A merging minor compaction will merge the tablet's smallest file with the data in memory at minor compaction time. Therefore the number of files will not grow beyond this limit. This will make minor compactions take longer, which will cause ingest performance to decrease. This can cause ingest to slow down until major compactions have enough time to catch up. When adjusting this property, also consider adjusting the compaction ratio. Ideally, merging minor compactions never need to occur and major compactions will keep up. It is possible to configure the file max and compaction ratio such that only merging minor compactions occur and major compactions never occur. This should be avoided because doing only merging minor compactions causes ![$O(N^2)$][19] work to be done. The amount of work done by major compactions is ![$O(N*\log_R(N))$][
20] where *R* is the compaction ratio.
+
+Compactions can be initiated manually for a table. To initiate a minor compaction, use the flush command in the shell. To initiate a major compaction, use the compact command in the shell. The compact command will compact all tablets in a table to one file. Even tablets with one file are compacted. This is useful for the case where a major compaction filter is configured for a table. In 1.4 the ability to compact a range of a table was added. To use this feature specify start and stop rows for the compact command. This will only compact tablets that overlap the given row range.
## <a id=Pre-splitting_tables></a> Pre-splitting tables
@@ -422,11 +479,12 @@ The du command in the shell shows how mu
[10]: Table_Configuration.html#Constraints
[11]: Table_Configuration.html#Bloom_Filters
[12]: Table_Configuration.html#Iterators
- [13]: Table_Configuration.html#Versioning_Iterators_and_Timestamps
- [14]: Table_Configuration.html#Filters
- [15]: Table_Configuration.html#Aggregating_Iterators
- [16]: Table_Configuration.html#Pre-splitting_tables
- [17]: Table_Configuration.html#Merging_tablets
- [18]: Table_Configuration.html#Delete_Range
- [19]: Table_Configuration.html#Cloning_Tables
+ [13]: Table_Configuration.html#Block_Cache
+ [14]: Table_Configuration.html#Compaction
+ [15]: Table_Configuration.html#Pre-splitting_tables
+ [16]: Table_Configuration.html#Merging_tablets
+ [17]: Table_Configuration.html#Delete_Range
+ [18]: Table_Configuration.html#Cloning_Tables
+ [19]: img2.png
+ [20]: img3.png
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Table_Design.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Table_Design.mdtext?rev=1202418&r1=1202417&r2=1202418&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Table_Design.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/Table_Design.mdtext Tue Nov 15 20:53:02 2011
@@ -200,8 +200,8 @@ Of course, fast servers will return thei
[11]: Table_Design.html#Indexing
[12]: Table_Design.html#Entity-Attribute_and_Graph_Tables
[13]: Table_Design.html#Document-Partitioned_Indexing
- [14]: img2.png
- [15]: img3.png
- [16]: img4.png
- [17]: img5.png
+ [14]: img4.png
+ [15]: img5.png
+ [16]: img6.png
+ [17]: img7.png
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img2.png
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img2.png?rev=1202418&r1=1202417&r2=1202418&view=diff
==============================================================================
Binary files - no diff available.
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img3.png
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img3.png?rev=1202418&r1=1202417&r2=1202418&view=diff
==============================================================================
Binary files - no diff available.
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img4.png
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img4.png?rev=1202418&r1=1202417&r2=1202418&view=diff
==============================================================================
Binary files - no diff available.
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img5.png
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img5.png?rev=1202418&r1=1202417&r2=1202418&view=diff
==============================================================================
Binary files - no diff available.
Added: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img6.png
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img6.png?rev=1202418&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img6.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img7.png
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img7.png?rev=1202418&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.4-incubating/img7.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Propchange: incubator/accumulo/trunk/docs/src/user_manual/
------------------------------------------------------------------------------
--- svn:ignore (original)
+++ svn:ignore Tue Nov 15 20:53:02 2011
@@ -1 +1,2 @@
accumulo_user_manual.toc
+accumulo_user_manual.out
Modified: incubator/accumulo/trunk/docs/src/user_manual/chapters/table_configuration.tex
URL: http://svn.apache.org/viewvc/incubator/accumulo/trunk/docs/src/user_manual/chapters/table_configuration.tex?rev=1202418&r1=1202417&r2=1202418&view=diff
==============================================================================
--- incubator/accumulo/trunk/docs/src/user_manual/chapters/table_configuration.tex (original)
+++ incubator/accumulo/trunk/docs/src/user_manual/chapters/table_configuration.tex Tue Nov 15 20:53:02 2011
@@ -456,8 +456,8 @@ never need to occur and major compaction
configure the file max and compaction ratio such that only merging minor
compactions occur and major compactions never occur. This should be avoided
because doing only merging minor compactions causes $O(N^2)$ work to be done.
-The amount of work done by major compactions is $O(N*\log_R(N))$ where $R$ is
-the compaction ratio.
+The amount of work done by major compactions is $O(N*\log_R(N))$ where
+\textit{R} is the compaction ratio.
Compactions can be initiated manually for a table. To initiate a minor
compaction, use the flush command in the shell. To initiate a major compaction,