You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by bu...@apache.org on 2012/03/23 20:08:14 UTC
svn commit: r809758 [8/9] - in /websites/staging/accumulo/trunk/content: ./
accumulo/ accumulo/1.4/ accumulo/1.4/examples/ accumulo/1.4/user_manual/
accumulo/downloads/ accumulo/downloads/1.3.5/ accumulo/example/
accumulo/governance/ accumulo/user_manu...
Modified: websites/staging/accumulo/trunk/content/accumulo/governance/voting.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/governance/voting.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/governance/voting.html Fri Mar 23 19:08:12 2012
@@ -63,22 +63,14 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - klzzwxh:0006 -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
-<h1 id="asf_links">ASF links</h1>
+<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
@@ -95,19 +87,19 @@
<p>Occasionally a "feel" for consensus is not enough. Sometimes we need to have a
measurable consensus. For example, when voting in new committers or to approve a
release. </p>
-<h2 id="preparing_for_a_vote">Preparing for a Vote</h2>
+<h2 id="preparing-for-a-vote">Preparing for a Vote</h2>
<p>Before calling a vote it is important to ensure that the community is given time to
discuss the upcoming vote. This will be done by posting an email to the list
indicating the intention to call a vote and the options available. By the time a
vote is called there should already be <a href="http://incubator.apache.org/accumulo/governance/consensusBuilding.html">consensus in the community</a>. The vote
itself is, normally, a formality.</p>
-<h2 id="calling_a_vote">Calling a Vote</h2>
+<h2 id="calling-a-vote">Calling a Vote</h2>
<p>Once it is time to call the vote a mail is posted with a subject starting with
"[VOTE]". This enables the community members to ensure they do not miss an important
vote thread. It also indicates that this is not consensus building but a formal
vote. The initiator is responsible for the vote. That means also to count the votes
and present the results. Everyone has 1 vote.</p>
-<h3 id="casting_your_vote">Casting Your Vote</h3>
+<h3 id="casting-your-vote">Casting Your Vote</h3>
<p>The notation used in voting is:</p>
<p>+1 (means I vote positive)
You can say why you vote positive but it's not a must-have.</p>
@@ -115,7 +107,7 @@ and present the results. Everyone has 1
<p>-1 (means I vote negative because of the following reason)
Yes, you must support your objection and provide an alternative course of action
that you are willing and able to implement (where appropriate).</p>
-<h4 id="example_for_a_vote_mail">Example for a vote mail:</h4>
+<h4 id="example-for-a-vote-mail">Example for a vote mail:</h4>
<div class="codehilite"><pre><span class="err">Address:</span> <span class="kd">private</span><span class="err">@</span>
<span class="err">Subject:</span> <span class="err">[VOTE]</span> <span class="err">John</span> <span class="err">Doe</span> <span class="err">should</span> <span class="err">become</span> <span class="err">a</span> <span class="err">regular</span> <span class="err">committer</span>
@@ -132,7 +124,7 @@ and present the results. Everyone has 1
</pre></div>
-<h4 id="example_for_a_reply_mail">Example for a reply mail:</h4>
+<h4 id="example-for-a-reply-mail">Example for a reply mail:</h4>
<div class="codehilite"><pre><span class="n">Text:</span>
<span class="o">+</span><span class="mi">1</span>
@@ -140,7 +132,7 @@ and present the results. Everyone has 1
</pre></div>
-<h4 id="example_for_a_result_mail">Example for a result mail:</h4>
+<h4 id="example-for-a-result-mail">Example for a result mail:</h4>
<div class="codehilite"><pre><span class="n">Subject:</span> <span class="p">[</span><span class="n">VOTE</span><span class="p">][</span><span class="n">RESULTS</span><span class="p">]</span> <span class="n">John</span> <span class="n">Doe</span> <span class="n">should</span> <span class="n">become</span> <span class="n">a</span> <span class="n">regular</span> <span class="n">committer</span>
<span class="n">Text:</span>
Modified: websites/staging/accumulo/trunk/content/accumulo/index.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/index.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/index.html Fri Mar 23 19:08:12 2012
@@ -63,22 +63,14 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - klzzwxh:0006 -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
-<h1 id="asf_links">ASF links</h1>
+<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
Modified: websites/staging/accumulo/trunk/content/accumulo/mailing_list.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/mailing_list.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/mailing_list.html Fri Mar 23 19:08:12 2012
@@ -63,22 +63,14 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - klzzwxh:0006 -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
-<h1 id="asf_links">ASF links</h1>
+<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
Modified: websites/staging/accumulo/trunk/content/accumulo/notable_features.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/notable_features.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/notable_features.html Fri Mar 23 19:08:12 2012
@@ -63,22 +63,14 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - klzzwxh:0006 -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
-<h1 id="asf_links">ASF links</h1>
+<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
@@ -105,12 +97,12 @@
<li><a href="#ondemand_dm">On-demand Data Management</a></li>
</ul>
<hr />
-<h2 id="table_design_and_configuration_a_iddesigna">Table Design and Configuration <a id="design"></a></h2>
+<h2 id="table-design-and-configuration-wzxhzdk0wzxhzdk1">Table Design and Configuration <a id="design"></a></h2>
<h3 id="iterators">Iterators</h3>
<p>A server-side programming mechanism to encode functions such as filtering and
aggregation within the data management steps (scopes where data is read from or
written to disk) that happen in the tablet server.</p>
-<h3 id="cell_labels">Cell labels</h3>
+<h3 id="cell-labels">Cell labels</h3>
<p>An additional portion of the Key that sorts after the column qualifier and
before the timestamp. It is called column visibility and enables expressive
cell-level access control. Authorizations are passed with each query to control
@@ -124,21 +116,21 @@ Constraints are written in Java and conf
<p>Through the use of specialized iterators, Accumulo can be a parallel sharded
document store. For example wikipedia could be stored and searched for
documents containing certain words.</p>
-<h3 id="large_rows">Large Rows</h3>
+<h3 id="large-rows">Large Rows</h3>
<p>When reading rows, there is no requirement that an entire row fits into memory.</p>
-<h2 id="integrityavailability_a_idintegritya">Integrity/Availability <a id="integrity"></a></h2>
-<h3 id="master_fail_over">Master fail over</h3>
+<h2 id="integrityavailability-wzxhzdk2wzxhzdk3">Integrity/Availability <a id="integrity"></a></h2>
+<h3 id="master-fail-over">Master fail over</h3>
<p>Multiple masters can be configured. Zookeeper locks are used to determine
which master is active. The remaining masters simply wait for the current
master to lose its lock. Current master state is held in the metadata table
and Zookeeper (see <a href="#fate">FATE</a>).</p>
-<h3 id="write_ahead_log">Write ahead log</h3>
+<h3 id="write-ahead-log">Write ahead log</h3>
<p>Tablet servers send mutations to loggers, not HDFS. Each mutation is written
to multiple loggers, before being committed. In normal configurations, loggers
are run on every server that runs a tablet server. When a log is needed for
recovery, it is copied into HDFS. For efficient recovery, the log is sorted as
it is copied.</p>
-<h3 id="logical_time">Logical time</h3>
+<h3 id="logical-time">Logical time</h3>
<p>A mechanism to ensure that server set times never go backwards, even when time
across the cluster is incorrect. This ensures that updates and deletes are not
lost. If a tablet is served on machine with time a year in the future, then the
@@ -150,7 +142,7 @@ and sends this back to the tablet server
to adjust the timestamps they issue. So logical time ensures ordering is
always correct and master authoritative time tries to ensure that timestamps
are meaningful.</p>
-<h3 id="logical_time_for_bulk_import_14">Logical Time for bulk import (1.4)</h3>
+<h3 id="logical-time-for-bulk-import-14">Logical Time for bulk import (1.4)</h3>
<p>Logical time as described above works with streaming (batch) ingest, where the
tablet server assigns the timestamp. Logical time is also important for bulk
imported data, for which the client code may be choosing a timestamp. Accumulo
@@ -162,7 +154,7 @@ system that created the file for bulk im
file is imported, but whenever it is read by scans or compactions. At import, a
time is obtained and always used by the specialized system iterator to set that
time.</p>
-<h3 id="fate_14">FATE (1.4)</h3>
+<h3 id="fate-14">FATE (1.4)</h3>
<p>Fault Tolerant Executor. A framework for executing operations in a fault
tolerant manner. In the previous release, if the master process died in the
middle of creating a table it could leave the system in an inconsistent state.
@@ -173,23 +165,23 @@ before attempting to do the work. Client
with work, and then wait for it to finish. Most table operations are executed
using this framework. Persistent, per table, read-write locks are created in
Zookeeper to synchronize operations across process faults.</p>
-<h3 id="scalable_master">Scalable master</h3>
+<h3 id="scalable-master">Scalable master</h3>
<p>Stores its metadata in an Accumulo table and Zookeeper.</p>
<h3 id="isolation">Isolation</h3>
<p>Scans will not see data inserted into a row after the scan of that row begins.</p>
-<h2 id="performance_a_idperformancea">Performance <a id="performance"></a></h2>
-<h3 id="relative_encoding">Relative encoding</h3>
+<h2 id="performance-wzxhzdk4wzxhzdk5">Performance <a id="performance"></a></h2>
+<h3 id="relative-encoding">Relative encoding</h3>
<p>If consecutive keys have identical portions (row, colf, colq, or colvis), there
is a flag to indicate that a portion is the same as that of the previous key.
This is applied when keys are stored on disk and when transferred over the
network.</p>
-<h3 id="native_in-memory_map">Native In-Memory Map</h3>
+<h3 id="native-in-memory-map">Native In-Memory Map</h3>
<p>By default data written is stored outside of Java managed memory into a C++ STL
map of maps. It maps rows to columns to values. This hierarchical structure
improves performance of inserting a mutation with multiple column values in a
single row. A custom STL allocator is used to avoid the global malloc lock and
memory fragmentation.</p>
-<h3 id="scan_pipeline">Scan pipeline</h3>
+<h3 id="scan-pipeline">Scan pipeline</h3>
<p>A long running Accumulo scan will eventually cause multiple threads to start.
One server thread to read data from disk, one server thread to serialize and
send data, and one client thread to deserialize and read data. When pipelining
@@ -198,7 +190,7 @@ does not activate for short scans.</p>
<h3 id="caching">Caching</h3>
<p>Recently scanned data is cached into memory. There are separate caches for
indexes and data. Caching can be turned on and off for individual tables.</p>
-<h3 id="multi-level_rfile_index_14">Multi-level RFile Index (1.4)</h3>
+<h3 id="multi-level-rfile-index-14">Multi-level RFile Index (1.4)</h3>
<p>RFiles store an index of the last key in each block. For large files the index
can become quite large. When the index is large a lot of memory is consumed and
files take a long time to open. To avoid this problem, RFiles in 1.4 have a
@@ -207,17 +199,17 @@ blocks. The entire index never has to be
written. When an index block exceeds the configurable size threshold, its
written out between data blocks. The size of index blocks is configurable on a
per table basis.</p>
-<h2 id="testing_a_idtestinga">Testing <a id="testing"></a></h2>
+<h2 id="testing-wzxhzdk6wzxhzdk7">Testing <a id="testing"></a></h2>
<h3 id="mock">Mock</h3>
<p>The Accumulo client API has a mock implementation that is useful writing unit
test against Accumulo. Mock Accumulo is in memory and in process.</p>
-<h3 id="functional_test">Functional Test</h3>
+<h3 id="functional-test">Functional Test</h3>
<p>Small, system-level tests of basic Accumulo features run in a test harness,
external to the build and unit-tests. These tests start a complete Accumulo
instance, and require Hadoop and Zookeeper to be running. They attempt to
simulate the basic functions of Accumulo, as well as common failure conditions,
such as lost disks, killed processes, and read-only file systems.</p>
-<h3 id="scale_test">Scale Test</h3>
+<h3 id="scale-test">Scale Test</h3>
<p>A test suite that verifies data is not lost at scale. This test runs many
ingest clients that continually create linked lists containing 25 million
nodes. At some point the clients are stopped and a map reduce job is run to
@@ -226,7 +218,7 @@ The Agitator can be run in conjunction w
servers. This test suite has uncovered many obscure data loss bugs. This test
also helps find bugs that impact uptime and stability when run for days or
weeks.</p>
-<h3 id="random_walk_test">Random Walk Test</h3>
+<h3 id="random-walk-test">Random Walk Test</h3>
<p>A test suite that looks for unexpected system states that may emerge in
plausible real-world applications. Application components are defined as test
nodes (such as create table, insert data, scan data, delete table, etc.), and
@@ -237,14 +229,14 @@ of the test graphs have a concept of cor
Other tests have no concept of data correctness and have the simple goal of
crashing Accumulo. Many obscure bugs have been uncovered by this testing
framework and subsequently corrected.</p>
-<h2 id="client_api_a_idclienta">Client API <a id="client"></a></h2>
-<h3 id="batch_scanner4"><a href="/accumulo/user_manual_1.3-incubating/Writing_Accumulo_Clients.html">Batch Scanner</a></h3>
+<h2 id="client-api-wzxhzdk8wzxhzdk9">Client API <a id="client"></a></h2>
+<h3 id="batch-scanner"><a href="/accumulo/1.4/user_manual/Writing_Accumulo_Clients.html">Batch Scanner</a></h3>
<p>Takes a list of Ranges, batches them to the appropriate tablet servers, and
returns data as it is received (i.e. not in sorted order).</p>
-<h3 id="batch_writer4"><a href="/accumulo/user_manual_1.3-incubating/Writing_Accumulo_Clients.html">Batch Writer</a></h3>
+<h3 id="batch-writer"><a href="/accumulo/1.4/user_manual/Writing_Accumulo_Clients.html">Batch Writer</a></h3>
<p>Clients buffer writes in memory before sending them in batches to the
appropriate tablet servers.</p>
-<h3 id="bulk_import6"><a href="/accumulo/user_manual_1.3-incubating/High_Speed_Ingest.html#Bulk_Ingest">Bulk Import</a></h3>
+<h3 id="bulk-import"><a href="/accumulo/1.4/user_manual/High_Speed_Ingest.html#Bulk_Ingest">Bulk Import</a></h3>
<p>Instead of writing individual mutations to Accumulo, entire files of sorted key
value pairs can be imported. These files are moved into the Accumulo directory
and referenced by Accumulo. This feature is useful for ingesting a large amount
@@ -253,10 +245,10 @@ higher latency for data availability for
using map reduce and then bulk imported. This method of ingest also allows for
flexibility in resource allocation. The nodes running map reduce to sort data
could be different from the Accumulo nodes.</p>
-<h3 id="map_reduce4"><a href="/accumulo/user_manual_1.3-incubating/Writing_Accumulo_Clients.html">Map Reduce</a></h3>
+<h3 id="map-reduce"><a href="/accumulo/1.4/user_manual/Writing_Accumulo_Clients.html">Map Reduce</a></h3>
<p>Accumulo can be a source and/or sink for map reduce jobs.</p>
-<h2 id="extensible_behaviors_a_idbehaviorsa">Extensible Behaviors <a id="behaviors"></a></h2>
-<h3 id="pluggable_balancer">Pluggable balancer</h3>
+<h2 id="extensible-behaviors-wzxhzdk10wzxhzdk11">Extensible Behaviors <a id="behaviors"></a></h2>
+<h3 id="pluggable-balancer">Pluggable balancer</h3>
<p>Users can provide a balancer plugin that decides how to distribute tablets
across a table. These plugins can be provided on a per table basis. This is
useful for ensuring a particular table's tablets are placed optimally for
@@ -266,21 +258,21 @@ previously hosted to leverage locality.
balancer moves one child to another tablet server. The assumption here is that
splitting tablets are being actively written to, so this keeps write load evenly
spread.</p>
-<h3 id="pluggable_memory_manager">Pluggable memory manager</h3>
+<h3 id="pluggable-memory-manager">Pluggable memory manager</h3>
<p>The plugin that decides when and what tablets to minor compact is configurable.
The default plugin compacts the largest tablet when memory is over a certain
threshold. It varies the threshold over time depending on minor compaction
speed. It flushes tablets that are not written to for a configurable time
period.</p>
-<h3 id="pluggable_logger_assignment_strategy">Pluggable logger assignment strategy</h3>
+<h3 id="pluggable-logger-assignment-strategy">Pluggable logger assignment strategy</h3>
<p>The plugin that decided which loggers should be assigned to which tablet
servers is configurable.</p>
-<h2 id="general_administration_a_idadmina">General Administration <a id="admin"></a></h2>
-<h3 id="monitor_page">Monitor page</h3>
+<h2 id="general-administration-wzxhzdk12wzxhzdk13">General Administration <a id="admin"></a></h2>
+<h3 id="monitor-page">Monitor page</h3>
<p>A simple web server provides basic information about the system health and
performance. It displays table sizes, ingest and query statistics, server
load, and last-update information. It also allows the user to view recent
-diagnostic logs and <a href="#tracing5">traces</a>.</p>
+diagnostic logs and traces.</p>
<h3 id="tracing">Tracing</h3>
<p>It can be difficult to determine why some operations are taking longer than
expected. For example, you may be looking up items with very low latency, but
@@ -290,15 +282,15 @@ Accumulo has been instrumented to record
when tracing is turned on. The fact that tracing is enabled follows all the
requests made on behalf of the user throughout the distributed infrastructure
of Accumulo, and across all threads of execution.</p>
-<h3 id="online_reconfiguration">Online reconfiguration</h3>
+<h3 id="online-reconfiguration">Online reconfiguration</h3>
<p>System and per table configuration is stored in Zookeeper. Many, but not all,
configuration changes take effect while Accumulo is running. Some do not take
effect until server processes are restarted.</p>
-<h3 id="table_renaming">Table renaming</h3>
+<h3 id="table-renaming">Table renaming</h3>
<p>Tables can be renamed easily because Accumulo uses internal table IDs and
stores mappings between names and IDs in Zookeeper.</p>
-<h2 id="internal_data_management_a_idinternal_dma">Internal Data Management <a id="internal_dm"></a></h2>
-<h3 id="locality_groups">Locality groups</h3>
+<h2 id="internal-data-management-wzxhzdk14wzxhzdk15">Internal Data Management <a id="internal_dm"></a></h2>
+<h3 id="locality-groups">Locality groups</h3>
<p>Groups columns within a single file. There is a default locality group so that
not all columns need be specified. The locality groups can be restructured
while the table is online and the changes will take effect on the next
@@ -308,13 +300,13 @@ compactions rewrite all files. After re
force a table to compact in order to write all data into the new locality
groups. Alternatively, the change could be allowed to happen over time as
writes to the table cause compactions to happen.</p>
-<h3 id="smart_compaction_algorithm">Smart compaction algorithm</h3>
+<h3 id="smart-compaction-algorithm">Smart compaction algorithm</h3>
<p>It is inefficient to merge small files with large files. Accumulo merges files
only if all files are larger than a configurable ratio (default is 3)
multiplied by the largest file size. If this cannot be done with all the
files, the largest file is removed from consideration, and the remaining files
are considered for compaction. This is done until there are no files to merge.</p>
-<h3 id="merging_minor_compaction_14">Merging Minor Compaction (1.4)</h3>
+<h3 id="merging-minor-compaction-14">Merging Minor Compaction (1.4)</h3>
<p>When a max number of files per tablet is reached, minor compactions will merge
data from the in-memory map with the smallest file instead of creating new
files. This throttles ingest. In previous releases new files were just created
@@ -322,30 +314,30 @@ even if major compactions were falling b
was growing. Without this feature, ingest performance can roughly continue at a
constant rate, even as scan performance decreases because tablets have too many
files.</p>
-<h2 id="on-demand_data_management_a_idondemand_dma">On-demand Data Management <a id="ondemand_dm"></a></h2>
+<h2 id="on-demand-data-management-wzxhzdk16wzxhzdk17">On-demand Data Management <a id="ondemand_dm"></a></h2>
<h3 id="compactions">Compactions</h3>
<p>Ability to force tablets to compact to one file. Even tablets with one file are
compacted. This is useful for improving query performance, permanently
applying iterators, or using a new locality group configuration. One example
of using iterators is applying a filtering iterator to remove data from a
table. </p>
-<h3 id="split_points">Split points</h3>
+<h3 id="split-points">Split points</h3>
<p>Arbitrary split points can be added to an online table at any point in time.
This is useful for increasing ingest performance on a new table. It can also be
used to accommodate new data patterns in an existing table.</p>
-<h3 id="tablet_merging_14">Tablet Merging (1.4)</h3>
+<h3 id="tablet-merging-14">Tablet Merging (1.4)</h3>
<p>Tablet merging is a new feature. Merging of tablets can be requested in the
shell; Accumulo does not merge tablets automatically.</p>
-<h3 id="table_cloning_14">Table Cloning (1.4)</h3>
+<h3 id="table-cloning-14">Table Cloning (1.4)</h3>
<p>Allows users to quickly create a new table that references an existing table's
data and copies its configuration. A cloned table and its source table can be
mutated independently. Testing was the motivating reason behind this new
feature. For example to test a new filtering iterator, clone the table, add the
filter to the clone, and force a major compaction.</p>
-<h3 id="compact_range_14">Compact Range (1.4)</h3>
+<h3 id="compact-range-14">Compact Range (1.4)</h3>
<p>Compact each tablet that falls within a row range down to a single file.<br />
</p>
-<h3 id="delete_range_14">Delete Range (1.4)</h3>
+<h3 id="delete-range-14">Delete Range (1.4)</h3>
<p>Added an operation to efficiently delete a range of rows from a table. Tablets
that fall completely within a range are simply dropped. Tablets overlapping the
beginning and end of the range are split, compacted, and then merged.<br />
Modified: websites/staging/accumulo/trunk/content/accumulo/papers.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/papers.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/papers.html Fri Mar 23 19:08:12 2012
@@ -63,21 +63,13 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - <a href="/accumulo/getting_started.html">Getting Started</a> -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
Modified: websites/staging/accumulo/trunk/content/accumulo/screenshots.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/screenshots.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/screenshots.html Fri Mar 23 19:08:12 2012
@@ -63,22 +63,14 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - klzzwxh:0006 -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
-<h1 id="asf_links">ASF links</h1>
+<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
@@ -93,11 +85,9 @@
<div id="content">
<h1 class="title">Apache Accumulo Screenshots</h1>
<p>The following is a screenshot of the Accumulo monitor overview web page. Accumulo was running on an eleven node cluster. The continuous ingest test suite was running to generate load. Ten continuous ingest clients were running.</p>
-<img src=/accumulo/images/overview2.png />
-
+<p><img src=/accumulo/images/overview2.png /></p>
<p>The following screenshot shows the monitor master page. This page gives information about individual tables within accumulo. Continuous ingest is running against the table ci.</p>
-<img src=/accumulo/images/master2.png />
-
+<p><img src=/accumulo/images/master2.png /></p>
<p>The data generated by the continuous ingest test suite looks like the following. The data is random, but forms many giant linked list. Each ingest client continually generates linked list containing 25 million nodes. In the shell session below a few scans are issued to start following the linked list that the first node in the table is part of.</p>
<div class="codehilite"><pre><span class="n">root</span><span class="nv">@test15</span> <span class="n">ci</span><span class="o">></span> <span class="n">scan</span>
<span class="mo">000000000135</span><span class="n">fbbe</span> <span class="mo">074</span><span class="n">c:569c</span> <span class="o">[]</span> <span class="mi">2</span><span class="n">db1de18</span><span class="o">-</span><span class="n">cd37</span><span class="o">-</span><span class="mi">407</span><span class="n">d</span><span class="o">-</span><span class="n">b060</span><span class="o">-</span><span class="mi">481</span><span class="n">a0a214c90:000000002b2bcfe1:2e15cb9f62fd22ab:fe6f84c7</span>
Modified: websites/staging/accumulo/trunk/content/accumulo/source.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/source.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/source.html Fri Mar 23 19:08:12 2012
@@ -63,21 +63,13 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - <a href="/accumulo/getting_started.html">Getting Started</a> -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
Modified: websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Design.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Design.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Design.html Fri Mar 23 19:08:12 2012
@@ -63,22 +63,14 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - klzzwxh:0006 -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
-<h1 id="asf_links">ASF links</h1>
+<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
@@ -105,35 +97,35 @@
<li><a href="Accumulo_Design.html#Fault-Tolerance">Fault-Tolerance</a></li>
</ul>
<hr />
-<h2 id="a_idaccumulo_designa_accumulo_design"><a id=Accumulo_Design></a> Accumulo Design</h2>
-<h2 id="a_iddata_modela_data_model"><a id=Data_Model></a> Data Model</h2>
+<h2 id="wzxhzdk2wzxhzdk3-accumulo-design"><a id=Accumulo_Design></a> Accumulo Design</h2>
+<h2 id="wzxhzdk4wzxhzdk5-data-model"><a id=Data_Model></a> Data Model</h2>
<p>Accumulo provides a richer data model than simple key-value stores, but is not a fully relational database. Data is represented as key-value pairs, where the key and value are comprised of the following elements: </p>
<p><img alt="converted table" src="img1.png" /></p>
<p>All elements of the Key and the Value are represented as byte arrays except for Timestamp, which is a Long. Accumulo sorts keys by element and lexicographically in ascending order. Timestamps are sorted in descending order so that later versions of the same Key appear first in a sequential scan. Tables consist of a set of sorted key-value pairs. </p>
-<h2 id="a_idarchitecturea_architecture"><a id=Architecture></a> Architecture</h2>
+<h2 id="wzxhzdk6wzxhzdk7-architecture"><a id=Architecture></a> Architecture</h2>
<p>Accumulo is a distributed data storage and retrieval system and as such consists of several architectural components, some of which run on many individual servers. Much of the work Accumulo does involves maintaining certain properties of the data, such as organization, availability, and integrity, across many commodity-class machines. </p>
-<h2 id="a_idcomponentsa_components"><a id=Components></a> Components</h2>
+<h2 id="wzxhzdk8wzxhzdk9-components"><a id=Components></a> Components</h2>
<p>An instance of Accumulo includes many TabletServers, write-ahead Logger servers, one Garbage Collector process, one Master server and many Clients. </p>
-<h3 id="a_idtablet_servera_tablet_server"><a id=Tablet_Server></a> Tablet Server</h3>
+<h3 id="wzxhzdk10wzxhzdk11-tablet-server"><a id=Tablet_Server></a> Tablet Server</h3>
<p>The TabletServer manages some subset of all the tablets (partitions of tables). This includes receiving writes from clients, persisting writes to a writeâahead log, sorting new keyâvalue pairs in memory, periodically flushing sorted keyâvalue pairs to new files in HDFS, and responding to reads from clients, forming a mergeâsorted view of all keys and values from all the files it has created and the sorted inâmemory store. </p>
<p>TabletServers also perform recovery of a tablet that was previously on a server that failed, reapplying any writes found in the write-ahead log to the tablet. </p>
-<h3 id="a_idloggersa_loggers"><a id=Loggers></a> Loggers</h3>
+<h3 id="wzxhzdk12wzxhzdk13-loggers"><a id=Loggers></a> Loggers</h3>
<p>The Loggers accept updates to Tablet servers and write them to local on-disk storage. Each tablet server will write their updates to multiple loggers to preserve data in case of hardware failure. </p>
-<h3 id="a_idgarbage_collectora_garbage_collector"><a id=Garbage_Collector></a> Garbage Collector</h3>
+<h3 id="wzxhzdk14wzxhzdk15-garbage-collector"><a id=Garbage_Collector></a> Garbage Collector</h3>
<p>Accumulo processes will share files stored in HDFS. Periodically, the Garbage Collector will identify files that are no longer needed by any process, and delete them. </p>
-<h3 id="a_idmastera_master"><a id=Master></a> Master</h3>
+<h3 id="wzxhzdk16wzxhzdk17-master"><a id=Master></a> Master</h3>
<p>The Accumulo Master is responsible for detecting and responding to TabletServer failure. It tries to balance the load across TabletServer by assigning tablets carefully and instructing TabletServers to migrate tablets when necessary. The Master ensures all tablets are assigned to one TabletServer each, and handles table creation, alteration, and deletion requests from clients. The Master also coordinates startup, graceful shutdown and recovery of changes in write-ahead logs when Tablet servers fail. </p>
-<h3 id="a_idclienta_client"><a id=Client></a> Client</h3>
+<h3 id="wzxhzdk18wzxhzdk19-client"><a id=Client></a> Client</h3>
<p>Accumulo includes a client library that is linked to every application. The client library contains logic for finding servers managing a particular tablet, and communicating with TabletServers to write and retrieve key-value pairs. </p>
-<h2 id="a_iddata_managementa_data_management"><a id=Data_Management></a> Data Management</h2>
+<h2 id="wzxhzdk20wzxhzdk21-data-management"><a id=Data_Management></a> Data Management</h2>
<p>Accumulo stores data in tables, which are partitioned into tablets. Tablets are partitioned on row boundaries so that all of the columns and values for a particular row are found together within the same tablet. The Master assigns Tablets to one TabletServer at a time. This enables row-level transactions to take place without using distributed locking or some other complicated synchronization mechanism. As clients insert and query data, and as machines are added and removed from the cluster, the Master migrates tablets to ensure they remain available and that the ingest and query load is balanced across the cluster. </p>
<p><img alt="Image data_distribution" src="./data_distribution.png" /></p>
-<h2 id="a_idtablet_servicea_tablet_service"><a id=Tablet_Service></a> Tablet Service</h2>
+<h2 id="wzxhzdk22wzxhzdk23-tablet-service"><a id=Tablet_Service></a> Tablet Service</h2>
<p>When a write arrives at a TabletServer it is written to a WriteâAhead Log and then inserted into a sorted data structure in memory called a MemTable. When the MemTable reaches a certain size the TabletServer writes out the sorted key-value pairs to a file in HDFS called Indexed Sequential Access Method (ISAM) file. This process is called a minor compaction. A new MemTable is then created and the fact of the compaction is recorded in the WriteâAhead Log. </p>
<p>When a request to read data arrives at a TabletServer, the TabletServer does a binary search across the MemTable as well as the in-memory indexes associated with each ISAM file to find the relevant values. If clients are performing a scan, several keyâvalue pairs are returned to the client in order from the MemTable and the set of ISAM files by performing a mergeâsort as they are read. </p>
-<h2 id="a_idcompactionsa_compactions"><a id=Compactions></a> Compactions</h2>
+<h2 id="wzxhzdk24wzxhzdk25-compactions"><a id=Compactions></a> Compactions</h2>
<p>In order to manage the number of files per tablet, periodically the TabletServer performs Major Compactions of files within a tablet, in which some set of ISAM files are combined into one file. The previous files will eventually be removed by the Garbage Collector. This also provides an opportunity to permanently remove deleted keyâvalue pairs by omitting keyâvalue pairs suppressed by a delete entry when the new file is created. </p>
-<h2 id="a_idfault-tolerancea_fault-tolerance"><a id=Fault-Tolerance></a> Fault-Tolerance</h2>
+<h2 id="wzxhzdk26wzxhzdk27-fault-tolerance"><a id=Fault-Tolerance></a> Fault-Tolerance</h2>
<p>If a TabletServer fails, the Master detects it and automatically reassigns the tablets assigned from the failed server to other servers. Any key-value pairs that were in memory at the time the TabletServer are automatically reapplied from the Write-Ahead Log to prevent any loss of data. </p>
<p>The Master will coordinate the copying of write-ahead logs to HDFS so the logs are available to all tablet servers. To make recovery efficient, the updates within a log are grouped by tablet. The sorting process can be performed by Hadoops MapReduce or the Logger server. TabletServers can quickly apply the mutations from the sorted logs that are destined for the tablets they have now been assigned. </p>
<p>TabletServer failures are noted on the Master's monitor page, accessible via <br />
Modified: websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Shell.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Shell.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Accumulo_Shell.html Fri Mar 23 19:08:12 2012
@@ -63,22 +63,14 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - klzzwxh:0006 -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
-<h1 id="asf_links">ASF links</h1>
+<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
@@ -101,7 +93,7 @@
<li><a href="Accumulo_Shell.html#User_Administration">User Administration</a></li>
</ul>
<hr />
-<h2 id="a_idaccumulo_shella_accumulo_shell"><a id=Accumulo_Shell></a> Accumulo Shell</h2>
+<h2 id="wzxhzdk9wzxhzdk10-accumulo-shell"><a id=Accumulo_Shell></a> Accumulo Shell</h2>
<p>Accumulo provides a simple shell that can be used to examine the contents and configuration settings of tables, apply individual mutations, and change configuration settings. </p>
<p>The shell can be started by the following command: </p>
<div class="codehilite"><pre><span class="nv">$ACCUMULO_HOME</span><span class="sr">/bin/</span><span class="n">accumulo</span> <span class="n">shell</span> <span class="o">-</span><span class="n">u</span> <span class="p">[</span><span class="n">username</span><span class="p">]</span>
@@ -120,7 +112,7 @@
</pre></div>
-<h2 id="a_idbasic_administrationa_basic_administration"><a id=Basic_Administration></a> Basic Administration</h2>
+<h2 id="wzxhzdk11wzxhzdk12-basic-administration"><a id=Basic_Administration></a> Basic Administration</h2>
<p>The Accumulo shell can be used to create and delete tables, as well as to configure table and instance specific options. </p>
<div class="codehilite"><pre><span class="n">root</span><span class="nv">@myinstance</span><span class="o">></span> <span class="n">tables</span>
<span class="o">!</span><span class="n">METADATA</span>
@@ -154,7 +146,7 @@
</pre></div>
-<h2 id="a_idtable_maintenancea_table_maintenance"><a id=Table_Maintenance></a> Table Maintenance</h2>
+<h2 id="wzxhzdk13wzxhzdk14-table-maintenance"><a id=Table_Maintenance></a> Table Maintenance</h2>
<p>The <strong>compact</strong> command instructs Accumulo to schedule a compaction of the table during which files are consolidated and deleted entries are removed. </p>
<div class="codehilite"><pre><span class="n">root</span><span class="nv">@myinstance</span> <span class="n">mytable</span><span class="o">></span> <span class="n">compact</span> <span class="o">-</span><span class="n">t</span> <span class="n">mytable</span>
<span class="mo">07</span> <span class="mi">16</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mi">53</span><span class="p">,</span><span class="mi">201</span> <span class="p">[</span><span class="n">shell</span><span class="o">.</span><span class="n">Shell</span><span class="p">]</span> <span class="n">INFO</span> <span class="p">:</span> <span class="n">Compaction</span> <span class="n">of</span> <span class="n">table</span> <span class="n">mytable</span>
@@ -169,7 +161,7 @@
</pre></div>
-<h2 id="a_iduser_administrationa_user_administration"><a id=User_Administration></a> User Administration</h2>
+<h2 id="wzxhzdk15wzxhzdk16-user-administration"><a id=User_Administration></a> User Administration</h2>
<p>The Shell can be used to add, remove, and grant privileges to users. </p>
<div class="codehilite"><pre><span class="n">root</span><span class="nv">@myinstance</span> <span class="n">mytable</span><span class="o">></span> <span class="n">createuser</span> <span class="n">bob</span>
<span class="n">Enter</span> <span class="k">new</span> <span class="n">password</span> <span class="k">for</span> <span class="s">'bob'</span><span class="p">:</span> <span class="o">*********</span>
Modified: websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Administration.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Administration.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Administration.html Fri Mar 23 19:08:12 2012
@@ -63,22 +63,14 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - klzzwxh:0006 -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
-<h1 id="asf_links">ASF links</h1>
+<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
@@ -108,25 +100,25 @@
<li><a href="Administration.html#Recovery">Recovery</a></li>
</ul>
<hr />
-<h2 id="a_idadministrationa_administration"><a id=Administration></a> Administration</h2>
-<h2 id="a_idhardwarea_hardware"><a id=Hardware></a> Hardware</h2>
+<h2 id="wzxhzdk4wzxhzdk5-administration"><a id=Administration></a> Administration</h2>
+<h2 id="wzxhzdk6wzxhzdk7-hardware"><a id=Hardware></a> Hardware</h2>
<p>Because we are running essentially two or three systems simultaneously layered across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have at least one core and 2 - 4 GB each. </p>
<p>One core running HDFS can typically keep 2 to 4 disks busy, so each machine may typically have as little as 2 x 300GB disks and as much as 4 x 1TB or 2TB disks. </p>
<p>It is possible to do with less than this, such as with 1u servers with 2 cores and 4GB each, but in this case it is recommended to only run up to two processes per machine - i.e. DataNode and TabletServer or DataNode and MapReduce worker but not all three. The constraint here is having enough available heap space for all the processes on a machine. </p>
-<h2 id="a_idnetworka_network"><a id=Network></a> Network</h2>
+<h2 id="wzxhzdk8wzxhzdk9-network"><a id=Network></a> Network</h2>
<p>Accumulo communicates via remote procedure calls over TCP/IP for both passing data and control messages. In addition, Accumulo uses HDFS clients to communicate with HDFS. To achieve good ingest and query performance, sufficient network bandwidth must be available between any two machines. </p>
-<h2 id="a_idinstallationa_installation"><a id=Installation></a> Installation</h2>
+<h2 id="wzxhzdk10wzxhzdk11-installation"><a id=Installation></a> Installation</h2>
<p>Choose a directory for the Accumulo installation. This directory will be referenced by the environment variable $ACCUMULO_HOME. Run the following: </p>
<div class="codehilite"><pre><span class="nv">$</span> <span class="nv">tar</span> <span class="n">xzf</span> <span class="nv">$ACCUMULO_HOME</span><span class="o">/</span><span class="n">accumulo</span><span class="o">.</span><span class="n">tar</span><span class="o">.</span><span class="n">gz</span>
</pre></div>
<p>Repeat this step at each machine within the cluster. Usually all machines have the same $ACCUMULO_HOME. </p>
-<h2 id="a_iddependenciesa_dependencies"><a id=Dependencies></a> Dependencies</h2>
+<h2 id="wzxhzdk12wzxhzdk13-dependencies"><a id=Dependencies></a> Dependencies</h2>
<p>Accumulo requires HDFS and ZooKeeper to be configured and running before starting. Password-less SSH should be configured between at least the Accumulo master and TabletServer machines. It is also a good idea to run Network Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of sync, which can cause problems with automatically timestamped data. Accumulo will remove from the set of TabletServers those machines whose times differ too much from the master's. </p>
-<h2 id="a_idconfigurationa_configuration"><a id=Configuration></a> Configuration</h2>
+<h2 id="wzxhzdk14wzxhzdk15-configuration"><a id=Configuration></a> Configuration</h2>
<p>Accumulo is configured by editing several Shell and XML files found in $ACCUMULO_HOME/conf. The structure closely resembles Hadoop's configuration files. </p>
-<h3 id="a_idedit_confaccumulo-envsha_edit_confaccumulo-envsh"><a id=Edit_conf/accumulo-env.sh></a> Edit conf/accumulo-env.sh</h3>
+<h3 id="wzxhzdk16wzxhzdk17-edit-confaccumulo-envsh"><a id=Edit_conf/accumulo-env.sh></a> Edit conf/accumulo-env.sh</h3>
<p>Accumulo needs to know where to find the software it depends on. Edit accumuloenv. sh and specify the following: </p>
<ol>
<li>Enter the location of the installation directory of Accumulo for $ACCUMULO_HOME</li>
@@ -138,7 +130,7 @@
<p>By default Accumulo TabletServers are set to use 1GB of memory. You may change this by altering the value of $ACCUMULO_TSERVER_OPTS. Note the syntax is that of the Java JVM command line options. This value should be less than the physical memory of the machines running TabletServers. </p>
<p>There are similar options for the master's memory usage and the garbage collector process. Reduce these if they exceed the physical RAM of your hardware and increase them, within the bounds of the physical RAM, if a process fails because of insufficient memory. </p>
<p>Note that you will be specifying the Java heap space in accumulo-env.sh. You should make sure that the total heap space used for the Accumulo tserver and the Hadoop DataNode and TaskTracker is less than the available memory on each slave node in the cluster. On large clusters, it is recommended that the Accumulo master, Hadoop NameNode, secondary NameNode, and Hadoop JobTracker all be run on separate machines to allow them to use more heap space. If you are running these on the same machine on a small cluster, likewise make sure their heap space settings fit within the available memory. </p>
-<h3 id="a_idcluster_specificationa_cluster_specification"><a id=Cluster_Specification></a> Cluster Specification</h3>
+<h3 id="wzxhzdk18wzxhzdk19-cluster-specification"><a id=Cluster_Specification></a> Cluster Specification</h3>
<p>On the machine that will serve as the Accumulo master: </p>
<ol>
<li>Write the IP address or domain name of the Accumulo Master to the <br />
@@ -147,7 +139,7 @@ $ACCUMULO_HOME/conf/masters file. </li>
$ACCUMULO_HOME/conf/slaves, one per line. </li>
</ol>
<p>Note that if using domain names rather than IP addresses, DNS must be configured properly for all machines participating in the cluster. DNS can be a confusing source of errors. </p>
-<h3 id="a_idaccumulo_settingsa_accumulo_settings"><a id=Accumulo_Settings></a> Accumulo Settings</h3>
+<h3 id="wzxhzdk20wzxhzdk21-accumulo-settings"><a id=Accumulo_Settings></a> Accumulo Settings</h3>
<p>Specify appropriate values for the following settings in <br />
$ACCUMULO_HOME/conf/accumulo-site.xml : </p>
<div class="codehilite"><pre><span class="nt"><property></span>
@@ -166,27 +158,27 @@ $ACCUMULO_HOME/conf/accumulo-site.xml :
<p>This enables Accumulo to find ZooKeeper. Accumulo uses ZooKeeper to coordinate settings between processes and helps finalize TabletServer failure. </p>
<p>Accumulo records all changes to tables to a write-ahead log before committing them to the table. The `walog' setting specifies the local directory on each machine to which write-ahead logs are written. This directory should exist on all machines acting as TabletServers. </p>
<p>Some settings can be modified via the Accumulo shell and take effect immediately. However, any settings that should be persisted across system restarts must be recorded in the accumulo-site.xml file. </p>
-<h3 id="a_iddeploy_configurationa_deploy_configuration"><a id=Deploy_Configuration></a> Deploy Configuration</h3>
+<h3 id="wzxhzdk22wzxhzdk23-deploy-configuration"><a id=Deploy_Configuration></a> Deploy Configuration</h3>
<p>Copy the masters, slaves, accumulo-env.sh, and if necessary, accumulo-site.xml from the <br />
$ACCUMULO_HOME/conf/ directory on the master to all the machines specified in the slaves file. </p>
-<h2 id="a_idinitializationa_initialization"><a id=Initialization></a> Initialization</h2>
+<h2 id="wzxhzdk24wzxhzdk25-initialization"><a id=Initialization></a> Initialization</h2>
<p>Accumulo must be initialized to create the structures it uses internally to locate data across the cluster. HDFS is required to be configured and running before Accumulo can be initialized. </p>
<p>Once HDFS is started, initialization can be performed by executing <br />
$ACCUMULO_HOME/bin/accumulo init . This script will prompt for a name for this instance of Accumulo. The instance name is used to identify a set of tables and instance-specific settings. The script will then write some information into HDFS so Accumulo can start properly. </p>
<p>The initialization script will prompt you to set a root password. Once Accumulo is initialized it can be started. </p>
-<h2 id="a_idrunninga_running"><a id=Running></a> Running</h2>
-<h3 id="a_idstarting_accumuloa_starting_accumulo"><a id=Starting_Accumulo></a> Starting Accumulo</h3>
+<h2 id="wzxhzdk26wzxhzdk27-running"><a id=Running></a> Running</h2>
+<h3 id="wzxhzdk28wzxhzdk29-starting-accumulo"><a id=Starting_Accumulo></a> Starting Accumulo</h3>
<p>Make sure Hadoop is configured on all of the machines in the cluster, including access to a shared HDFS instance. Make sure HDFS and ZooKeeper are running. Make sure ZooKeeper is configured and running on at least one machine in the cluster. Start Accumulo using the bin/start-all.sh script. </p>
<p>To verify that Accumulo is running, check the Status page as described under <em>Monitoring</em>. In addition, the Shell can provide some information about the status of tables via reading the !METADATA table. </p>
-<h3 id="a_idstopping_accumuloa_stopping_accumulo"><a id=Stopping_Accumulo></a> Stopping Accumulo</h3>
+<h3 id="wzxhzdk30wzxhzdk31-stopping-accumulo"><a id=Stopping_Accumulo></a> Stopping Accumulo</h3>
<p>To shutdown cleanly, run bin/stop-all.sh and the master will orchestrate the shutdown of all the tablet servers. Shutdown waits for all minor compactions to finish, so it may take some time for particular configurations. </p>
-<h2 id="a_idmonitoringa_monitoring"><a id=Monitoring></a> Monitoring</h2>
+<h2 id="wzxhzdk32wzxhzdk33-monitoring"><a id=Monitoring></a> Monitoring</h2>
<p>The Accumulo Master provides an interface for monitoring the status and health of Accumulo components. This interface can be accessed by pointing a web browser to <br />
http://accumulomaster:50095/status</p>
-<h2 id="a_idlogginga_logging"><a id=Logging></a> Logging</h2>
+<h2 id="wzxhzdk34wzxhzdk35-logging"><a id=Logging></a> Logging</h2>
<p>Accumulo processes each write to a set of log files. By default these are found under <br />
$ACCUMULO/logs/. </p>
-<h2 id="a_idrecoverya_recovery"><a id=Recovery></a> Recovery</h2>
+<h2 id="wzxhzdk36wzxhzdk37-recovery"><a id=Recovery></a> Recovery</h2>
<p>In the event of TabletServer failure or error on shutting Accumulo down, some mutations may not have been minor compacted to HDFS properly. In this case, Accumulo will automatically reapply such mutations from the write-ahead log either when the tablets from the failed server are reassigned by the Master, in the case of a single TabletServer failure or the next time Accumulo starts, in the event of failure during shutdown. </p>
<p>Recovery is performed by asking the loggers to copy their write-ahead logs into HDFS. As the logs are copied, they are also sorted, so that tablets can easily find their missing updates. The copy/sort status of each file is displayed on Accumulo monitor status page. Once the recovery is complete any tablets involved should return to an ``online" state. Until then those tablets will be unavailable to clients. </p>
<p>The Accumulo client library is configured to retry failed mutations and in many cases clients will be able to continue processing after the recovery process without throwing an exception. </p>
Modified: websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Analytics.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Analytics.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Analytics.html Fri Mar 23 19:08:12 2012
@@ -63,22 +63,14 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - klzzwxh:0006 -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
-<h1 id="asf_links">ASF links</h1>
+<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
@@ -101,16 +93,16 @@
<li><a href="Analytics.html#Statistical_Modeling">Statistical Modeling</a></li>
</ul>
<hr />
-<h2 id="a_idanalyticsa_analytics"><a id=Analytics></a> Analytics</h2>
+<h2 id="wzxhzdk10wzxhzdk11-analytics"><a id=Analytics></a> Analytics</h2>
<p>Accumulo supports more advanced data processing than simply keeping keys sorted and performing efficient lookups. Analytics can be developed by using MapReduce and Iterators in conjunction with Accumulo tables. </p>
-<h2 id="a_idmapreducea_mapreduce"><a id=MapReduce></a> MapReduce</h2>
+<h2 id="wzxhzdk12wzxhzdk13-mapreduce"><a id=MapReduce></a> MapReduce</h2>
<p>Accumulo tables can be used as the source and destination of MapReduce jobs. To use an Accumulo table with a MapReduce job (specifically with the new Hadoop API as of version 0.20), configure the job parameters to use the AccumuloInputFormat and AccumuloOutputFormat. Accumulo specific parameters can be set via these two format classes to do the following: </p>
<ul>
<li>Authenticate and provide user credentials for the input </li>
<li>Restrict the scan to a range of rows </li>
<li>Restrict the input to a subset of available columns </li>
</ul>
-<h3 id="a_idmapper_and_reducer_classesa_mapper_and_reducer_classes"><a id=Mapper_and_Reducer_classes></a> Mapper and Reducer classes</h3>
+<h3 id="wzxhzdk14wzxhzdk15-mapper-and-reducer-classes"><a id=Mapper_and_Reducer_classes></a> Mapper and Reducer classes</h3>
<p>To read from an Accumulo table create a Mapper with the following class parameterization and be sure to configure the AccumuloInputFormat. </p>
<div class="codehilite"><pre><span class="n">class</span> <span class="n">MyMapper</span> <span class="n">extends</span> <span class="n">Mapper</span><span class="sr"><Key,Value,WritableComparable,Writable></span> <span class="p">{</span>
<span class="n">public</span> <span class="n">void</span> <span class="nb">map</span><span class="p">(</span><span class="n">Key</span> <span class="n">k</span><span class="p">,</span> <span class="n">Value</span> <span class="n">v</span><span class="p">,</span> <span class="n">Context</span> <span class="n">c</span><span class="p">)</span> <span class="p">{</span>
@@ -136,7 +128,7 @@
<p>The Text object passed as the output should contain the name of the table to which this mutation should be applied. The Text can be null in which case the mutation will be applied to the default table name specified in the AccumuloOutputFormat options. </p>
-<h3 id="a_idaccumuloinputformat_optionsa_accumuloinputformat_options"><a id=AccumuloInputFormat_options></a> AccumuloInputFormat options</h3>
+<h3 id="wzxhzdk16wzxhzdk17-accumuloinputformat-options"><a id=AccumuloInputFormat_options></a> AccumuloInputFormat options</h3>
<div class="codehilite"><pre><span class="n">Job</span> <span class="n">job</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Job</span><span class="p">(</span><span class="n">getConf</span><span class="p">());</span>
<span class="n">AccumuloInputFormat</span><span class="o">.</span><span class="n">setInputInfo</span><span class="p">(</span><span class="n">job</span><span class="p">,</span>
<span class="s">"user"</span><span class="p">,</span>
@@ -169,7 +161,7 @@
</pre></div>
-<h3 id="a_idaccumulooutputformat_optionsa_accumulooutputformat_options"><a id=AccumuloOutputFormat_options></a> AccumuloOutputFormat options</h3>
+<h3 id="wzxhzdk18wzxhzdk19-accumulooutputformat-options"><a id=AccumuloOutputFormat_options></a> AccumuloOutputFormat options</h3>
<div class="codehilite"><pre><span class="n">boolean</span> <span class="n">createTables</span> <span class="o">=</span> <span class="n">true</span><span class="p">;</span>
<span class="n">String</span> <span class="n">defaultTable</span> <span class="o">=</span> <span class="s">"mytable"</span><span class="p">;</span>
@@ -192,13 +184,13 @@
<p>An example of using MapReduce with Accumulo can be found at <br />
accumulo/docs/examples/README.mapred </p>
-<h2 id="a_idaggregating_iteratorsa_aggregating_iterators"><a id=Aggregating_Iterators></a> Aggregating Iterators</h2>
+<h2 id="wzxhzdk20wzxhzdk21-aggregating-iterators"><a id=Aggregating_Iterators></a> Aggregating Iterators</h2>
<p>Many applications can benefit from the ability to aggregate values across common keys. This can be done via aggregating iterators and is similar to the Reduce step in MapReduce. This provides the ability to define online, incrementally updated analytics without the overhead or latency associated with batch-oriented MapReduce jobs. </p>
<p>All that is needed to aggregate values of a table is to identify the fields over which values will be grouped, insert mutations with those fields as the key, and configure the table with an aggregating iterator that supports the summarization operation desired. </p>
<p>The only restriction on an aggregating iterator is that the aggregator developer should not assume that all values for a given key have been seen, since new mutations can be inserted at anytime. This precludes using the total number of values in the aggregation such as when calculating an average, for example. </p>
-<h3 id="a_idfeature_vectorsa_feature_vectors"><a id=Feature_Vectors></a> Feature Vectors</h3>
+<h3 id="wzxhzdk22wzxhzdk23-feature-vectors"><a id=Feature_Vectors></a> Feature Vectors</h3>
<p>An interesting use of aggregating iterators within an Accumulo table is to store feature vectors for use in machine learning algorithms. For example, many algorithms such as k-means clustering, support vector machines, anomaly detection, etc. use the concept of a feature vector and the calculation of distance metrics to learn a particular model. The columns in an Accumulo table can be used to efficiently store sparse features and their weights to be incrementally updated via the use of an aggregating iterator. </p>
-<h2 id="a_idstatistical_modelinga_statistical_modeling"><a id=Statistical_Modeling></a> Statistical Modeling</h2>
+<h2 id="wzxhzdk24wzxhzdk25-statistical-modeling"><a id=Statistical_Modeling></a> Statistical Modeling</h2>
<p>Statistical models that need to be updated by many machines in parallel could be similarly stored within an Accumulo table. For example, a MapReduce job that is iteratively updating a global statistical model could have each map or reduce worker reference the parts of the model to be read and updated through an embedded Accumulo client. </p>
<p>Using Accumulo this way enables efficient and fast lookups and updates of small pieces of information in a random access pattern, which is complementary to MapReduce's sequential access model. </p>
<hr />
Modified: websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Contents.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Contents.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Contents.html Fri Mar 23 19:08:12 2012
@@ -63,22 +63,14 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - klzzwxh:0006 -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
-<h1 id="asf_links">ASF links</h1>
+<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
@@ -94,7 +86,7 @@
<h1 class="title">Apache Accumulo User Manual: Contents</h1>
<p><strong> Next:</strong> <a href="Introduction.html">Introduction</a> <strong> Up:</strong> <a href="accumulo_user_manual.html">Apache Accumulo User Manual Version 1.3</a> <strong> Previous:</strong> <a href="accumulo_user_manual.html">Apache Accumulo User Manual Version 1.3</a> <br />
</p>
-<h3 id="a_idcontentsa_contents"><a id=Contents></a> Contents</h3>
+<h3 id="wzxhzdk0wzxhzdk1-contents"><a id=Contents></a> Contents</h3>
<ul>
<li><a href="Introduction.html">Introduction</a></li>
<li>
Modified: websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/High_Speed_Ingest.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/High_Speed_Ingest.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/High_Speed_Ingest.html Fri Mar 23 19:08:12 2012
@@ -63,22 +63,14 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - klzzwxh:0006 -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
-<h1 id="asf_links">ASF links</h1>
+<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
@@ -102,9 +94,9 @@
<li><a href="High_Speed_Ingest.html#MapReduce_Ingest">MapReduce Ingest</a></li>
</ul>
<hr />
-<h2 id="a_idhigh-speed_ingesta_high-speed_ingest"><a id=High-Speed_Ingest></a> High-Speed Ingest</h2>
+<h2 id="wzxhzdk5wzxhzdk6-high-speed-ingest"><a id=High-Speed_Ingest></a> High-Speed Ingest</h2>
<p>Accumulo is often used as part of a larger data processing and storage system. To maximize the performance of a parallel system involving Accumulo, the ingestion and query components should be designed to provide enough parallelism and concurrency to avoid creating bottlenecks for users and other systems writing to and reading from Accumulo. There are several ways to achieve high ingest performance. </p>
-<h2 id="a_idpre-splitting_new_tablesa_pre-splitting_new_tables"><a id=Pre-Splitting_New_Tables></a> Pre-Splitting New Tables</h2>
+<h2 id="wzxhzdk7wzxhzdk8-pre-splitting-new-tables"><a id=Pre-Splitting_New_Tables></a> Pre-Splitting New Tables</h2>
<p>New tables consist of a single tablet by default. As mutations are applied, the table grows and splits into multiple tablets which are balanced by the Master across TabletServers. This implies that the aggregate ingest rate will be limited to fewer servers than are available within the cluster until the table has reached the point where there are tablets on every TabletServer. </p>
<p>Pre-splitting a table ensures that there are as many tablets as desired available before ingest begins to take advantage of all the parallelism possible with the cluster hardware. Tables can be split anytime by using the shell: </p>
<div class="codehilite"><pre><span class="n">user</span><span class="nv">@myinstance</span> <span class="n">mytable</span><span class="o">></span> <span class="n">addsplits</span> <span class="o">-</span><span class="n">sf</span> <span class="o">/</span><span class="n">local_splitfile</span> <span class="o">-</span><span class="n">t</span> <span class="n">mytable</span>
@@ -112,10 +104,10 @@
<p>For the purposes of providing parallelism to ingest it is not necessary to create more tablets than there are physical machines within the cluster as the aggregate ingest rate is a function of the number of physical machines. Note that the aggregate ingest rate is still subject to the number of machines running ingest clients, and the distribution of rowIDs across the table. The aggregation ingest rate will be suboptimal if there are many inserts into a small number of rowIDs. </p>
-<h2 id="a_idmultiple_ingester_clientsa_multiple_ingester_clients"><a id=Multiple_Ingester_Clients></a> Multiple Ingester Clients</h2>
+<h2 id="wzxhzdk9wzxhzdk10-multiple-ingester-clients"><a id=Multiple_Ingester_Clients></a> Multiple Ingester Clients</h2>
<p>Accumulo is capable of scaling to very high rates of ingest, which is dependent upon not just the number of TabletServers in operation but also the number of ingest clients. This is because a single client, while capable of batching mutations and sending them to all TabletServers, is ultimately limited by the amount of data that can be processed on a single machine. The aggregate ingest rate will scale linearly with the number of clients up to the point at which either the aggregate I/O of TabletServers or total network bandwidth capacity is reached. </p>
<p>In operational settings where high rates of ingest are paramount, clusters are often configured to dedicate some number of machines solely to running Ingester Clients. The exact ratio of clients to TabletServers necessary for optimum ingestion rates will vary according to the distribution of resources per machine and by data type. </p>
-<h2 id="a_idbulk_ingesta_bulk_ingest"><a id=Bulk_Ingest></a> Bulk Ingest</h2>
+<h2 id="wzxhzdk11wzxhzdk12-bulk-ingest"><a id=Bulk_Ingest></a> Bulk Ingest</h2>
<p>Accumulo supports the ability to import files produced by an external process such as MapReduce into an existing table. In some cases it may be faster to load data this way rather than via ingesting through clients using BatchWriters. This allows a large number of machines to format data the way Accumulo expects. The new files can then simply be introduced to Accumulo via a shell command. </p>
<p>To configure MapReduce to format data in preparation for bulk loading, the job should be set to use a range partitioner instead of the default hash partitioner. The range partitioner uses the split points of the Accumulo table that will receive the data. The split points can be obtained from the shell and used by the MapReduce RangePartitioner. Note that this is only useful if the existing table is already split into multiple tablets. </p>
<div class="codehilite"><pre><span class="n">user</span><span class="nv">@myinstance</span> <span class="n">mytable</span><span class="o">></span> <span class="n">getsplits</span>
@@ -137,7 +129,7 @@
<p>Note that the paths referenced are directories within the same HDFS instance over which Accumulo is running. Accumulo places any files that failed to be added to the second directory specified. </p>
<p>A complete example of using Bulk Ingest can be found at <br />
accumulo/docs/examples/README.bulkIngest </p>
-<h2 id="a_idmapreduce_ingesta_mapreduce_ingest"><a id=MapReduce_Ingest></a> MapReduce Ingest</h2>
+<h2 id="wzxhzdk13wzxhzdk14-mapreduce-ingest"><a id=MapReduce_Ingest></a> MapReduce Ingest</h2>
<p>It is possible to efficiently write many mutations to Accumulo in parallel via a MapReduce job. In this scenario the MapReduce is written to process data that lives in HDFS and write mutations to Accumulo using the AccumuloOutputFormat. See the MapReduce section under Analytics for details. </p>
<p>An example of using MapReduce can be found under <br />
accumulo/docs/examples/README.mapred </p>
Modified: websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Introduction.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Introduction.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Introduction.html Fri Mar 23 19:08:12 2012
@@ -63,22 +63,14 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - klzzwxh:0006 -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
-<h1 id="asf_links">ASF links</h1>
+<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
@@ -94,7 +86,7 @@
<h1 class="title">Apache Accumulo User Manual: Introduction</h1>
<p><strong> Next:</strong> <a href="Accumulo_Design.html">Accumulo Design</a> <strong> Up:</strong> <a href="accumulo_user_manual.html">Apache Accumulo User Manual Version 1.3</a> <strong> Previous:</strong> <a href="Contents.html">Contents</a> <strong> <a href="Contents.html">Contents</a></strong> <br />
</p>
-<h2 id="a_idintroductiona_introduction"><a id=Introduction></a> Introduction</h2>
+<h2 id="wzxhzdk0wzxhzdk1-introduction"><a id=Introduction></a> Introduction</h2>
<p>Apache Accumulo is a highly scalable structured store based on Google's BigTable. Accumulo is written in Java and operates over the Hadoop Distributed File System (HDFS), which is part of the popular Apache Hadoop project. Accumulo supports efficient storage and retrieval of structured data, including queries for ranges, and provides support for using Accumulo tables as input and output for MapReduce jobs. </p>
<p>Accumulo features automatic load-balancing and partitioning, data compression and fine-grained security labels. </p>
<hr />
Modified: websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Security.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Security.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Security.html Fri Mar 23 19:08:12 2012
@@ -63,22 +63,14 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - klzzwxh:0006 -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
-<h1 id="asf_links">ASF links</h1>
+<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
@@ -103,9 +95,9 @@
<li><a href="Security.html#Query_Services_Layer">Query Services Layer</a></li>
</ul>
<hr />
-<h2 id="a_idsecuritya_security"><a id=Security></a> Security</h2>
+<h2 id="wzxhzdk6wzxhzdk7-security"><a id=Security></a> Security</h2>
<p>Accumulo extends the BigTable data model to implement a security mechanism known as cell-level security. Every key-value pair has its own security label, stored under the column visibility element of the key, which is used to determine whether a given user meets the security requirements to read the value. This enables data of various security levels to be stored within the same row, and users of varying degrees of access to query the same table, while preserving data confidentiality. </p>
-<h2 id="a_idsecurity_label_expressionsa_security_label_expressions"><a id=Security_Label_Expressions></a> Security Label Expressions</h2>
+<h2 id="wzxhzdk8wzxhzdk9-security-label-expressions"><a id=Security_Label_Expressions></a> Security Label Expressions</h2>
<p>When mutations are applied, users can specify a security label for each value. This is done as the Mutation is created by passing a ColumnVisibility object to the put() method: </p>
<div class="codehilite"><pre><span class="n">Text</span> <span class="n">rowID</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Text</span><span class="p">(</span><span class="s">"row1"</span><span class="p">);</span>
<span class="n">Text</span> <span class="n">colFam</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Text</span><span class="p">(</span><span class="s">"myColFam"</span><span class="p">);</span>
@@ -120,7 +112,7 @@
</pre></div>
-<h2 id="a_idsecurity_label_expression_syntaxa_security_label_expression_syntax"><a id=Security_Label_Expression_Syntax></a> Security Label Expression Syntax</h2>
+<h2 id="wzxhzdk10wzxhzdk11-security-label-expression-syntax"><a id=Security_Label_Expression_Syntax></a> Security Label Expression Syntax</h2>
<p>Security labels consist of a set of user-defined tokens that are required to read the value the label is associated with. The set of tokens required can be specified using syntax that supports logical AND and OR combinations of tokens, as well as nesting groups of tokens together. </p>
<p>For example, suppose within our organization we want to label our data values with security labels defined in terms of user roles. We might have tokens such as: </p>
<div class="codehilite"><pre><span class="n">admin</span>
@@ -145,7 +137,7 @@
<p>When both <code>|</code> and <code>&</code> operators are used, parentheses must be used to specify precedence of the operators. </p>
-<h2 id="a_idauthorizationa_authorization"><a id=Authorization></a> Authorization</h2>
+<h2 id="wzxhzdk12wzxhzdk13-authorization"><a id=Authorization></a> Authorization</h2>
<p>When clients attempt to read data from Accumulo, any security labels present are examined against the set of authorizations passed by the client code when the Scanner or BatchScanner are created. If the authorizations are determined to be insufficient to satisfy the security label, the value is suppressed from the set of results sent back to the client. </p>
<p>Authorizations are specified as a comma-separated list of tokens the user possesses: </p>
<div class="codehilite"><pre><span class="sr">//</span> <span class="n">user</span> <span class="n">possess</span> <span class="n">both</span> <span class="n">admin</span> <span class="ow">and</span> <span class="nb">system</span> <span class="n">level</span> <span class="n">access</span>
@@ -155,10 +147,10 @@
</pre></div>
-<h2 id="a_idsecure_authorizations_handlinga_secure_authorizations_handling"><a id=Secure_Authorizations_Handling></a> Secure Authorizations Handling</h2>
+<h2 id="wzxhzdk14wzxhzdk15-secure-authorizations-handling"><a id=Secure_Authorizations_Handling></a> Secure Authorizations Handling</h2>
<p>Because the client can pass any authorization tokens to Accumulo, applications must be designed to obtain users' authorization tokens from a trusted 3rd party rather than having the users specify their authorizations directly. </p>
<p>Often production systems will integrate with Public-Key Infrastructure (PKI) and designate client code within the query layer to negotiate with PKI servers in order to authenticate users and retrieve their authorization tokens (credentials). This requires users to specify only the information necessary to authenticate themselves to the system. Once user identity is established, their credentials can be accessed by the client code and passed to Accumulo outside of the reach of the user. </p>
-<h2 id="a_idquery_services_layera_query_services_layer"><a id=Query_Services_Layer></a> Query Services Layer</h2>
+<h2 id="wzxhzdk16wzxhzdk17-query-services-layer"><a id=Query_Services_Layer></a> Query Services Layer</h2>
<p>Since the primary method of interaction with Accumulo is through the Java API, production environments often call for the implementation of a Query layer. This can be done using web services in containers such as Apache Tomcat, but is not a requirement. The Query Services Layer provides a mechanism for providing a platform on which user facing applications can be built. This allows the application designers to isolate potentially complex query logic, and enables a convenient point at which to perform essential security functions. </p>
<p>Several production environments choose to implement authentication at this layer, where users identifiers are used to retrieve their access credentials which are then cached within the query layer and presented to Accumulo through the Authorizations mechanism. </p>
<p>Typically, the query services layer sits between Accumulo and user workstations. </p>
Modified: websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Shell_Commands.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Shell_Commands.html (original)
+++ websites/staging/accumulo/trunk/content/accumulo/user_manual_1.3-incubating/Shell_Commands.html Fri Mar 23 19:08:12 2012
@@ -63,22 +63,14 @@
</ul>
<h1 id="documentation">Documentation</h1>
<ul>
-<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
-<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
-<!-- - klzzwxh:0006 -->
-<!-- - Javadoc -->
-<!-- - Examples --></li>
+<li><a href="/accumulo/1.4/user_manual">Manual 1.4</a></li>
+<li><a href="/accumulo/1.4/apidocs">Javadoc 1.4</a></li>
+<li><a href="/accumulo/1.4/examples">Examples 1.4</a></li>
<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
-<li><a href="/accumulo/papers.html">Papers</a></li>
+<li><a href="/accumulo/papers.html">Papers & Other Links</a></li>
+<li><a href="/accumulo/glossary.html">Glossary</a></li>
</ul>
-<!--
-# Development
- - Source code
- - Building
--->
-
-<h1 id="asf_links">ASF links</h1>
+<h1 id="asf-links">ASF links</h1>
<ul>
<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
@@ -94,7 +86,7 @@
<h1 class="title">Apache Accumulo User Manual: Shell Commands</h1>
<p><strong> Up:</strong> <a href="accumulo_user_manual.html">Apache Accumulo User Manual Version 1.3</a> <strong> Previous:</strong> <a href="Administration.html">Administration</a> <strong> <a href="Contents.html">Contents</a></strong> <br />
</p>
-<h2 id="a_idshell_commandsa_shell_commands"><a id=Shell_Commands></a> Shell Commands</h2>
+<h2 id="wzxhzdk59wzxhzdk60-shell-commands"><a id=Shell_Commands></a> Shell Commands</h2>
<p><strong>?</strong> <br />
</p>
<div class="codehilite"><pre><span class="n">usage:</span> <span class="p">?</span> <span class="p">[</span> <span class="sr"><command></span> <span class="sr"><command></span> <span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="p">?]</span> <span class="p">[</span><span class="o">-</span><span class="n">np</span><span class="p">]</span>