You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by el...@apache.org on 2014/10/03 18:22:41 UTC
svn commit: r1629249 -
/accumulo/site/trunk/content/1.6/accumulo_user_manual.html
Author: elserj
Date: Fri Oct 3 16:22:41 2014
New Revision: 1629249
URL: http://svn.apache.org/r1629249
Log:
ACCUMULO-3142 Hand-edit latex changes into asciidoc-generated HTML user manual
Modified:
accumulo/site/trunk/content/1.6/accumulo_user_manual.html
Modified: accumulo/site/trunk/content/1.6/accumulo_user_manual.html
URL: http://svn.apache.org/viewvc/accumulo/site/trunk/content/1.6/accumulo_user_manual.html?rev=1629249&r1=1629248&r2=1629249&view=diff
==============================================================================
--- accumulo/site/trunk/content/1.6/accumulo_user_manual.html (original)
+++ accumulo/site/trunk/content/1.6/accumulo_user_manual.html Fri Oct 3 16:22:41 2014
@@ -548,6 +548,8 @@ span.icon > [class^="icon-"], span.icon
<li><a href="#_cluster_specification">11.5.3. Cluster Specification</a></li>
<li><a href="#_accumulo_settings">11.5.4. Accumulo Settings</a></li>
<li><a href="#_deploy_configuration">11.5.5. Deploy Configuration</a></li>
+<li><a href="#_sensitive_configuration_values">11.5.6 Sensitive Configuration Values</a></li>
+<li><a href="#_using_a_javakeystorecredentialprovider_for_storage">11.5.7. Using a JavaKeyStoreCredentialProvider for storage</a></li>
</ul>
</li>
<li><a href="#_initialization">11.6. Initialization</a></li>
@@ -597,7 +599,8 @@ span.icon > [class^="icon-"], span.icon
<li><a href="#zookeeper_failure">13.8.2. ZooKeeper Failure</a></li>
</ul>
</li>
-<li><a href="#_file_naming_conventions">13.9. File Naming Conventions</a></li>
+<li><a href="#_upgrade_issues">13.9. Upgrade Issues</a></li>
+<li><a href="#_file_naming_conventions">13.10. File Naming Conventions</a></li>
</ul>
</li>
<li><a href="#configuration">14. Appendix A: Configuration Management</a></li>
@@ -1296,6 +1299,17 @@ Instance inst = new ZooKeeperInstance(in
Connector conn = inst.getConnector("user", new PasswordToken("passwd"));</code></pre>
</div>
</div>
+<div class="paragraph">
+<p>The PasswordToken is the most common implementation of an <code>AuthenticationToken</code>.
+This general interface allows authentication as an Accumulo user to come from a variety of sources
+or means. The CredentialProviderToken leverages the Hadoop CredentialProviders (new in Hadoop 2.6).</p>
+</div>
+<div class="paragraph">
+<p>For example, the CredentialProviderToken can be used in conjunction with a Java KeyStore
+to alleviate passwords stored in cleartext. When stored in HDFS, a single KeyStore can be
+used across an entire instance. Be aware that KeyStores stored on the local filesystem
+must be made available to all nodes in the Accumulo cluster.</p>
+</div>
</div>
<div class="sect2">
<h3 id="_writing_data">4.3. Writing Data</h3>
@@ -1405,10 +1419,10 @@ Scanner scan =
conn.createScanner("table", auths);
scan.setRange(new Range("harry","john"));
-scan.fetchFamily("attributes");
+scan.fetchColumnFamily(new Text("attributes"));
for(Entry<Key,Value> entry : scan) {
- String row = entry.getKey().getRow();
+ Text row = entry.getKey().getRow();
Value value = entry.getValue();
}</code></pre>
</div>
@@ -1478,7 +1492,7 @@ TabletServers in parallel.</p>
BatchScanner bscan =
conn.createBatchScanner("table", auths, 10);
bscan.setRanges(ranges);
-bscan.fetchFamily("attributes");
+bscan.fetchColumnFamily(new Text("attributes"));
for(Entry<Key,Value> entry : scan) {
System.out.println(entry.getValue());
@@ -1944,6 +1958,14 @@ config -t mytable -s table.iterator.minc
config -t mytable -s table.iterator.majc.myiter.opt.myoptionname=myoptionvalue</pre>
</div>
</div>
+<div class="paragraph">
+<p>Typically, a table will have multiple iterators. Accumulo configures a set of
+system level iterators for each table. These iterators provide core functionality
+like visibility label filtering and may not be removed by users. User level iterators
+are applied in the order of their priority. Priority is a user-configured integer;
+iterators with lower numbers go first, passing the results of their iteration on
+to the other iterators up the stack.</p>
+</div>
</div>
<div class="sect3">
<h4 id="_setting_iterators_programmatically">6.4.2. Setting Iterators Programmatically</h4>
@@ -3569,8 +3591,14 @@ mutation.put(colFam, colQual, colVis, ti
<div class="paragraph">
<p>Security labels consist of a set of user-defined tokens that are required to read the
value the label is associated with. The set of tokens required can be specified using
-syntax that supports logical AND and OR combinations of tokens, as well as nesting
-groups of tokens together.</p>
+syntax that supports logical AND <code>&</code> and OR <code>|</code> combinations of tokens, as well as nesting
+groups <code>()</code> of tokens together.</p>
+</div>
+<div class="paragraph">
+<p>Each term is comprised of one to many alpha-numeric characters, hyphens, underscores
+or periods. Optionally, each term may be wrapped in quotation marks which removes the
+restriction on valid characters. In quoted terms, quotation marks and backslash characters
+can be used as characters in the term by escaping them with a backslash.</p>
</div>
<div class="paragraph">
<p>For example, suppose within our organization we want to label our data values with
@@ -3926,17 +3954,20 @@ native map also avoids the performance i
in the JVM by causing it to pause much less frequently.</p>
</div>
<div class="paragraph">
-<p>32-bit and 64-bit Linux versions of the native map ship with the Accumulo dist package.
-For other operating systems, the native map can be built from the codebase in two ways-
-from maven or from the Makefile.</p>
+<p>32-bit and 64-bit Linux and Mac OS X versions of the native map can be built from the
+Accumulo bin package by executing <code>$ACCUMULO_HOME/bin/build_native_library.sh</code>.
+If your system's default compiler options are insufficient, you can add additional compiler
+options to the command line, such as options for the architecture. These will be passed
+to the Makefile in the environment variable USERFLAGS.</p>
+<p>Examples:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
-<p>Build from maven using the following command: <code>mvn clean package -Pnative</code>.</p>
+<p><code>$ACCUMULO_HOME/bin/build_native_library.sh</code></p>
</li>
<li>
-<p>Build from the c++ source by running <code>make</code> in the <code>$ACCUMULO_HOME/server/src/main/c++</code> directory.</p>
+<p><code>$ACCUMULO_HOME/bin/build_native_library.sh -m32</code></p>
</li>
</ol>
</div>
@@ -3944,7 +3975,9 @@ from maven or from the Makefile.</p>
<p>After building the native map from the source, you will find the artifact in
<code>$ACCUMULO_HOME/lib/native</code>. Upon starting up, the tablet server will look
in this directory for the map library. If the file is renamed or moved from its
-target directory, the tablet server may not be able to find it.</p>
+target directory, the tablet server may not be able to find it. The system can also
+locate the native maps shared library by setting <code>LD_LIBRARY_PATH</code> (or
+<code>DYLD_LIBRARY_PATH</code> on Mac OS X) in <code>$ACCUMULO_HOME/conf/accumulo-env.sh</code></p>
</div>
</div>
<div class="sect3">
@@ -3997,7 +4030,9 @@ settings between processes and helps fin
</div>
<div class="paragraph">
<p>The instance needs a secret to enable secure communication between servers. Configure your
-secret and make sure that the <code>accumulo-site.xml</code> file is not readable to other users.</p>
+secret and make sure that the <code>accumulo-site.xml</code> file is not readable to other users.
+For alternatives to storing the <code>instance.secret</code> in plaintext, please read the
+<code>Sensitive Configuration Values</code> section.</p>
</div>
<div class="paragraph">
<p>Some settings can be modified via the Accumulo shell and take effect immediately, but
@@ -4013,6 +4048,69 @@ from the <code>$ACCUMULO_HOME/conf/</cod
specified in the slaves file.</p>
</div>
</div>
+<div class="sect3">
+<h4 id="_sensitive_configuration_values">11.5.6. Sensitive Configuration Values</h4>
+<div class="paragraph">
+<p>Accumulo has a number of properties that can be specified via the accumulo-site.xml
+file which are sensitive in nature, instance.secret and trace.token.property.password
+are two common examples. Both of these properties, if compromised, have the ability
+to result in data being leaked to users who should not have access to that data.</p>
+</div>
+<div class="paragraph">
+<p>In Hadoop-2.6.0, a new CredentialProvider class was introduced which serves as a common
+implementation to abstract away the storage and retrieval of passwords from plaintext
+storage in configuration files. Any Property marked with the <code>Sensitive</code> annotation
+is a candidate for use with these CredentialProviders. For version of Hadoop which lack
+these classes, the feature will just be unavailable for use.</p>
+</div>
+<div class="paragraph">
+<p>A comma separated list of CredentialProviders can be configured using the Accumulo Property
+<code>general.security.credential.provider.paths</code>. Each configured URL will be consulted
+when the Configuration object for accumulo-site.xml is accessed.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_using_a_javakeystorecredentialprovider_for_storage">11.5.7. Using a JavaKeyStoreCredentialProvider for storage</h4>
+<div class="paragraph">
+<p>One of the implementations provided in Hadoop-2.6.0 is a Java KeyStore CredentialProvider.
+Each entry in the KeyStore is the Accumulo Property key name. For example, to store the
+\texttt{instance.secret}, the following command can be used:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>hadoop credential create instance.secret --provider jceks://file/etc/accumulo/conf/accumulo.jceks</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The command will then prompt you to enter the secret to use and create a keystore in:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>/etc/accumulo/conf/accumulo.jceks</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Then, accumulo-site.xml must be configured to use this KeyStore as a CredentialProvider:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlight"><code class="xml language-xml"><property>
+ <name>general.security.credential.provider.paths</name>
+ <value>jceks://file/etc/accumulo/conf/accumulo.jceks</value>
+</property></code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>This configuration will then transparently extract the <code>instance.secret</code> from
+the configured KeyStore and alleviates a human readable storage of the sensitive
+property.</p>
+</div>
+<div class="paragraph">
+<p>A KeyStore can also be stored in HDFS, which will make the KeyStore readily available to
+all Accumulo servers. If the local filesystem is used, be aware that each Accumulo server
+will expect the KeyStore in the same location.</p>
+</div>
+</div>
</div>
<div class="sect2">
<h3 id="_initialization">11.6. Initialization</h3>
@@ -5344,7 +5442,7 @@ the gap between how many splits the old
<div class="paragraph">
<p>For each table, make a new directory that you can move (or copy if you have the HDFS space to do so)
all of the rfiles for a given table into. For example, to process the table with an ID of <code>1</code>, make a new directory,
-say <code>/new-table-1</code> and then copy all files from <code>/accumulo/tables/1/<strong>/</strong>.rf</code> into that directory. Additionally,
+say <code>/new-table-1</code> and then copy all files from <code>/accumulo/tables/1/*/*.rf</code> into that directory. Additionally,
make a directory, <code>/new-table-1-failures</code>, for any failures during the import process. Then, issue the import
command using the Accumulo shell into the new table, telling Accumulo to not re-set the timestamp:</p>
</div>
@@ -5380,7 +5478,25 @@ omission of new data.</p>
</div>
</div>
<div class="sect2">
-<h3 id="_file_naming_conventions">13.9. File Naming Conventions</h3>
+<h3 id="_upgrade_issues">13.9. Upgrade Issues</h3>
+<div class="paragraph">
+<p><strong>Q</strong>: I upgrade from 1.4 to 1.5 to 1.6 but still have some WAL files on local disk. Do I have any way
+to recover them?</p>
+</div>
+<div class="paragraph">
+<p><strong>A</strong>: Yes, you can recover them by running the LocalWALRecovery utility on each node that needs
+recovery performed. The utility will default to using the directory specified by <code>logger.dir.walog</code> in your
+configuration, or can be overriden by using the <code>--local-wal-directories</code> option on the tool. It can be
+invoked as follows:</p>
+<div class="literalblock">
+<div class="content">
+<pre>$ACCUMULO_HOME/bin/accumulo org.apache.accumulo.tserver.log.LocalWALRecovery</pre>
+</div>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_file_naming_conventions">13.10. File Naming Conventions</h3>
<div class="paragraph">
<p><strong>Q</strong>: Why are files named like they are? Why do some start with <code>C</code> and others with <code>F</code>?</p>
</div>