You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by bh...@apache.org on 2014/05/01 23:02:38 UTC
svn commit: r1591779 - in /accumulo/site/branches/redesign14: ./
content/bylaws.mdtext content/glossary.mdtext
content/release_notes/1.6.0.mdtext
Author: bhavanki
Date: Thu May 1 21:02:38 2014
New Revision: 1591779
URL: http://svn.apache.org/r1591779
Log:
Merged trunk into redesign14 branch - bylaw update, glossary update, relnotes 1.6.0 update.
Modified:
accumulo/site/branches/redesign14/ (props changed)
accumulo/site/branches/redesign14/content/bylaws.mdtext
accumulo/site/branches/redesign14/content/glossary.mdtext
accumulo/site/branches/redesign14/content/release_notes/1.6.0.mdtext
Propchange: accumulo/site/branches/redesign14/
------------------------------------------------------------------------------
Merged /accumulo/site/trunk:r1587699-1591778
Modified: accumulo/site/branches/redesign14/content/bylaws.mdtext
URL: http://svn.apache.org/viewvc/accumulo/site/branches/redesign14/content/bylaws.mdtext?rev=1591779&r1=1591778&r2=1591779&view=diff
==============================================================================
--- accumulo/site/branches/redesign14/content/bylaws.mdtext (original)
+++ accumulo/site/branches/redesign14/content/bylaws.mdtext Thu May 1 21:02:38 2014
@@ -17,7 +17,7 @@ Notice: Licensed to the Apache Softwa
specific language governing permissions and limitations
under the License.
-This is version 2 of the bylaws. Community work actively continues on the bylaws, and so key segments of them are subject to change.
+This is version 3 of the bylaws. Community work actively continues on the bylaws, and so key segments of them are subject to change.
# Introduction
@@ -210,7 +210,7 @@ For Code Change actions, a committer may
<td>3</td>
<tr><td>Modifying Bylaws</td>
<td>Modifying this document.</td>
- <td>Majority approval</td>
+ <td>Consensus approval</td>
<td>Active PMC members</td>
<td>7</td>
</table>
Modified: accumulo/site/branches/redesign14/content/glossary.mdtext
URL: http://svn.apache.org/viewvc/accumulo/site/branches/redesign14/content/glossary.mdtext?rev=1591779&r1=1591778&r2=1591779&view=diff
==============================================================================
--- accumulo/site/branches/redesign14/content/glossary.mdtext (original)
+++ accumulo/site/branches/redesign14/content/glossary.mdtext Thu May 1 21:02:38 2014
@@ -37,7 +37,7 @@ Notice: Licensed to the Apache Softwa
<dt>iterator scopes</dt>
<dd>the possible scopes for iterators are where the tablet server is already reading and/or writing data: minor compaction / flush time (<em>minc</em> scope), major compaction / file merging time (<em>majc</em> scope), and query time (<em>scan</em> scope).</dd>
<dt>gc</dt>
-<dd> </dd>
+<dd>process that identifies temporary files in HDFS that are no longer needed by any process, and deletes them.</dd>
<dt>key</dt>
<dd>the key into the distributed sorted map which is accumulo. The key is subdivided into row, column, and timestamp. The column is further divided into family, qualifier, and visibility.</dd>
<dt>locality group</dt>
@@ -45,19 +45,19 @@ Notice: Licensed to the Apache Softwa
<dt>log-structured merge-tree</dt>
<dd>the sorting / flushing / merging scheme on which BigTable's design is based.</dd>
<dt>logger</dt>
-<dd> </dd>
+<dd>in 1.4 and older, process that accepts updates to tablet servers and writes them to local on-disk storage for redundancy. in 1.5 the functionality was subsumed by the tablet server and datanode with HDFS writes.</dd>
<dt>major compaction</dt>
<dd>merging multiple files into a single file. If all of a tablet's files are merged into a single file, it is called a <em>full major compaction</em>.</dd>
<dt>master</dt>
-<dd> </dd>
+<dd>process that detects and responds to tablet failures, balances load across tablet servers by assigning and migrating tablets when required, coordinates table operations, and handles tablet server logistics (startup, shutdown, recovery).</dd>
<dt>minor compaction</dt>
<dd>flushing data from memory to disk. Usually this creates a new file for a tablet, but if the memory flushed is merge-sorted in with data from an existing file (replacing that file), it is called a <em>merging minor compaction</em>.</dd>
<dt>monitor</dt>
-<dd> </dd>
+<dd>process that displays status and usage information for all Accumulo components.</dd>
<dt>permissions</dt>
<dd>administrative abilities that must be given to a user such as creating tables or users and changing permissions or configuration parameters.</dd>
<dt>row</dt>
-<dd>the portion of the key that is controls atomicity. Keys with the same row are guaranteed to remain on a single tablet hosted by a single tablet server, therefore multiple key/value pairs can be added to or removed from a row at the same time. The row is used for the primary sorting of the key.</dd>
+<dd>the portion of the key that controls atomicity. Keys with the same row are guaranteed to remain on a single tablet hosted by a single tablet server, therefore multiple key/value pairs can be added to or removed from a row at the same time. The row is used for the primary sorting of the key.</dd>
<dt>scan</dt>
<dd>reading a range of key/value pairs.</dd>
<dt>tablet</dt>
@@ -68,4 +68,5 @@ Notice: Licensed to the Apache Softwa
<dd>the portion of the key that controls versioning. Otherwise identical keys with differing timestamps are considered to be versions of a single <em>cell</em>. Accumulo can be configured to keep the <em>N</em> newest versions of each <em>cell</em>. When a deletion entry is inserted, it deletes all earlier versions for its cell.</dd>
<dt>value</dt>
<dd>immutable bytes associated with a particular key.</dd>
-</dl>
\ No newline at end of file
+</dl>
+
Modified: accumulo/site/branches/redesign14/content/release_notes/1.6.0.mdtext
URL: http://svn.apache.org/viewvc/accumulo/site/branches/redesign14/content/release_notes/1.6.0.mdtext?rev=1591779&r1=1591778&r2=1591779&view=diff
==============================================================================
--- accumulo/site/branches/redesign14/content/release_notes/1.6.0.mdtext (original)
+++ accumulo/site/branches/redesign14/content/release_notes/1.6.0.mdtext Thu May 1 21:02:38 2014
@@ -1,4 +1,4 @@
-Title:
+Title: Release Notes for Apache Accumulo 1.6.0
Notice: Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
@@ -24,7 +24,7 @@ Accumulo 1.6.0 runs on Hadoop 1, however
## Notable Improvements
-### Multiple namenode support
+### Multiple volume support
[BigTable's][1] design allows for its internal metadata to automatically spread across multiple nodes. Accumulo has followed this design and scales very well as a result. There is one impediment to scaling though, and this is the HDFS namenode. There are two problems with the namenode when it comes to scaling. First, the namenode stores all of its filesystem metadata in memory on a single machine. This introduces an upper bound on the number of files Accumulo can have. Second, there is an upper bound on the number of file operations per second that a single namenode can support. For example, a namenode can only support a few thousand delete or create file request per second.
@@ -46,7 +46,9 @@ The only built in test that conditional
### Encryption
-Support for encrypting Accumulo's persistent and over the wire data was added. [ACCUMULO-998][ACCUMULO-998], [ACCUMULO-958][ACCUMULO-958], and [ACCUMULO-980][ACCUMULO-980] cover encrypting data at rest in write ahead logs and rfiles. [ACCUMULO-1009][ACCUMULO-1009] covers encrypting data over the wire using SSL.
+Encryption is still an experimental feature, but much progress has been made since 1.5.0. Support for encrypting rfiles and write ahead logs were added in [ACCUMULO-958][ACCUMULO-958] and [ACCUMULO-980][ACCUMULO-980]. Support for encrypting data over the wire using SSL was added in [ACCUMULO-1009][ACCUMULO-1009].
+
+When a tablet server fails, its write ahead logs are sorted and stored in HDFS. In 1.6.0, encrypting these sorted write ahead logs is not supported. [ACCUMULO-981][ACCUMULO-981] addresses this issue.
### Pluggable compaction strategies
@@ -82,6 +84,42 @@ While Hadoop [does not support IPv6 netw
Multiple bug-fixes were made to support running Accumulo over multiple HDFS instances using ViewFS. [ACCUMULO-2047][ACCUMULO-2047] is the parent
ticket that contains numerous fixes to enable this support.
+### Maven Plugin
+
+This version of Accumulo is accompanied by a new maven plugin for testing client apps ([ACCUMULO-1030][ACCUMULO-1030]). You can execute the accumulo-maven-plugin inside your project by adding the following to your pom.xml's build plugins section:
+
+ <plugin>
+ <groupId>org.apache.accumulo</groupId>
+ <artifactId>accumulo-maven-plugin</artifactId>
+ <version>1.6.0</version>
+ <configuration>
+ <instanceName>plugin-it-instance</instanceName>
+ <rootPassword>ITSecret</rootPassword>
+ </configuration>
+ <executions>
+ <execution>
+ <id>run-plugin</id>
+ <goals>
+ <goal>start</goal>
+ <goal>stop</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+
+This plugin is designed to work in conjunction with the maven-failsafe-plugin. A small test instance of Accumulo will run during the pre-integration-test phase of the Maven build lifecycle, and will be stopped in the post-integration-test phase. Your integration tests, executed by maven-failsafe-plugin can access this instance with a MiniAccumuloInstance connector (the plugin uses MiniAccumuloInstance, internally), as in the following example:
+
+ private static Connector conn;
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ String instanceName = "plugin-it-instance";
+ Instance instance = new MiniAccumuloInstance(instanceName, new File("target/accumulo-maven-plugin/" + instanceName));
+ conn = instance.getConnector("root", new PasswordToken("ITSecret"));
+ }
+
+This plugin is quite limited, currently only supporting an instance name and a root user password as configuration parameters. Improvements are expected in future releases, so feedback is welcome and appreciated (file bugs/requests under the "maven-plugin" component in the Accumulo JIRA).
+
### Packaging
One notable change that was made to the binary tarball is the purposeful omission of a pre-built copy of the Accumulo "native map" library.
@@ -94,6 +132,12 @@ It is recommended that users invoke the
Be aware that you will need a C++ compiler/toolchain installed to build this library. Check your GNU/Linux distribution documentation for the package manager command.
+#### Size-Based Constraint on New Tables
+
+A Constraint is an interface that can determine if a Mutation should be applied or rejected server-side. New tables that are created in 1.6.0 will automatically have the `DefaultKeySizeConstraint` set.
+As performance can suffer when large Keys are inserted into a table, this Constraint will reject any Key that is larger than 1MB. If this constraint is undesired, it can be removed using the `constraint` shell
+command. See the help message on the command for more information.
+
### Other notable changes
* [ACCUMULO-842][ACCUMULO-842] Added FATE administration to shell
@@ -132,6 +176,8 @@ Be aware that you will need a C++ compil
## Known Issues
+### Slower writes than previous Accumulo versions
+
When using Accumulo 1.6 and Hadoop 2, Accumulo will call hsync() on HDFS.
Calling hsync improves durability by ensuring data is on disk (where other older
Hadoop versions might lose data in the face of power failure); however, calling
@@ -145,8 +191,20 @@ mutation queues.
For more information, see [ACCUMULO-1950][ACCUMULO-1950] and [this comment][ACCUMULO-1905-comment].
+Another possible cause of slower writes is the change in write ahead log replication
+between 1.4 and 1.5. Accumulo 1.4. defaulted to two loggers servers. Accumulo 1.5 and 1.6 store
+write ahead logs in HDFS and default to using three datanodes.
+
+### BatchWriter hold time error
+
+If a `BatchWriter` fails with `MutationsRejectedException` and the message contains
+`"# server errors 1"` then it may be [ACCUMULO-2388][ACCUMULO-2388]. To confirm this look in the tablet server logs
+for `org.apache.accumulo.tserver.HoldTimeoutException` around the time the `BatchWriter` failed.
+If this is happening often a possible work around is to set `general.rpc.timeout` to `240s`.
+
### Other known issues
+ * [ACCUMULO-981][ACCUMULO-981] Sorted write ahead logs are not encrypted.
* [ACCUMULO-1507][ACCUMULO-1507] Dynamic Classloader still can't keep proper track of jars
* [ACCUMULO-1588][ACCUMULO-1588] Monitor XML and JSON differ
* [ACCUMULO-1628][ACCUMULO-1628] NPE on deep copied dumped memory iterator
@@ -214,13 +272,14 @@ and, in HDFS High-Availability instances
[ACCUMULO-802]: https://issues.apache.org/jira/browse/ACCUMULO-802 "Table namespaces"
[ACCUMULO-842]: https://issues.apache.org/jira/browse/ACCUMULO-842 "Add FATE administration to shell"
[ACCUMULO-958]: https://issues.apache.org/jira/browse/ACCUMULO-958 "Support pluggable encryption in walogs"
-[ACCUMULO-998]: https://issues.apache.org/jira/browse/ACCUMULO-998 "Support encryption at rest"
[ACCUMULO-970]: https://issues.apache.org/jira/browse/ACCUMULO-970
[ACCUMULO-980]: https://issues.apache.org/jira/browse/ACCUMULO-980 "Support pluggable codecs for RFile"
+[ACCUMULO-981]: https://issues.apache.org/jira/browse/ACCUMULO-981 "support pluggable encryption when recovering write-ahead logs"
[ACCUMULO-1000]: https://issues.apache.org/jira/browse/ACCUMULO-1000 "Conditional Mutations"
[ACCUMULO-1009]: https://issues.apache.org/jira/browse/ACCUMULO-1009 "Support encryption over the wire"
[ACCUMULO-1018]: https://issues.apache.org/jira/browse/ACCUMULO-1018 "Client does not give informative message when user can not read table"
[ACCUMULO-1024]: https://issues.apache.org/jira/browse/ACCUMULO-1024
+[ACCUMULO-1030]: https://issues.apache.org/jira/browse/ACCUMULO-1030 "Create a Maven plugin to run MiniAccumuloCluster for integration testing"
[ACCUMULO-1218]: https://issues.apache.org/jira/browse/ACCUMULO-1218 "document the recovery from a failed zookeeper"
[ACCUMULO-1336]: https://issues.apache.org/jira/browse/ACCUMULO-1336 "Add lexicoders from Typo to Accumulo"
[ACCUMULO-1375]: https://issues.apache.org/jira/browse/ACCUMULO-1375 "Update README files in proxy module."
@@ -270,6 +329,7 @@ and, in HDFS High-Availability instances
[ACCUMULO-2262]: https://issues.apache.org/jira/browse/ACCUMULO-2262 "Include java.net.preferIPv4Stack=true in process startup"
[ACCUMULO-2334]: https://issues.apache.org/jira/browse/ACCUMULO-2334 "Lacking fallback when ACCUMULO_LOG_HOST isn't set"
[ACCUMULO-2360]: https://issues.apache.org/jira/browse/ACCUMULO-2360 "Need a way to configure TNonblockingServer.maxReadBufferBytes to prevent OOMs"
+[ACCUMULO-2388]: https://issues.apache.org/jira/browse/ACCUMULO-2388
[ACCUMULO-2408]: https://issues.apache.org/jira/browse/ACCUMULO-2408 "metadata table not assigned after root table is loaded"
[ACCUMULO-2441]: https://issues.apache.org/jira/browse/ACCUMULO-2441 "Document internal state stored in RFile names"
[ACCUMULO-2495]: https://issues.apache.org/jira/browse/ACCUMULO-2495 "OOM exception didn't bring down tserver"