You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by mikewalch <gi...@git.apache.org> on 2017/03/23 14:06:42 UTC
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
GitHub user mikewalch opened a pull request:
https://github.com/apache/accumulo/pull/235
ACCUMULO-4612 - Simplify Accumulo memory configuration
* Tablet server memory setttings now default to a percentage
of Java heap space given to process. Default settings will
work for various heap space settings and fixed memory
settings are still supported.
* Remove 'accumulo create-config' command as it no longer needed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mikewalch/accumulo config-refactor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/accumulo/pull/235.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #235
----
commit d156d7008edeaaa219edc35ea10a14d43abe6a6e
Author: Mike Walch <mw...@apache.org>
Date: 2017-03-20T20:36:45Z
ACCUMULO-4612 - Simplify Accumulo memory configuration
* Tablet server memory setttings now default to a percentage
of Java heap space given to process. Default settings will
work for various heap space settings and fixed memory
settings are still supported.
* Remove 'accumulo create-config' command as it no longer needed.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107943447
--- Diff: INSTALL.md ---
@@ -36,62 +36,49 @@ to manage Accumulo:
These scripts will be used in the remaining instructions to configure and run Accumulo.
For convenience, consider adding `accumulo-X.Y.Z/bin/` to your shell's path.
-## Configuring
+## Configuring Accumulo
-Accumulo has some optional native code that improves its performance and
-stability. Before configuring Accumulo, attempt to build this native code
-with the following command.
+Accumulo requires running [Zookeeper][3] and [HDFS][4] instances which should be set up
+before configuring Accumulo.
- accumulo-util build-native
+The primary configuration files for Accumulo are `accumulo-env.sh` and `accumulo-site.xml`
+which are located in the `conf/` directory.
-If the command fails, its OK to continue with setup and resolve the issue later.
+Follow the steps below to configure `accumulo-site.xml`:
-Accumulo is configured by the files `accumulo-site.xml` and `accumulo-env.sh` in the `conf/`
-directory. You can either edit these files for your environment or run the command below which will
-overwrite them with files configured for your environment.
+1. Run `accumulo-util build-native` to build native code. If this command fails, disable
+ native maps by setting `tserver.memory.maps.native.enabled` to `false`.
- accumulo-util create-config
+2. Set `instance.volumes` to HDFS location where Accumulo will store data. If your namenode
+ is running at 192.168.1.9:9000 and you want to store data in `/accumulo` in HDFS, then set
--- End diff --
This is now fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by ctubbsii <gi...@git.apache.org>.
Github user ctubbsii commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107805516
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java ---
@@ -165,30 +165,35 @@ private void checkType(Property property, PropertyType type) {
}
/**
- * Gets a property of type {@link PropertyType#MEMORY}, interpreting the value properly.
+ * Gets a property of type {@link PropertyType#BYTES}, interpreting the value properly.
*
* @param property
- * property to get
+ * Property to get
* @return property value
* @throws IllegalArgumentException
- * if the property is of the wrong type
- * @see #getMemoryInBytes(String)
+ * if the property is of the wrong type @see #getBytes(String)
*/
- public long getMemoryInBytes(Property property) {
- checkType(property, PropertyType.MEMORY);
+ public long getAsBytes(Property property) {
+ checkType(property, PropertyType.BYTES);
String memString = get(property);
- return getMemoryInBytes(memString);
+ if (property.getType().equals(PropertyType.MEMORY)) {
+ return getMemoryAsBytes(memString);
+ } else if (property.getType().equals(PropertyType.BYTES)) {
+ return getBytes(memString);
+ } else {
+ throw new IllegalArgumentException(property.getKey() + " is not of BYTES or MEMORY type");
+ }
}
/**
- * Interprets a string specifying a memory size. A memory size is specified as a long integer followed by an optional B (bytes), K (KB), M (MB), or G (GB).
+ * Interprets a string specifying bytes. A bytes type is specified as a long integer followed by an optional B (bytes), K (KB), M (MB), or G (GB).
*
* @param str
- * string value
- * @return interpreted memory size
+ * String value
+ * @return interpreted memory size in bytes
*/
- static public long getMemoryInBytes(String str) {
+ static public long getBytes(String str) {
--- End diff --
getBytes is too common of a method name (Example `String#getBytes()`). The type is still for bytes of memory, so there's no reason to change that. Keeping the old name makes the changeset much smaller. The new one could be something more like: `getMemoryRatioAsBytes()`, and could be parsed as either a ratio or fall back to bytes (if not ending with '%').
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107920669
--- Diff: assemble/bin/accumulo ---
@@ -80,9 +65,6 @@ function main() {
JAVA=($ACCUMULO_JAVA_PREFIX $JAVA)
--- End diff --
I am ok with splitting on whitespace. i tested this current code with numactl commands and it worked fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107690518
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/Property.java ---
@@ -253,13 +252,13 @@
+ " max number of concurrent writer when configuring. When using Hadoop 2, Accumulo will call hsync() on the WAL . For a small number of "
+ "concurrent writers, increasing this buffer size decreases the frequncy of hsync calls. For a large number of concurrent writers a small buffers "
+ "size is ok because of group commit."),
- TSERV_TOTAL_MUTATION_QUEUE_MAX("tserver.total.mutation.queue.max", "50M", PropertyType.MEMORY,
+ TSERV_TOTAL_MUTATION_QUEUE_MAX("tserver.total.mutation.queue.max", "5%", PropertyType.MEMORY,
"The amount of memory used to store write-ahead-log mutations before flushing them."),
TSERV_TABLET_SPLIT_FINDMIDPOINT_MAXOPEN("tserver.tablet.split.midpoint.files.max", "300", PropertyType.COUNT,
"To find a tablets split points, all index files are opened. This setting determines how many index "
+ "files can be opened at once. When there are more index files than this setting multiple passes "
+ "must be made, which is slower. However opening too many files at once can cause problems."),
- TSERV_WALOG_MAX_SIZE("tserver.walog.max.size", "1G", PropertyType.MEMORY,
+ TSERV_WALOG_MAX_SIZE("tserver.walog.max.size", "33%", PropertyType.MEMORY,
--- End diff --
Ok. I will change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107685917
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/PropertyType.java ---
@@ -45,8 +45,8 @@
+ "Examples of invalid durations are '1w', '1h30m', '1s 200ms', 'ms', '', and 'a'.\n"
+ "Unless otherwise stated, the max value for the duration represented in milliseconds is " + Long.MAX_VALUE),
- MEMORY("memory", boundedUnits(0, Long.MAX_VALUE, false, "", "B", "K", "M", "G"),
- "A positive integer optionally followed by a unit of memory (whitespace disallowed), as in 2G.\n"
+ MEMORY("memory", boundedUnits(0, Long.MAX_VALUE, false, "", "B", "K", "M", "G", "%"),
+ "A positive integer optionally followed by a unit of memory or percentage (whitespace disallowed), as in 2G.\n"
--- End diff --
Need to describe behavior of percentage.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #235: ACCUMULO-4612 - Simplify Accumulo memory configuration
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on the issue:
https://github.com/apache/accumulo/pull/235
I would like to merge today if there are no objections.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107939700
--- Diff: INSTALL.md ---
@@ -36,62 +36,49 @@ to manage Accumulo:
These scripts will be used in the remaining instructions to configure and run Accumulo.
For convenience, consider adding `accumulo-X.Y.Z/bin/` to your shell's path.
-## Configuring
+## Configuring Accumulo
-Accumulo has some optional native code that improves its performance and
-stability. Before configuring Accumulo, attempt to build this native code
-with the following command.
+Accumulo requires running [Zookeeper][3] and [HDFS][4] instances which should be set up
+before configuring Accumulo.
- accumulo-util build-native
+The primary configuration files for Accumulo are `accumulo-env.sh` and `accumulo-site.xml`
+which are located in the `conf/` directory.
-If the command fails, its OK to continue with setup and resolve the issue later.
+Follow the steps below to configure `accumulo-site.xml`:
-Accumulo is configured by the files `accumulo-site.xml` and `accumulo-env.sh` in the `conf/`
-directory. You can either edit these files for your environment or run the command below which will
-overwrite them with files configured for your environment.
+1. Run `accumulo-util build-native` to build native code. If this command fails, disable
+ native maps by setting `tserver.memory.maps.native.enabled` to `false`.
- accumulo-util create-config
+2. Set `instance.volumes` to HDFS location where Accumulo will store data. If your namenode
+ is running at 192.168.1.9:9000 and you want to store data in `/accumulo` in HDFS, then set
--- End diff --
If you did the s/9000/8020/ elsewhere, could also do it here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107944635
--- Diff: core/src/test/java/org/apache/accumulo/core/conf/AccumuloConfigurationTest.java ---
@@ -49,12 +50,12 @@ public void testGetTimeInMillis() {
@Test(expected = IllegalArgumentException.class)
public void testGetMemoryInBytesFailureCases1() throws Exception {
- AccumuloConfiguration.getMemoryInBytes("42x");
+ AccumuloConfiguration.getFixedMemoryAsBytes("42x");
--- End diff --
Could duplicate these two failure test for getMemoryAsBytes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107721682
--- Diff: docs/src/main/asciidoc/chapters/administration.txt ---
@@ -404,6 +399,51 @@ and BatchScanner passing in the name of the context, app1 in the example above.
iterators to load classes from the locations defined by the context. Passing the context name to the scanners allows you to override the table setting
to load only scan time iterators from a different location.
+=== Vendor-specific configuration
--- End diff --
> If vendors release new versions and these docs need to be updated, the user manual can be updated in the Accumulo repo and pushed to the website without creating a new bugfix release.
That addresses part of my concern. Still does not allow linking to the documentation from accumulo-env.sh. However, the inability to link between different kinds of Accumulo documentation is a more general issue that should not be addresses here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107684856
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/Property.java ---
@@ -170,10 +170,9 @@
// general properties
GENERAL_PREFIX("general.", null, PropertyType.PREFIX,
"Properties in this category affect the behavior of accumulo overall, but do not have to be consistent throughout a cloud."),
- GENERAL_CLASSPATHS(AccumuloClassLoader.CLASSPATH_PROPERTY_NAME, AccumuloClassLoader.ACCUMULO_CLASSPATH_VALUE, PropertyType.STRING,
+ GENERAL_CLASSPATHS(AccumuloClassLoader.CLASSPATH_PROPERTY_NAME, "", PropertyType.STRING,
--- End diff --
Should we deprecate this property?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107759896
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java ---
@@ -209,13 +209,19 @@ static public long getMemoryInBytes(String str) {
case 'B':
multiplier = 0;
break;
+ case '%':
--- End diff --
I pushed another commit that creates a `BYTES` type.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107944404
--- Diff: core/src/test/java/org/apache/accumulo/core/conf/AccumuloConfigurationTest.java ---
@@ -25,16 +25,17 @@
@Test
public void testGetMemoryInBytes() throws Exception {
- assertEquals(42l, AccumuloConfiguration.getMemoryInBytes("42"));
- assertEquals(42l, AccumuloConfiguration.getMemoryInBytes("42b"));
- assertEquals(42l, AccumuloConfiguration.getMemoryInBytes("42B"));
- assertEquals(42l * 1024l, AccumuloConfiguration.getMemoryInBytes("42K"));
- assertEquals(42l * 1024l, AccumuloConfiguration.getMemoryInBytes("42k"));
- assertEquals(42l * 1024l * 1024l, AccumuloConfiguration.getMemoryInBytes("42M"));
- assertEquals(42l * 1024l * 1024l, AccumuloConfiguration.getMemoryInBytes("42m"));
- assertEquals(42l * 1024l * 1024l * 1024l, AccumuloConfiguration.getMemoryInBytes("42G"));
- assertEquals(42l * 1024l * 1024l * 1024l, AccumuloConfiguration.getMemoryInBytes("42g"));
-
+ assertEquals(42l, AccumuloConfiguration.getFixedMemoryAsBytes("42"));
--- End diff --
Could apply all of these fixed memory string test to both functions. The following is one way to do this in Java 8 I was playing around with. Is there a shorter way to create the list of functions?
```java
List<Function<String,Long>> funcs =
Arrays.asList(AccumuloConfiguration::getFixedMemoryAsBytes, AccumuloConfiguration::getMemoryAsBytes);
for(Function<String,Long> memFunc : funcs) {
assertEquals(42l, memFunc.apply("42"));
//rest of asserts
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #235: ACCUMULO-4612 - Simplify Accumulo memory configuration
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on the issue:
https://github.com/apache/accumulo/pull/235
With the percentages essentially being transformed into discrete values, our same sanity checks should be in place that would prevent users from providing configuration that exceeds the heap size?
I think it's just the one: `'-Xmx' >= (block caches) + (non-native-maps)`? Do we need to update that check? Does anyone remember where that check is? :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107682883
--- Diff: docs/src/main/asciidoc/chapters/administration.txt ---
@@ -404,6 +399,51 @@ and BatchScanner passing in the name of the context, app1 in the example above.
iterators to load classes from the locations defined by the context. Passing the context name to the scanners allows you to override the table setting
to load only scan time iterators from a different location.
+=== Vendor-specific configuration
--- End diff --
I think this info would better on the website, where it can be easily updated and appended to.
We could create the web page now w/o any links to it on the website .. and a disclaimer about the info being for 2.0.0-SNAP. Then 2.0.0-SNAP docs could point to it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by ctubbsii <gi...@git.apache.org>.
Github user ctubbsii commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107797419
--- Diff: assemble/conf/accumulo-site.xml ---
@@ -0,0 +1,51 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!-- This is the main configuration file for Apache Accumulo. Available configuration properties
+ (and their default values) can be found in the user manual (docs/accumulo_user_manual.html). -->
+<configuration>
+ <!-- Set location in HDFS where Accumulo will store data -->
+ <property>
+ <name>instance.volumes</name>
+ <value>hdfs://localhost:9000/accumulo</value>
--- End diff --
Should probably use Hadoop's default port here instead of `9000`. I think it's `8192`, but I can't remember right now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107720701
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java ---
@@ -209,13 +209,19 @@ static public long getMemoryInBytes(String str) {
case 'B':
multiplier = 0;
break;
+ case '%':
--- End diff --
Could have a new type `BYTES` which has everything but the `%`. And `MEMORY` could be what `BYTES` supports plus `%`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by ctubbsii <gi...@git.apache.org>.
Github user ctubbsii commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107804432
--- Diff: docs/src/main/asciidoc/chapters/administration.txt ---
@@ -404,6 +399,51 @@ and BatchScanner passing in the name of the context, app1 in the example above.
iterators to load classes from the locations defined by the context. Passing the context name to the scanners allows you to override the table setting
to load only scan time iterators from a different location.
+=== Vendor-specific configuration
--- End diff --
The user manual is tied to a version of Accumulo. It cannot be updated over time between releases. The copy on the website can, but the "1.8.0" user manual is fixed at a point in time and is distributed with the tarball.
I agree with Keith it's better to put this on the website, outside the user manual. The user manual should have generic information applicable to everybody to keep it succinct. Specific documentation for a particular environment can be discussed in a blog, which refers to the appropriate section of the user manual.
In other words:
usermanual: "Section X: You should configure your CLASSPATH by doing Y"
blog: "You can set your class path for HDP to ..., according to section X" (or similar)
This also makes more sense from a versioning perspective. The classpath for HDP might not change, but the instructions in "Section X" (in my example) might be different. Keeping them separate like this keeps them both relevant longer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107680034
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/Property.java ---
@@ -253,13 +252,13 @@
+ " max number of concurrent writer when configuring. When using Hadoop 2, Accumulo will call hsync() on the WAL . For a small number of "
+ "concurrent writers, increasing this buffer size decreases the frequncy of hsync calls. For a large number of concurrent writers a small buffers "
+ "size is ok because of group commit."),
- TSERV_TOTAL_MUTATION_QUEUE_MAX("tserver.total.mutation.queue.max", "50M", PropertyType.MEMORY,
+ TSERV_TOTAL_MUTATION_QUEUE_MAX("tserver.total.mutation.queue.max", "5%", PropertyType.MEMORY,
"The amount of memory used to store write-ahead-log mutations before flushing them."),
TSERV_TABLET_SPLIT_FINDMIDPOINT_MAXOPEN("tserver.tablet.split.midpoint.files.max", "300", PropertyType.COUNT,
"To find a tablets split points, all index files are opened. This setting determines how many index "
+ "files can be opened at once. When there are more index files than this setting multiple passes "
+ "must be made, which is slower. However opening too many files at once can cause problems."),
- TSERV_WALOG_MAX_SIZE("tserver.walog.max.size", "1G", PropertyType.MEMORY,
+ TSERV_WALOG_MAX_SIZE("tserver.walog.max.size", "33%", PropertyType.MEMORY,
--- End diff --
This is a file size limit. Should probably be left at 1G
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107951460
--- Diff: core/src/test/java/org/apache/accumulo/core/conf/AccumuloConfigurationTest.java ---
@@ -49,12 +50,12 @@ public void testGetTimeInMillis() {
@Test(expected = IllegalArgumentException.class)
public void testGetMemoryInBytesFailureCases1() throws Exception {
- AccumuloConfiguration.getMemoryInBytes("42x");
+ AccumuloConfiguration.getFixedMemoryAsBytes("42x");
--- End diff --
This was added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107721029
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java ---
@@ -209,13 +209,19 @@ static public long getMemoryInBytes(String str) {
case 'B':
multiplier = 0;
break;
+ case '%':
--- End diff --
I would rather not create another `PropertyType`. If the defaults are fixed memory, users probably won't switch to percentage. If you are worried about certain properties, I could add some text to the description of those properties that fixed memory amounts should be used. Even if we limit percentage, users can still misconfigure things by setting fixed memory values that are too high or low for that property.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by ctubbsii <gi...@git.apache.org>.
Github user ctubbsii commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107797066
--- Diff: assemble/bin/accumulo ---
@@ -80,9 +65,6 @@ function main() {
JAVA=($ACCUMULO_JAVA_PREFIX $JAVA)
--- End diff --
Should this be resolved as `JAVA=("${ACCUMULO_JAVA_PREFIX[@]}" "$JAVA")` ? Or are we okay with just splitting on whitespace for both of these variables?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107727382
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java ---
@@ -209,13 +209,19 @@ static public long getMemoryInBytes(String str) {
case 'B':
multiplier = 0;
break;
+ case '%':
--- End diff --
> If you are worried about certain properties, I could add some text to the description of those properties that fixed memory amounts should be used
That would also assuage my fears. I am not worried about the general misconfiguration -- just users trying to set percentages where it makes no sense.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107686386
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java ---
@@ -209,13 +209,19 @@ static public long getMemoryInBytes(String str) {
case 'B':
multiplier = 0;
break;
+ case '%':
+ int percent = Integer.parseInt(str.substring(0, str.length() - 1));
+ if (percent <= 0 || percent >= 100) {
+ throw new IllegalArgumentException("The value of '" + str + "' is not a valid memory percentage setting.");
+ }
+ return Runtime.getRuntime().maxMemory() * percent / 100;
default:
return Long.parseLong(str);
}
return Long.parseLong(str.substring(0, str.length() - 1)) << multiplier;
} catch (Exception ex) {
throw new IllegalArgumentException("The value '" + str + "' is not a valid memory setting. A valid value would a number "
- + "possibily followed by an optional 'G', 'M', 'K', or 'B'.");
+ + "possibly followed by an optional 'G', 'M', 'K', or 'B'.");
--- End diff --
Since this is catching exception, it possible that something could go wrong parsing a `%`. So error message should include `%` as valid suffix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107805661
--- Diff: assemble/conf/accumulo-site.xml ---
@@ -0,0 +1,51 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!-- This is the main configuration file for Apache Accumulo. Available configuration properties
+ (and their default values) can be found in the user manual (docs/accumulo_user_manual.html). -->
+<configuration>
+ <!-- Set location in HDFS where Accumulo will store data -->
+ <property>
+ <name>instance.volumes</name>
+ <value>hdfs://localhost:9000/accumulo</value>
--- End diff --
8020
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107713794
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java ---
@@ -209,13 +209,19 @@ static public long getMemoryInBytes(String str) {
case 'B':
multiplier = 0;
break;
+ case '%':
--- End diff --
I'm wondering if it would make more sense to create a new `PropertyType` instead of piggy-backing on `MEMORY`. Is it confusing to allow percent-values for some of these other properties which we really never expect users to do?
For example, some properties like `table.file.blocksize` or `tserver.compaction.major.throughput` make no sense to allow a percentage of total heap.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #235: ACCUMULO-4612 - Simplify Accumulo memory configuration
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on the issue:
https://github.com/apache/accumulo/pull/235
> Does anyone remember where that check is?
I think the check is in `TabletServerResourceManager` constructor. I think the check will be ok as it uses the resolved memory values, but not completely sure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107717380
--- Diff: INSTALL.md ---
@@ -36,62 +36,49 @@ to manage Accumulo:
These scripts will be used in the remaining instructions to configure and run Accumulo.
For convenience, consider adding `accumulo-X.Y.Z/bin/` to your shell's path.
-## Configuring
+## Configuring Accumulo
-Accumulo has some optional native code that improves its performance and
-stability. Before configuring Accumulo, attempt to build this native code
-with the following command.
+Accumulo requires running [Zookeeper][3] and [HDFS][4] instances which should be set up
+before configuring Accumulo.
- accumulo-util build-native
+The primary configuration files for Accumulo are `accumulo-env.sh` and `accumulo-site.xml`
+which are located in the `conf/` directory.
-If the command fails, its OK to continue with setup and resolve the issue later.
+Follow the steps below to configure `accumulo-site.xml`:
-Accumulo is configured by the files `accumulo-site.xml` and `accumulo-env.sh` in the `conf/`
-directory. You can either edit these files for your environment or run the command below which will
-overwrite them with files configured for your environment.
+1. Run `accumulo-util build-native` to build native code. If this command fails, disable
+ native maps by setting `tserver.memory.maps.native.enabled` to `false`.
- accumulo-util create-config
+2. Set `instance.volumes` to HDFS location where Accumulo will store data. If your namenode
+ is running at 192.168.1.9:9000 and you want to store data in `/accumulo` in HDFS, then set
+ `instance.volumes` to `hdfs://192.168.1.9:9000/accumulo`.
-The script will ask you questions about your set up. Below are some suggestions:
+3. Set `instance.zookeeper.host` to the location of your Zookeepers
-* When the script asks about memory-map type, choose Native if the build native script
- was successful. Otherwise, choose Java.
-* The script will prompt for memory usage. Please note that the footprints are
- only for the Accumulo system processes, so ample space should be left for other
- processes like Hadoop, Zookeeper, and the Accumulo client code. If Accumulo
- worker processes are swapped out and unresponsive, they may be killed.
+4. (Optional) Change `instance.secret` (which is used by Accumulo processes to communicate)
+ from the default. This value should match on all servers.
-While `accumulo-util create-config` creates `accumulo-env.sh` and `accumulo-site.xml` files
-targeted for your environment, these files still require a few more edits before starting Accumulo.
+Follow the steps below to configure `accumulo-env.sh`:
-### Secret
+1. Set `HADOOP_PREFIX` and `ZOOKEEPER_HOME` to help Accumulo locate Hadoop and Zookeeper jars
+ and add them to the `CLASSPATH` variable. If you are running a vendor-specific release of Hadoop
+ or Zookeeper, see the `Vendor-specific configuration` documentation in the Administration section
+ of the Accumulo user manual as you may need to change how your `CLASSPATH` is built. If Accumulo
+ has problems later on finding jars, run `accumulo classpath -d` to print Accumulo's classpath.
-Accumulo coordination and worker processes can only communicate with each other
-if they share the same secret key. To change the secret key set
-`instance.secret` in `accumulo-site.xml`. Changing this secret key from
-the default is highly recommended.
+2. Accumulo tablet servers are configured by default to use 1GB of memory (768MB is allocated to
+ JVM and 256MB is allocated for native maps). Native maps are allocated memory equal to 33% of
+ the tserver JVM heap. The table below can be used if you would like to change tsever memory
+ usage in the `JAVA_OPTS` section of `accumulo-env.sh`:
-### Dependencies
+ | Native? | 512MB | 1GB | 2GB | 3GB |
+ |---------|-------------------|-------------------|---------------------|---------------|
+ | Yes | -Xmx384m -Xms384m | -Xmx768m -Xms768m | -Xmx1536m -Xms1536m | -Xmx2g -Xms2g |
+ | No | -Xmx512m -Xms512m | -Xmx1g -Xms1g | -Xmx2g -Xms2g | -Xmx3g -Xms3g |
-Accumulo requires running [Zookeeper][3] and [HDFS][4] instances. Also, the
-Accumulo binary distribution does not include jars for Zookeeper and Hadoop.
-When configuring Accumulo the following information about these dependencies
-must be provided.
-
- * **Location of Zookeepers** : Provide this by setting `instance.zookeeper.host`
- in `accumulo-site.xml`.
- * **Where to store data** : Provide this by setting `instance.volumes` in
- `accumulo-site.xml`. If your namenode is running at 192.168.1.9:9000
- and you want to store data in `/accumulo` in HDFS, then set
- `instance.volumes` to `hdfs://192.168.1.9:9000/accumulo`.
- * **Location of Zookeeper and Hadoop jars** : Setting `ZOOKEEPER_HOME` and
- `HADOOP_PREFIX` in `accumulo-env.sh` will help Accumulo find these jars
- when using the default setting for `general.classpaths` in accumulo-site.xml.
-
-If Accumulo has problems later on finding jars, then run `bin/accumulo
-classpath` to print out info about where Accumulo is finding jars. If the
-settings mentioned above are correct, then inspect `general.classpaths` in
-`accumulo-site.xml`.
+3. (Optional) The Accumulo master is configured by default to use 512MB while the garbage collector
--- End diff --
Should be fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #235: ACCUMULO-4612 - Simplify Accumulo memory configuration
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on the issue:
https://github.com/apache/accumulo/pull/235
> I think the check is in TabletServerResourceManager constructor. I think the check will be ok as it uses the resolved memory values, but not completely sure.
Your memory is much better than mine.
https://github.com/apache/accumulo/blob/94cdcc4d3f0a8ccf95894f206cb71e6117f4e51d/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServerResourceManager.java#L192-L202
I would agree that I think this will be fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107684327
--- Diff: assemble/conf/accumulo-env.sh ---
@@ -36,6 +36,22 @@ export HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-${HADOOP_PREFIX}/etc/hadoop}"
## Zookeeper installation
export ZOOKEEPER_HOME="${ZOOKEEPER_HOME:-/path/to/zookeeper}"
+##########################
+# Build CLASSPATH variable
+##########################
+
+## Adds external Hadoop & Zookeeper dependencies to CLASSPATH. See "Vendor configuration" section of Accumulo user manual
+## for different settings if you installed vendor's distribution of Hadoop or Zookeeper.
--- End diff --
If we put info about vendor specific configs on website, then we could put a link to that info here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107951433
--- Diff: core/src/test/java/org/apache/accumulo/core/conf/AccumuloConfigurationTest.java ---
@@ -25,16 +25,17 @@
@Test
public void testGetMemoryInBytes() throws Exception {
- assertEquals(42l, AccumuloConfiguration.getMemoryInBytes("42"));
- assertEquals(42l, AccumuloConfiguration.getMemoryInBytes("42b"));
- assertEquals(42l, AccumuloConfiguration.getMemoryInBytes("42B"));
- assertEquals(42l * 1024l, AccumuloConfiguration.getMemoryInBytes("42K"));
- assertEquals(42l * 1024l, AccumuloConfiguration.getMemoryInBytes("42k"));
- assertEquals(42l * 1024l * 1024l, AccumuloConfiguration.getMemoryInBytes("42M"));
- assertEquals(42l * 1024l * 1024l, AccumuloConfiguration.getMemoryInBytes("42m"));
- assertEquals(42l * 1024l * 1024l * 1024l, AccumuloConfiguration.getMemoryInBytes("42G"));
- assertEquals(42l * 1024l * 1024l * 1024l, AccumuloConfiguration.getMemoryInBytes("42g"));
-
+ assertEquals(42l, AccumuloConfiguration.getFixedMemoryAsBytes("42"));
--- End diff --
This was added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by ctubbsii <gi...@git.apache.org>.
Github user ctubbsii commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107796164
--- Diff: INSTALL.md ---
@@ -36,62 +36,48 @@ to manage Accumulo:
These scripts will be used in the remaining instructions to configure and run Accumulo.
For convenience, consider adding `accumulo-X.Y.Z/bin/` to your shell's path.
-## Configuring
+## Configuring Accumulo
-Accumulo has some optional native code that improves its performance and
-stability. Before configuring Accumulo, attempt to build this native code
-with the following command.
+Accumulo requires running [Zookeeper][3] and [HDFS][4] instances which should be set up
--- End diff --
sp: s/Zookeeper/ZooKeeper/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107933477
--- Diff: docs/src/main/asciidoc/chapters/administration.txt ---
@@ -404,6 +399,51 @@ and BatchScanner passing in the name of the context, app1 in the example above.
iterators to load classes from the locations defined by the context. Passing the context name to the scanners allows you to override the table setting
to load only scan time iterators from a different location.
+=== Vendor-specific configuration
--- End diff --
I pushed another commit that removes the vendor-specific documentation from the user manual.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by ctubbsii <gi...@git.apache.org>.
Github user ctubbsii commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107811132
--- Diff: assemble/conf/accumulo-site.xml ---
@@ -0,0 +1,51 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!-- This is the main configuration file for Apache Accumulo. Available configuration properties
+ (and their default values) can be found in the user manual (docs/accumulo_user_manual.html). -->
+<configuration>
+ <!-- Set location in HDFS where Accumulo will store data -->
+ <property>
+ <name>instance.volumes</name>
+ <value>hdfs://localhost:9000/accumulo</value>
--- End diff --
Thanks @joshelser :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/accumulo/pull/235
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107683551
--- Diff: INSTALL.md ---
@@ -36,62 +36,49 @@ to manage Accumulo:
These scripts will be used in the remaining instructions to configure and run Accumulo.
For convenience, consider adding `accumulo-X.Y.Z/bin/` to your shell's path.
-## Configuring
+## Configuring Accumulo
-Accumulo has some optional native code that improves its performance and
-stability. Before configuring Accumulo, attempt to build this native code
-with the following command.
+Accumulo requires running [Zookeeper][3] and [HDFS][4] instances which should be set up
+before configuring Accumulo.
- accumulo-util build-native
+The primary configuration files for Accumulo are `accumulo-env.sh` and `accumulo-site.xml`
+which are located in the `conf/` directory.
-If the command fails, its OK to continue with setup and resolve the issue later.
+Follow the steps below to configure `accumulo-site.xml`:
-Accumulo is configured by the files `accumulo-site.xml` and `accumulo-env.sh` in the `conf/`
-directory. You can either edit these files for your environment or run the command below which will
-overwrite them with files configured for your environment.
+1. Run `accumulo-util build-native` to build native code. If this command fails, disable
+ native maps by setting `tserver.memory.maps.native.enabled` to `false`.
- accumulo-util create-config
+2. Set `instance.volumes` to HDFS location where Accumulo will store data. If your namenode
+ is running at 192.168.1.9:9000 and you want to store data in `/accumulo` in HDFS, then set
+ `instance.volumes` to `hdfs://192.168.1.9:9000/accumulo`.
-The script will ask you questions about your set up. Below are some suggestions:
+3. Set `instance.zookeeper.host` to the location of your Zookeepers
-* When the script asks about memory-map type, choose Native if the build native script
- was successful. Otherwise, choose Java.
-* The script will prompt for memory usage. Please note that the footprints are
- only for the Accumulo system processes, so ample space should be left for other
- processes like Hadoop, Zookeeper, and the Accumulo client code. If Accumulo
- worker processes are swapped out and unresponsive, they may be killed.
+4. (Optional) Change `instance.secret` (which is used by Accumulo processes to communicate)
+ from the default. This value should match on all servers.
-While `accumulo-util create-config` creates `accumulo-env.sh` and `accumulo-site.xml` files
-targeted for your environment, these files still require a few more edits before starting Accumulo.
+Follow the steps below to configure `accumulo-env.sh`:
-### Secret
+1. Set `HADOOP_PREFIX` and `ZOOKEEPER_HOME` to help Accumulo locate Hadoop and Zookeeper jars
+ and add them to the `CLASSPATH` variable. If you are running a vendor-specific release of Hadoop
+ or Zookeeper, see the `Vendor-specific configuration` documentation in the Administration section
+ of the Accumulo user manual as you may need to change how your `CLASSPATH` is built. If Accumulo
+ has problems later on finding jars, run `accumulo classpath -d` to print Accumulo's classpath.
-Accumulo coordination and worker processes can only communicate with each other
-if they share the same secret key. To change the secret key set
-`instance.secret` in `accumulo-site.xml`. Changing this secret key from
-the default is highly recommended.
+2. Accumulo tablet servers are configured by default to use 1GB of memory (768MB is allocated to
+ JVM and 256MB is allocated for native maps). Native maps are allocated memory equal to 33% of
+ the tserver JVM heap. The table below can be used if you would like to change tsever memory
+ usage in the `JAVA_OPTS` section of `accumulo-env.sh`:
-### Dependencies
+ | Native? | 512MB | 1GB | 2GB | 3GB |
+ |---------|-------------------|-------------------|---------------------|---------------|
+ | Yes | -Xmx384m -Xms384m | -Xmx768m -Xms768m | -Xmx1536m -Xms1536m | -Xmx2g -Xms2g |
+ | No | -Xmx512m -Xms512m | -Xmx1g -Xms1g | -Xmx2g -Xms2g | -Xmx3g -Xms3g |
-Accumulo requires running [Zookeeper][3] and [HDFS][4] instances. Also, the
-Accumulo binary distribution does not include jars for Zookeeper and Hadoop.
-When configuring Accumulo the following information about these dependencies
-must be provided.
-
- * **Location of Zookeepers** : Provide this by setting `instance.zookeeper.host`
- in `accumulo-site.xml`.
- * **Where to store data** : Provide this by setting `instance.volumes` in
- `accumulo-site.xml`. If your namenode is running at 192.168.1.9:9000
- and you want to store data in `/accumulo` in HDFS, then set
- `instance.volumes` to `hdfs://192.168.1.9:9000/accumulo`.
- * **Location of Zookeeper and Hadoop jars** : Setting `ZOOKEEPER_HOME` and
- `HADOOP_PREFIX` in `accumulo-env.sh` will help Accumulo find these jars
- when using the default setting for `general.classpaths` in accumulo-site.xml.
-
-If Accumulo has problems later on finding jars, then run `bin/accumulo
-classpath` to print out info about where Accumulo is finding jars. If the
-settings mentioned above are correct, then inspect `general.classpaths` in
-`accumulo-site.xml`.
+3. (Optional) The Accumulo master is configured by default to use 512MB while the garbage collector
+ and monitor is configure to use 256MB. These settings can be changed in the `JAVA_OPTS` section
--- End diff --
`monitor is configure to use`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107687845
--- Diff: docs/src/main/asciidoc/chapters/administration.txt ---
@@ -364,17 +357,19 @@ consideration. There is no enforcement of these warnings via the API.
==== Configuring the ClassLoader
-Accumulo loads classes from the locations specified in the +general.classpaths+ property. Additionally, Accumulo will load classes
-from the locations specified in the +general.dynamic.classpaths+ property and will monitor and reload them if they change. The reloading
-feature is useful during the development and testing of iterators as new or modified iterator classes can be deployed to Accumulo without
-having to restart the database.
+Accumulo builds its Java classpath in +accumulo-env.sh+. After Accumulo application have started, they will loads classes from the locations
--- End diff --
`they will loads classes`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107690464
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/Property.java ---
@@ -170,10 +170,9 @@
// general properties
GENERAL_PREFIX("general.", null, PropertyType.PREFIX,
"Properties in this category affect the behavior of accumulo overall, but do not have to be consistent throughout a cloud."),
- GENERAL_CLASSPATHS(AccumuloClassLoader.CLASSPATH_PROPERTY_NAME, AccumuloClassLoader.ACCUMULO_CLASSPATH_VALUE, PropertyType.STRING,
+ GENERAL_CLASSPATHS(AccumuloClassLoader.CLASSPATH_PROPERTY_NAME, "", PropertyType.STRING,
--- End diff --
I think it should be deprecated. I think users should avoid setting up the classpath in accumulo-site.xml.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by ctubbsii <gi...@git.apache.org>.
Github user ctubbsii commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107798096
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java ---
@@ -209,13 +209,19 @@ static public long getMemoryInBytes(String str) {
case 'B':
multiplier = 0;
break;
+ case '%':
+ int percent = Integer.parseInt(str.substring(0, str.length() - 1));
+ if (percent <= 0 || percent >= 100) {
+ throw new IllegalArgumentException("The value of '" + str + "' is not a valid memory percentage setting.");
+ }
+ return Runtime.getRuntime().maxMemory() * percent / 100;
default:
return Long.parseLong(str);
}
return Long.parseLong(str.substring(0, str.length() - 1)) << multiplier;
} catch (Exception ex) {
throw new IllegalArgumentException("The value '" + str + "' is not a valid memory setting. A valid value would a number "
- + "possibily followed by an optional 'G', 'M', 'K', or 'B'.");
+ + "possibly followed by an optional 'G', 'M', 'K', or 'B'.");
--- End diff --
Also keep in mind, that we do have a PERCENTAGE type already, with an appropriate error message. I realize this new type is a bit more narrow than the existing PERCENTAGE type, though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107684054
--- Diff: INSTALL.md ---
@@ -36,62 +36,49 @@ to manage Accumulo:
These scripts will be used in the remaining instructions to configure and run Accumulo.
For convenience, consider adding `accumulo-X.Y.Z/bin/` to your shell's path.
-## Configuring
+## Configuring Accumulo
-Accumulo has some optional native code that improves its performance and
-stability. Before configuring Accumulo, attempt to build this native code
-with the following command.
+Accumulo requires running [Zookeeper][3] and [HDFS][4] instances which should be set up
+before configuring Accumulo.
- accumulo-util build-native
+The primary configuration files for Accumulo are `accumulo-env.sh` and `accumulo-site.xml`
+which are located in the `conf/` directory.
-If the command fails, its OK to continue with setup and resolve the issue later.
+Follow the steps below to configure `accumulo-site.xml`:
-Accumulo is configured by the files `accumulo-site.xml` and `accumulo-env.sh` in the `conf/`
-directory. You can either edit these files for your environment or run the command below which will
-overwrite them with files configured for your environment.
+1. Run `accumulo-util build-native` to build native code. If this command fails, disable
+ native maps by setting `tserver.memory.maps.native.enabled` to `false`.
- accumulo-util create-config
+2. Set `instance.volumes` to HDFS location where Accumulo will store data. If your namenode
+ is running at 192.168.1.9:9000 and you want to store data in `/accumulo` in HDFS, then set
+ `instance.volumes` to `hdfs://192.168.1.9:9000/accumulo`.
-The script will ask you questions about your set up. Below are some suggestions:
+3. Set `instance.zookeeper.host` to the location of your Zookeepers
-* When the script asks about memory-map type, choose Native if the build native script
- was successful. Otherwise, choose Java.
-* The script will prompt for memory usage. Please note that the footprints are
- only for the Accumulo system processes, so ample space should be left for other
- processes like Hadoop, Zookeeper, and the Accumulo client code. If Accumulo
- worker processes are swapped out and unresponsive, they may be killed.
+4. (Optional) Change `instance.secret` (which is used by Accumulo processes to communicate)
+ from the default. This value should match on all servers.
-While `accumulo-util create-config` creates `accumulo-env.sh` and `accumulo-site.xml` files
-targeted for your environment, these files still require a few more edits before starting Accumulo.
+Follow the steps below to configure `accumulo-env.sh`:
-### Secret
+1. Set `HADOOP_PREFIX` and `ZOOKEEPER_HOME` to help Accumulo locate Hadoop and Zookeeper jars
+ and add them to the `CLASSPATH` variable. If you are running a vendor-specific release of Hadoop
+ or Zookeeper, see the `Vendor-specific configuration` documentation in the Administration section
+ of the Accumulo user manual as you may need to change how your `CLASSPATH` is built. If Accumulo
+ has problems later on finding jars, run `accumulo classpath -d` to print Accumulo's classpath.
-Accumulo coordination and worker processes can only communicate with each other
-if they share the same secret key. To change the secret key set
-`instance.secret` in `accumulo-site.xml`. Changing this secret key from
-the default is highly recommended.
+2. Accumulo tablet servers are configured by default to use 1GB of memory (768MB is allocated to
+ JVM and 256MB is allocated for native maps). Native maps are allocated memory equal to 33% of
+ the tserver JVM heap. The table below can be used if you would like to change tsever memory
+ usage in the `JAVA_OPTS` section of `accumulo-env.sh`:
-### Dependencies
+ | Native? | 512MB | 1GB | 2GB | 3GB |
+ |---------|-------------------|-------------------|---------------------|---------------|
+ | Yes | -Xmx384m -Xms384m | -Xmx768m -Xms768m | -Xmx1536m -Xms1536m | -Xmx2g -Xms2g |
+ | No | -Xmx512m -Xms512m | -Xmx1g -Xms1g | -Xmx2g -Xms2g | -Xmx3g -Xms3g |
-Accumulo requires running [Zookeeper][3] and [HDFS][4] instances. Also, the
-Accumulo binary distribution does not include jars for Zookeeper and Hadoop.
-When configuring Accumulo the following information about these dependencies
-must be provided.
-
- * **Location of Zookeepers** : Provide this by setting `instance.zookeeper.host`
- in `accumulo-site.xml`.
- * **Where to store data** : Provide this by setting `instance.volumes` in
- `accumulo-site.xml`. If your namenode is running at 192.168.1.9:9000
- and you want to store data in `/accumulo` in HDFS, then set
- `instance.volumes` to `hdfs://192.168.1.9:9000/accumulo`.
- * **Location of Zookeeper and Hadoop jars** : Setting `ZOOKEEPER_HOME` and
- `HADOOP_PREFIX` in `accumulo-env.sh` will help Accumulo find these jars
- when using the default setting for `general.classpaths` in accumulo-site.xml.
-
-If Accumulo has problems later on finding jars, then run `bin/accumulo
-classpath` to print out info about where Accumulo is finding jars. If the
-settings mentioned above are correct, then inspect `general.classpaths` in
-`accumulo-site.xml`.
+3. (Optional) The Accumulo master is configured by default to use 512MB while the garbage collector
--- End diff --
Can you word this differently so that the exact memory amounts are not specified here? If those defaults are changed in the env file, then the person making the change would need to know to make the changes here. I think its better to not duplicate the info here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107715725
--- Diff: docs/src/main/asciidoc/chapters/administration.txt ---
@@ -404,6 +399,51 @@ and BatchScanner passing in the name of the context, app1 in the example above.
iterators to load classes from the locations defined by the context. Passing the context name to the scanners allows you to override the table setting
to load only scan time iterators from a different location.
+=== Vendor-specific configuration
--- End diff --
I would prefer if it stayed in the user manual as these settings could vary between minor versions of Accumulo. If vendors release new versions and these docs need to be updated, the user manual can be updated in the Accumulo repo and pushed to the website without creating a new bugfix release.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107685017
--- Diff: docs/src/main/asciidoc/chapters/administration.txt ---
@@ -125,6 +124,12 @@ and specify the following:
. Enter the location of ZooKeeper for +$ZOOKEEPER_HOME+
. Optionally, choose a different location for Accumulo logs using +$ACCUMULO_LOG_DIR+
+Accumulo uses +HADOOP_PREFIX+ and +ZOOKEEPER_HOME+ to locate Hadoop and Zookeeper jars
+and add them the +CLASSPATH+ variable. If you are running a vendor-specific release of Hadoop
+or Zookeeper, see the `Vendor configuration` section as you may need to change how your `CLASSPATH`
--- End diff --
if info is on website, then could link to it from here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107732253
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java ---
@@ -209,13 +209,19 @@ static public long getMemoryInBytes(String str) {
case 'B':
multiplier = 0;
break;
+ case '%':
--- End diff --
> Even if we limit percentage, users can still misconfigure things by setting fixed memory values that are too high or low for that property.
Thats true. Using types just narrows the set of possible mistakes, it does not eliminate mistakes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #235: ACCUMULO-4612 - Simplify Accumulo memory configu...
Posted by ctubbsii <gi...@git.apache.org>.
Github user ctubbsii commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/235#discussion_r107797919
--- Diff: core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java ---
@@ -165,30 +165,35 @@ private void checkType(Property property, PropertyType type) {
}
/**
- * Gets a property of type {@link PropertyType#MEMORY}, interpreting the value properly.
+ * Gets a property of type {@link PropertyType#BYTES}, interpreting the value properly.
*
* @param property
- * property to get
+ * Property to get
* @return property value
* @throws IllegalArgumentException
- * if the property is of the wrong type
- * @see #getMemoryInBytes(String)
+ * if the property is of the wrong type @see #getBytes(String)
*/
- public long getMemoryInBytes(Property property) {
- checkType(property, PropertyType.MEMORY);
+ public long getAsBytes(Property property) {
+ checkType(property, PropertyType.BYTES);
String memString = get(property);
- return getMemoryInBytes(memString);
+ if (property.getType().equals(PropertyType.MEMORY)) {
--- End diff --
Can use `==` here for enum comparison. It helps with type checking, also, because `.equals` can mask type errors, because it accepts `Object`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---