You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by mw...@apache.org on 2017/02/17 19:18:29 UTC
[3/4] accumulo git commit: ACCUMULO-4528 Accumulo scripts improvements
ACCUMULO-4528 Accumulo scripts improvements
* Accumulo now determines hostname in Java
* Minimized use of environment variables
* Consolidated scripts by moving code in service.sh and cluster.sh to
accumulo-service and accumulo-cluster.
* Example host files (i.e master, tservers) are no longer included in
tarball distribution. User can create them using 'accumulo-cluster
create-config'
* Moved check-tservers from distribution to contrib/scripts in repo.
* NUMA and multiple tservers are no longer configured in scripts but
could be NUMA commands could be added using ACCUMULO_JAVA_PREFIX env
variable.
* Removed accumulo-watcher script as restarting services should not be
handled by Accumulo scripts. Script coud still be used in future
using ACCUMULO_JAVA_PREFIX.
* Made several updates to accumulo-env.sh
* Moved create-config and build-native from accumulo script to new
accumulo-util script which now includes scripts from contrib dir
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/9a3cc5c4
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/9a3cc5c4
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/9a3cc5c4
Branch: refs/heads/master
Commit: 9a3cc5c445293e8d4219cc066757a72a972804b4
Parents: 00b1535
Author: Mike Walch <mw...@apache.org>
Authored: Thu Feb 2 12:15:17 2017 -0500
Committer: Mike Walch <mw...@apache.org>
Committed: Fri Feb 17 13:47:06 2017 -0500
----------------------------------------------------------------------
INSTALL.md | 42 +-
assemble/bin/accumulo | 610 +++--------------
assemble/bin/accumulo-cluster | 300 +++++++-
assemble/bin/accumulo-service | 95 ++-
assemble/bin/accumulo-util | 686 +++++++++++++++++++
assemble/conf/examples/gc | 16 -
.../conf/examples/generic_logger.properties | 6 +-
assemble/conf/examples/generic_logger.xml | 6 +-
assemble/conf/examples/masters | 16 -
assemble/conf/examples/monitor | 16 -
.../conf/examples/templates/accumulo-env.sh | 64 ++
.../conf/examples/templates/accumulo-site.xml | 175 +++++
assemble/conf/examples/tracers | 16 -
assemble/conf/examples/tservers | 16 -
assemble/contrib/bootstrap-hdfs.sh | 91 ---
assemble/contrib/check-tservers | 199 ------
assemble/contrib/gen-monitor-cert.sh | 85 ---
assemble/contrib/tool.sh | 93 ---
assemble/libexec/accumulo-watcher.sh | 141 ----
assemble/libexec/cluster.sh | 325 ---------
assemble/libexec/load-env.sh | 155 -----
assemble/libexec/service.sh | 239 -------
assemble/libexec/templates/accumulo-env.sh | 90 ---
assemble/libexec/templates/accumulo-site.xml | 175 -----
assemble/pom.xml | 7 +-
assemble/src/main/assemblies/component.xml | 50 +-
.../main/scripts/generate-example-configs.sh | 2 +-
contrib/scripts/check-tservers.py | 199 ++++++
.../accumulo/core/conf/ConfigurationDocGen.java | 2 +-
.../org/apache/accumulo/core/conf/Property.java | 2 +-
.../main/asciidoc/chapters/administration.txt | 187 ++---
docs/src/main/asciidoc/chapters/clients.txt | 4 +-
docs/src/main/asciidoc/chapters/kerberos.txt | 14 +-
docs/src/main/asciidoc/chapters/replication.txt | 4 +-
docs/src/main/asciidoc/chapters/ssl.txt | 4 +-
.../main/asciidoc/chapters/troubleshooting.txt | 7 +-
.../apache/accumulo/server/init/Initialize.java | 2 +-
.../java/org/apache/accumulo/start/Main.java | 2 -
38 files changed, 1691 insertions(+), 2452 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/INSTALL.md
----------------------------------------------------------------------
diff --git a/INSTALL.md b/INSTALL.md
index b614991..6e924c4 100644
--- a/INSTALL.md
+++ b/INSTALL.md
@@ -25,11 +25,13 @@ source code. Unpack as follows.
tar xzf <some dir>/accumulo-X.Y.Z-bin.tar.gz
cd accumulo-X.Y.Z
-There are three scripts in the the `bin/` directory that are used to manage Accumulo:
+There are four scripts in the `bin` directory of the tarball distribution that are used
+to manage Accumulo:
1. `accumulo` - Runs Accumulo command-line tools and starts Accumulo processes
2. `accumulo-service` - Runs Accumulo processes as services
3. `accumulo-cluster` - Manages Accumulo cluster on a single node or several nodes
+4. `accumulo-util` - Accumulo utilities for creating configuration, native libraries, etc.
These scripts will be used in the remaining instructions to configure and run Accumulo.
For convenience, consider adding `accumulo-X.Y.Z/bin/` to your shell's path.
@@ -40,13 +42,13 @@ Accumulo has some optional native code that improves its performance and
stability. Before configuring Accumulo, attempt to build this native code
with the following command.
- accumulo build-native
+ accumulo-util build-native
If the command fails, its OK to continue with setup and resolve the issue later.
Run the command below to create configuration for Accumulo in `conf/`:
- accumulo create-config
+ accumulo-util create-config
The script will ask you questions about your set up. Below are some suggestions:
@@ -57,7 +59,7 @@ The script will ask you questions about your set up. Below are some suggestions:
processes like Hadoop, Zookeeper, and the Accumulo client code. If Accumulo
worker processes are swapped out and unresponsive, they may be killed.
-After the `create-config` command is run, the `conf/` directory will contain
+After the `accumulo-util create-config` command is run, the `conf/` directory will contain
`accumulo-env.sh`, `accumulo-site.xml`, and few a additional files. These files require
a few edits before starting Accumulo.
@@ -65,7 +67,7 @@ a few edits before starting Accumulo.
Accumulo coordination and worker processes can only communicate with each other
if they share the same secret key. To change the secret key set
-`instance.secret` in `conf/accumulo-site.xml`. Changing this secret key from
+`instance.secret` in `accumulo-site.xml`. Changing this secret key from
the default is highly recommended.
### Dependencies
@@ -76,19 +78,19 @@ When configuring Accumulo the following information about these dependencies
must be provided.
* **Location of Zookeepers** : Provide this by setting `instance.zookeeper.host`
- in `conf/accumulo-site.xml`.
+ in `accumulo-site.xml`.
* **Where to store data** : Provide this by setting `instance.volumes` in
- `conf/accumulo-site.xml`. If your namenode is running at 192.168.1.9:9000
+ `accumulo-site.xml`. If your namenode is running at 192.168.1.9:9000
and you want to store data in `/accumulo` in HDFS, then set
`instance.volumes` to `hdfs://192.168.1.9:9000/accumulo`.
* **Location of Zoookeeper and Hadoop jars** : Setting `ZOOKEEPER_HOME` and
- `HADOOP_PREFIX` in `conf/accumulo-env.sh` will help Accumulo find these
- jars.
+ `HADOOP_PREFIX` in `accumulo-env.sh` will help Accumulo find these jars
+ when using the default setting for `general.classpaths` in accumulo-site.xml.
If Accumulo has problems later on finding jars, then run `bin/accumulo
classpath` to print out info about where Accumulo is finding jars. If the
settings mentioned above are correct, then inspect `general.classpaths` in
-`conf/accumulo-site.xml`.
+`accumulo-site.xml`.
## Initialization
@@ -141,19 +143,19 @@ changed and the next section should be skipped.
#### Multi-node configuration
If you are running an Accumulo cluster on multiple nodes, the following files
-should be configured with a newline seperated list of node names:
+in `conf/` should be configured with a newline seperated list of node names:
- * `conf/masters` : Accumulo primary coordinating process. Must specify one
- node. Can specify a few for fault tolerance.
- * `conf/gc` : Accumulo garbage collector. Must specify one node. Can
- specify a few for fault tolerance.
- * `conf/monitor` : Node where Accumulo monitoring web server is run.
- * `conf/tservers`: Accumulo worker processes. List all of the nodes where
- tablet servers should run in this file.
- * `conf/tracers` : Optional capability. Can specify zero or more nodes.
+ * `masters` : Accumulo primary coordinating process. Must specify one node. Can
+ specify a few for fault tolerance.
+ * `gc` : Accumulo garbage collector. Must specify one node. Can specify a
+ few for fault tolerance.
+ * `monitor` : Node where Accumulo monitoring web server is run.
+ * `tservers`: Accumulo worker processes. List all of the nodes where tablet servers
+ should run in this file.
+ * `tracers` : Optional capability. Can specify zero or more nodes.
The Accumulo, Hadoop, and Zookeeper software should be present at the same
-location on every node. Also the files in the `conf` directory must be copied
+location on every node. Also the files in the `conf` directory must be copied
to every node. There are many ways to replicate the software and configuration,
two possible tools that can help replicate software and/or config are [pdcp][5]
and [prsync][6].
http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/bin/accumulo
----------------------------------------------------------------------
diff --git a/assemble/bin/accumulo b/assemble/bin/accumulo
index f725952..f530e3c 100755
--- a/assemble/bin/accumulo
+++ b/assemble/bin/accumulo
@@ -15,436 +15,21 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-function build_native() {
- if [[ -f "$basedir"/conf/accumulo-env.sh ]]; then
- source "$basedir"/libexec/load-env.sh
- fi
-
- native_tarball="$basedir/lib/accumulo-native.tar.gz"
- final_native_target="$basedir/lib/native"
-
- if [[ ! -f $native_tarball ]]; then
- echo "Could not find native code artifact: ${native_tarball}";
- exit 1
- fi
-
- # Make the destination for the native library
- mkdir -p "${final_native_target}" || exit 1
-
- # Make a directory for us to unpack the native source into
- TMP_DIR=$(mktemp -d /tmp/accumulo-native.XXXX) || exit 1
-
- # Unpack the tarball to our temp directory
- if ! tar xf "${native_tarball}" -C "${TMP_DIR}"
- then
- echo "Failed to unpack native tarball to ${TMP_DIR}"
- exit 1
- fi
-
- # Move to the first (only) directory in our unpacked tarball
- native_dir=$(find "${TMP_DIR}" -maxdepth 1 -mindepth 1 -type d)
-
- cd "${native_dir}" || exit 1
-
- # Make the native library
- export USERFLAGS="$*"
- make || { echo 'Make failed!'; exit 1; }
-
- # "install" the artifact
- cp libaccumulo.* "${final_native_target}" || exit 1
-
- # Clean up our temp directory
- rm -rf "${TMP_DIR}"
-
- echo "Successfully installed native library"
-}
-
-function create_config_usage() {
- cat <<EOF
-Usage: accumulo create-config [-options]
-where options include (long options not available on all platforms):
- -d, --dir Alternate directory to setup config files
- -s, --size Supported sizes: '1GB' '2GB' '3GB' '512MB'
- -n, --native Configure to use native libraries
- -j, --jvm Configure to use the jvm
- -o, --overwrite Overwrite the default config directory
- -v, --version Specify the Apache Hadoop version supported versions: '1' '2'
- -k, --kerberos Configure for use with Kerberos
- -h, --help Print this help message
-EOF
-}
-
-function create_config() {
- TEMPLATE_CONF_DIR="$basedir/libexec/templates"
- CONF_DIR="${ACCUMULO_CONF_DIR:-$basedir/conf}"
- ACCUMULO_SITE=accumulo-site.xml
- ACCUMULO_ENV=accumulo-env.sh
-
- SIZE=
- TYPE=
- HADOOP_VERSION=
- OVERWRITE="0"
- BASE_DIR=
- KERBEROS=
-
- #Execute getopt
- if [[ $(uname -s) == "Linux" ]]; then
- args=$(getopt -o "b:d:s:njokv:h" -l "basedir:,dir:,size:,native,jvm,overwrite,kerberos,version:,help" -q -- "$@")
- else # Darwin, BSD
- args=$(getopt b:d:s:njokv:h "$@")
- fi
-
- #Bad arguments
- if [[ $? != 0 ]]; then
- create_config_usage 1>&2
+function verify_env_dir() {
+ property=$1
+ directory=$2
+ if [[ -z "$directory" ]]; then
+ echo "$property is not set. Please make sure it's set globally or in $conf/accumulo-env.sh"
exit 1
fi
- eval set -- "${args[@]}"
-
- for i
- do
- case "$i" in
- -b|--basedir) #Hidden option used to set general.maven.project.basedir for developers
- BASE_DIR=$2; shift
- shift;;
- -d|--dir)
- CONF_DIR=$2; shift
- shift;;
- -s|--size)
- SIZE=$2; shift
- shift;;
- -n|--native)
- TYPE=native
- shift;;
- -j|--jvm)
- TYPE=jvm
- shift;;
- -o|--overwrite)
- OVERWRITE=1
- shift;;
- -v|--version)
- HADOOP_VERSION=$2; shift
- shift;;
- -k|--kerberos)
- KERBEROS="true"
- shift;;
- -h|--help)
- create_config_usage
- exit 0
- shift;;
- --)
- shift
- break;;
- esac
- done
-
- while [[ "${OVERWRITE}" = "0" ]]; do
- if [[ -e "${CONF_DIR}/${ACCUMULO_ENV}" || -e "${CONF_DIR}/${ACCUMULO_SITE}" ]]; then
- echo "Warning your current config files in ${CONF_DIR} will be overwritten!"
- echo
- echo "How would you like to proceed?:"
- select CHOICE in 'Continue with overwrite' 'Specify new conf dir'; do
- if [[ "${CHOICE}" = 'Specify new conf dir' ]]; then
- echo -n "Please specifiy new conf directory: "
- read CONF_DIR
- elif [[ "${CHOICE}" = 'Continue with overwrite' ]]; then
- OVERWRITE=1
- fi
- break
- done
- else
- OVERWRITE=1
- fi
- done
- echo "Copying configuration files to: ${CONF_DIR}"
-
- #Native 1GB
- native_1GB_tServer="-Xmx128m -Xms128m"
- _1GB_master="-Xmx128m -Xms128m"
- _1GB_monitor="-Xmx64m -Xms64m"
- _1GB_gc="-Xmx64m -Xms64m"
- _1GB_other="-Xmx128m -Xms64m"
- _1GB_shell="${_1GB_other}"
-
- _1GB_memoryMapMax="256M"
- native_1GB_nativeEnabled="true"
- _1GB_cacheDataSize="15M"
- _1GB_cacheIndexSize="40M"
- _1GB_sortBufferSize="50M"
- _1GB_waLogMaxSize="256M"
-
- #Native 2GB
- native_2GB_tServer="-Xmx256m -Xms256m"
- _2GB_master="-Xmx256m -Xms256m"
- _2GB_monitor="-Xmx128m -Xms64m"
- _2GB_gc="-Xmx128m -Xms128m"
- _2GB_other="-Xmx256m -Xms64m"
- _2GB_shell="${_2GB_other}"
-
- _2GB_memoryMapMax="512M"
- native_2GB_nativeEnabled="true"
- _2GB_cacheDataSize="30M"
- _2GB_cacheIndexSize="80M"
- _2GB_sortBufferSize="50M"
- _2GB_waLogMaxSize="512M"
-
- #Native 3GB
- native_3GB_tServer="-Xmx1g -Xms1g -XX:NewSize=500m -XX:MaxNewSize=500m"
- _3GB_master="-Xmx1g -Xms1g"
- _3GB_monitor="-Xmx1g -Xms256m"
- _3GB_gc="-Xmx256m -Xms256m"
- _3GB_other="-Xmx1g -Xms256m"
- _3GB_shell="${_3GB_other}"
-
- _3GB_memoryMapMax="1G"
- native_3GB_nativeEnabled="true"
- _3GB_cacheDataSize="128M"
- _3GB_cacheIndexSize="128M"
- _3GB_sortBufferSize="200M"
- _3GB_waLogMaxSize="1G"
-
- #Native 512MB
- native_512MB_tServer="-Xmx48m -Xms48m"
- _512MB_master="-Xmx128m -Xms128m"
- _512MB_monitor="-Xmx64m -Xms64m"
- _512MB_gc="-Xmx64m -Xms64m"
- _512MB_other="-Xmx128m -Xms64m"
- _512MB_shell="${_512MB_other}"
-
- _512MB_memoryMapMax="80M"
- native_512MB_nativeEnabled="true"
- _512MB_cacheDataSize="7M"
- _512MB_cacheIndexSize="20M"
- _512MB_sortBufferSize="50M"
- _512MB_waLogMaxSize="100M"
-
- #JVM 1GB
- jvm_1GB_tServer="-Xmx384m -Xms384m"
-
- jvm_1GB_nativeEnabled="false"
-
- #JVM 2GB
- jvm_2GB_tServer="-Xmx768m -Xms768m"
-
- jvm_2GB_nativeEnabled="false"
-
- #JVM 3GB
- jvm_3GB_tServer="-Xmx2g -Xms2g -XX:NewSize=1G -XX:MaxNewSize=1G"
-
- jvm_3GB_nativeEnabled="false"
-
- #JVM 512MB
- jvm_512MB_tServer="-Xmx128m -Xms128m"
-
- jvm_512MB_nativeEnabled="false"
-
-
- if [[ -z "${SIZE}" ]]; then
- echo "Choose the heap configuration:"
- select DIRNAME in 1GB 2GB 3GB 512MB; do
- echo "Using '${DIRNAME}' configuration"
- SIZE=${DIRNAME}
- break
- done
- elif [[ "${SIZE}" != "1GB" && "${SIZE}" != "2GB" && "${SIZE}" != "3GB" && "${SIZE}" != "512MB" ]]; then
- echo "Invalid memory size"
- echo "Supported sizes: '1GB' '2GB' '3GB' '512MB'"
- exit 1
- fi
-
- if [[ -z "${TYPE}" ]]; then
- echo
- echo "Choose the Accumulo memory-map type:"
- select TYPENAME in Java Native; do
- if [[ "${TYPENAME}" == "Native" ]]; then
- TYPE="native"
- echo "Don't forget to build the native libraries using the command 'bin/accumulo build-native'"
- elif [[ "${TYPENAME}" == "Java" ]]; then
- TYPE="jvm"
- fi
- echo "Using '${TYPE}' configuration"
- echo
- break
- done
- fi
-
- if [[ -z "${HADOOP_VERSION}" ]]; then
- echo
- echo "Choose the Apache Hadoop version:"
- select HADOOP in 'Hadoop 2' 'HDP 2.0/2.1' 'HDP 2.2' 'IOP 4.1'; do
- if [ "${HADOOP}" == "Hadoop 2" ]; then
- HADOOP_VERSION="2"
- elif [ "${HADOOP}" == "HDP 2.0/2.1" ]; then
- HADOOP_VERSION="HDP2"
- elif [ "${HADOOP}" == "HDP 2.2" ]; then
- HADOOP_VERSION="HDP2.2"
- elif [ "${HADOOP}" == "IOP 4.1" ]; then
- HADOOP_VERSION="IOP4.1"
- fi
- echo "Using Hadoop version '${HADOOP_VERSION}' configuration"
- echo
- break
- done
- elif [[ "${HADOOP_VERSION}" != "2" && "${HADOOP_VERSION}" != "HDP2" && "${HADOOP_VERSION}" != "HDP2.2" ]]; then
- echo "Invalid Hadoop version"
- echo "Supported Hadoop versions: '2', 'HDP2', 'HDP2.2'"
+ if [[ ! -d "$directory" ]]; then
+ echo "$property=$directory is not a valid directory. Please make sure it's set correctly globally or in $conf/accumulo-env.sh"
exit 1
fi
-
- TRACE_USER="root"
-
- if [[ ! -z "${KERBEROS}" ]]; then
- echo
- read -p "Enter server's Kerberos principal: " PRINCIPAL
- read -p "Enter server's Kerberos keytab: " KEYTAB
- TRACE_USER="${PRINCIPAL}"
- fi
-
- for var in SIZE TYPE HADOOP_VERSION; do
- if [[ -z ${!var} ]]; then
- echo "Invalid $var configuration"
- exit 1
- fi
- done
-
- TSERVER="${TYPE}_${SIZE}_tServer"
- MASTER="_${SIZE}_master"
- MONITOR="_${SIZE}_monitor"
- GC="_${SIZE}_gc"
- SHELL="_${SIZE}_shell"
- OTHER="_${SIZE}_other"
-
- MEMORY_MAP_MAX="_${SIZE}_memoryMapMax"
- NATIVE="${TYPE}_${SIZE}_nativeEnabled"
- CACHE_DATA_SIZE="_${SIZE}_cacheDataSize"
- CACHE_INDEX_SIZE="_${SIZE}_cacheIndexSize"
- SORT_BUFFER_SIZE="_${SIZE}_sortBufferSize"
- WAL_MAX_SIZE="_${SIZE}_waLogMaxSize"
-
- MAVEN_PROJ_BASEDIR=""
-
- if [[ ! -z "${BASE_DIR}" ]]; then
- MAVEN_PROJ_BASEDIR="\n <property>\n <name>general.maven.project.basedir</name>\n <value>${BASE_DIR}</value>\n </property>\n"
- fi
-
- mkdir -p "${CONF_DIR}" && cp "${TEMPLATE_CONF_DIR}"/* "${CONF_DIR}"/
-
- if [[ -f "${CONF_DIR}/examples/client.conf" ]]; then
- cp "${CONF_DIR}"/examples/client.conf "${CONF_DIR}"/
- fi
-
- #Configure accumulo-env.sh
- sed -e "s/\${tServerHigh_tServerLow}/${!TSERVER}/" \
- -e "s/\${masterHigh_masterLow}/${!MASTER}/" \
- -e "s/\${monitorHigh_monitorLow}/${!MONITOR}/" \
- -e "s/\${gcHigh_gcLow}/${!GC}/" \
- -e "s/\${shellHigh_shellLow}/${!SHELL}/" \
- -e "s/\${otherHigh_otherLow}/${!OTHER}/" \
- "${TEMPLATE_CONF_DIR}/$ACCUMULO_ENV" > "${CONF_DIR}/$ACCUMULO_ENV"
-
- #Configure accumulo-site.xml
- sed -e "s/\${memMapMax}/${!MEMORY_MAP_MAX}/" \
- -e "s/\${nativeEnabled}/${!NATIVE}/" \
- -e "s/\${cacheDataSize}/${!CACHE_DATA_SIZE}/" \
- -e "s/\${cacheIndexSize}/${!CACHE_INDEX_SIZE}/" \
- -e "s/\${sortBufferSize}/${!SORT_BUFFER_SIZE}/" \
- -e "s/\${waLogMaxSize}/${!WAL_MAX_SIZE}/" \
- -e "s=\${traceUser}=${TRACE_USER}=" \
- -e "s=\${mvnProjBaseDir}=${MAVEN_PROJ_BASEDIR}=" "${TEMPLATE_CONF_DIR}/$ACCUMULO_SITE" > "${CONF_DIR}/$ACCUMULO_SITE"
-
- # If we're not using kerberos, filter out the krb properties
- if [[ -z "${KERBEROS}" ]]; then
- sed -e 's/<!-- Kerberos requirements -->/<!-- Kerberos requirements --><!--/' \
- -e 's/<!-- End Kerberos requirements -->/--><!-- End Kerberos requirements -->/' \
- "${CONF_DIR}/$ACCUMULO_SITE" > temp
- mv temp "${CONF_DIR}/$ACCUMULO_SITE"
- else
- # Make the substitutions
- sed -e "s!\${keytab}!${KEYTAB}!" \
- -e "s!\${principal}!${PRINCIPAL}!" \
- "${CONF_DIR}/${ACCUMULO_SITE}" > temp
- mv temp "${CONF_DIR}/${ACCUMULO_SITE}"
- fi
-
- # Configure hadoop version
- if [[ "${HADOOP_VERSION}" == "2" ]]; then
- sed -e 's/<!-- HDP 2.0 requirements -->/<!-- HDP 2.0 requirements --><!--/' \
- -e 's/<!-- End HDP 2.0 requirements -->/--><!-- End HDP 2.0 requirements -->/' \
- "${CONF_DIR}/$ACCUMULO_SITE" > temp
- mv temp "${CONF_DIR}/$ACCUMULO_SITE"
- sed -e 's/<!-- HDP 2.2 requirements -->/<!-- HDP 2.2 requirements --><!--/' \
- -e 's/<!-- End HDP 2.2 requirements -->/--><!-- End HDP 2.2 requirements -->/' \
- "${CONF_DIR}/$ACCUMULO_SITE" > temp
- mv temp "${CONF_DIR}/$ACCUMULO_SITE"
- sed -e 's/<!-- IOP 4.1 requirements -->/<!-- IOP 4.1 requirements --><!--/' \
- -e 's/<!-- End IOP 4.1 requirements -->/--><!-- End IOP 4.1 requirements -->/' \
- "${CONF_DIR}/$ACCUMULO_SITE" > temp
- mv temp "${CONF_DIR}/$ACCUMULO_SITE"
- elif [[ "${HADOOP_VERSION}" == "HDP2" ]]; then
- sed -e 's/<!-- Hadoop 2 requirements -->/<!-- Hadoop 2 requirements --><!--/' \
- -e 's/<!-- End Hadoop 2 requirements -->/--><!-- End Hadoop 2 requirements -->/' \
- "${CONF_DIR}/$ACCUMULO_SITE" > temp
- mv temp "${CONF_DIR}/$ACCUMULO_SITE"
- sed -e 's/<!-- HDP 2.2 requirements -->/<!-- HDP 2.2 requirements --><!--/' \
- -e 's/<!-- End HDP 2.2 requirements -->/--><!-- End HDP 2.2 requirements -->/' \
- "${CONF_DIR}/$ACCUMULO_SITE" > temp
- mv temp "${CONF_DIR}/$ACCUMULO_SITE"
- sed -e 's/<!-- IOP 4.1 requirements -->/<!-- IOP 4.1 requirements --><!--/' \
- -e 's/<!-- End IOP 4.1 requirements -->/--><!-- End IOP 4.1 requirements -->/' \
- "${CONF_DIR}/$ACCUMULO_SITE" > temp
- mv temp "${CONF_DIR}/$ACCUMULO_SITE"
- elif [[ "${HADOOP_VERSION}" == "HDP2.2" ]]; then
- sed -e 's/<!-- Hadoop 2 requirements -->/<!-- Hadoop 2 requirements --><!--/' \
- -e 's/<!-- End Hadoop 2 requirements -->/--><!-- End Hadoop 2 requirements -->/' \
- "${CONF_DIR}/$ACCUMULO_SITE" > temp
- mv temp "${CONF_DIR}/$ACCUMULO_SITE"
- sed -e 's/<!-- HDP 2.0 requirements -->/<!-- HDP 2.0 requirements --><!--/' \
- -e 's/<!-- End HDP 2.0 requirements -->/--><!-- End HDP 2.0 requirements -->/' \
- "${CONF_DIR}/$ACCUMULO_SITE" > temp
- mv temp "${CONF_DIR}/$ACCUMULO_SITE"
- sed -e 's/<!-- IOP 4.1 requirements -->/<!-- IOP 4.1 requirements --><!--/' \
- -e 's/<!-- End IOP 4.1 requirements -->/--><!-- End IOP 4.1 requirements -->/' \
- "${CONF_DIR}/$ACCUMULO_SITE" > temp
- mv temp "${CONF_DIR}/$ACCUMULO_SITE"
- elif [[ "${HADOOP_VERSION}" == "IOP4.1" ]]; then
- sed -e 's/<!-- Hadoop 2 requirements -->/<!-- Hadoop 2 requirements --><!--/' \
- -e 's/<!-- End Hadoop 2 requirements -->/--><!-- End Hadoop 2 requirements -->/' \
- "${CONF_DIR}/$ACCUMULO_SITE" > temp
- mv temp "${CONF_DIR}/$ACCUMULO_SITE"
- sed -e 's/<!-- HDP 2.0 requirements -->/<!-- HDP 2.0 requirements --><!--/' \
- -e 's/<!-- End HDP 2.0 requirements -->/--><!-- End HDP 2.0 requirements -->/' \
- "${CONF_DIR}/$ACCUMULO_SITE" > temp
- mv temp "${CONF_DIR}/$ACCUMULO_SITE"
- sed -e 's/<!-- HDP 2.2 requirements -->/<!-- HDP 2.2 requirements --><!--/' \
- -e 's/<!-- End HDP 2.2 requirements -->/--><!-- End HDP 2.2 requirements -->/' \
- "${CONF_DIR}/$ACCUMULO_SITE" > temp
- mv temp "${CONF_DIR}/$ACCUMULO_SITE"
- fi
-
- #Additional setup steps for native configuration.
- if [[ ${TYPE} == native ]]; then
- if [[ $(uname) == Linux ]]; then
- if [[ -z $HADOOP_PREFIX ]]; then
- echo "WARNING: HADOOP_PREFIX not set, cannot automatically configure LD_LIBRARY_PATH to include Hadoop native libraries"
- else
- NATIVE_LIB=$(readlink -ef "$(dirname "$(for x in $(find "$HADOOP_PREFIX" -name libhadoop.so); do ld "$x" 2>/dev/null && echo "$x" && break; done)" 2>>/dev/null)" 2>>/dev/null)
- if [[ -z $NATIVE_LIB ]]; then
- echo -e "WARNING: The Hadoop native libraries could not be found for your sytem in: $HADOOP_PREFIX"
- else
- sed "/# Should the monitor/ i export LD_LIBRARY_PATH=${NATIVE_LIB}:\${LD_LIBRARY_PATH}" "${CONF_DIR}/$ACCUMULO_ENV" > temp
- mv temp "${CONF_DIR}/$ACCUMULO_ENV"
- echo -e "Added ${NATIVE_LIB} to the LD_LIBRARY_PATH"
- fi
- fi
- fi
- echo -e "Please remember to compile the Accumulo native libraries using the command 'bin/accumulo build-native' and to set the LD_LIBRARY_PATH variable in the ${CONF_DIR}/accumulo-env.sh script if needed."
- fi
- echo "Setup complete"
}
function main() {
- # Start: Resolve Script Directory
SOURCE="${BASH_SOURCE[0]}"
while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
bin="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
@@ -453,129 +38,108 @@ function main() {
done
bin="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
basedir=$( cd -P "${bin}"/.. && pwd )
- # Stop: Resolve Script Directory
+ conf="${basedir}/conf"
+ lib="${basedir}/lib"
- if [[ "$1" == "create-config" ]]; then
- create_config "${@:2}"
- exit 0
- elif [[ "$1" == "build-native" ]]; then
- build_native "${@:2}"
- exit 0
- fi
+ # Might be needed by accumulo-env.sh, accumulo-site.xml, and other Accumulo configuration
+ export ACCUMULO_HOME="$basedir"
+ export ACCUMULO_CONF_DIR="$conf"
- source "$basedir"/libexec/load-env.sh
+ export ACCUMULO_CMD="$1"
- # ACCUMULO_XTRAJARS is where all of the commandline -add items go into for reading by accumulo.
- # It also holds the JAR run with the jar command and, if possible, any items in the JAR manifest's Class-Path.
- if [[ "$1" = "-add" ]]; then
- export ACCUMULO_XTRAJARS="$2"
- shift 2
- else
- export ACCUMULO_XTRAJARS=""
+ if [[ -z $conf || ! -d $conf ]]; then
+ echo "$conf is not a valid directory. Please make sure it exists"
+ exit 1
fi
- if [[ "$1" = "jar" && -f "$2" ]]; then
- if [[ $2 =~ ^/ ]]; then
- jardir="$(dirname "$2")"
- jarfile="$2"
- else
- jardir="$(pwd)"
- jarfile="${jardir}/${2}"
- fi
- if jar tf "$jarfile" | grep -q META-INF/MANIFEST.MF ; then
- cp="$(unzip -p "$jarfile" META-INF/MANIFEST.MF | grep ^Class-Path: | sed 's/^Class-Path: *//')"
- if [[ -n "$cp" ]] ; then
- for j in $cp; do
- if [[ "$j" != "Class-Path:" ]] ; then
- ACCUMULO_XTRAJARS="${jardir}/${j},$ACCUMULO_XTRAJARS"
- fi
- done
- fi
- fi
- ACCUMULO_XTRAJARS="${jarfile},$ACCUMULO_XTRAJARS"
+ if [[ ! -f $conf/accumulo-env.sh || ! -f $conf/accumulo-site.xml ]]; then
+ echo "The configuration files 'accumulo-env.sh' & 'accumulo-site.xml' must exist in $conf"
+ echo "Copy them from $conf/examples and edit them for your environment."
+ exit 1
fi
+ source "$conf/accumulo-env.sh"
- # Set up -D switches for JAAS and Kerberos if files exist
- if [[ -f ${ACCUMULO_JAAS_CONF} ]]; then
- ACCUMULO_GENERAL_OPTS="${ACCUMULO_GENERAL_OPTS} -Djava.security.auth.login.config=${ACCUMULO_JAAS_CONF}"
- fi
- if [[ -f ${ACCUMULO_KRB5_CONF} ]]; then
- ACCUMULO_GENERAL_OPTS="${ACCUMULO_GENERAL_OPTS} -Djava.security.krb5.conf=${ACCUMULO_KRB5_CONF}"
- fi
+ # Verify setting in accumulo-env.sh
- # Add appropriate options for process type
- case "$1" in
- master) export ACCUMULO_OPTS="${ACCUMULO_GENERAL_OPTS} ${ACCUMULO_MASTER_OPTS}" ;;
- gc) export ACCUMULO_OPTS="${ACCUMULO_GENERAL_OPTS} ${ACCUMULO_GC_OPTS}" ;;
- tserver*) export ACCUMULO_OPTS="${ACCUMULO_GENERAL_OPTS} ${ACCUMULO_TSERVER_OPTS}" ;;
- monitor) export ACCUMULO_OPTS="${ACCUMULO_GENERAL_OPTS} ${ACCUMULO_MONITOR_OPTS}" ;;
- shell) export ACCUMULO_OPTS="${ACCUMULO_GENERAL_OPTS} ${ACCUMULO_SHELL_OPTS}" ;;
- *) export ACCUMULO_OPTS="${ACCUMULO_GENERAL_OPTS} ${ACCUMULO_OTHER_OPTS}" ;;
- esac
+ : "${JAVA_OPTS:?"variable is not set in accumulo-env.sh"}"
+ : "${ACCUMULO_LOG_DIR:?"variable is not set in accumulo-env.sh"}"
+ mkdir -p "${ACCUMULO_LOG_DIR}" 2>/dev/null
+ verify_env_dir "ACCUMULO_LOG_DIR" "${ACCUMULO_LOG_DIR}"
+ verify_env_dir "HADOOP_PREFIX" "${HADOOP_PREFIX}"
+ verify_env_dir "HADOOP_CONF_DIR" "${HADOOP_CONF_DIR}"
+ verify_env_dir "ZOOKEEPER_HOME" "${ZOOKEEPER_HOME}"
+ : "${MALLOC_ARENA_MAX:?"variable is not set in accumulo-env.sh"}"
- LOG4J_JAR=$(find -H "${HADOOP_PREFIX}/lib" "${HADOOP_PREFIX}"/share/hadoop/common/lib -name 'log4j*.jar' -print 2>/dev/null | head -1)
- SLF4J_JARS="${ACCUMULO_LIB_DIR}/slf4j-api.jar:${ACCUMULO_LIB_DIR}/slf4j-log4j12.jar"
+ export HADOOP_HOME=$HADOOP_PREFIX
+ export HADOOP_HOME_WARN_SUPPRESS=true
- # The `find` command could fail for environmental reasons or bad configuration
- # Avoid trying to run Accumulo when we can't find the jar
- if [[ -z "${LOG4J_JAR}" && -z "${CLASSPATH}" ]]; then
- echo "Could not locate Log4j jar in Hadoop installation at ${HADOOP_PREFIX}"
- exit 1
+ if [ -f "${conf}/jaas.conf" ]; then
+ export JAVA_OPTS=("${JAVA_OPTS[@]}" "-Djava.security.auth.login.config=${conf}/jaas.conf")
+ fi
+ if [ -f "${conf}/krb5.conf" ]; then
+ export JAVA_OPTS=("${JAVA_OPTS[@]}" "-Djava.security.krb5.conf=${conf}/krb5.conf")
fi
- if [[ ! "$1" =~ ^(gc|master|monitor|tserver|tracer)$ ]]; then
- if [[ -f ${ACCUMULO_CONF_DIR}/log4j.properties ]]; then
- export ACCUMULO_OPTS="${ACCUMULO_OPTS} -Dlog4j.configuration=file:${ACCUMULO_CONF_DIR}/log4j.properties"
+ if [[ ! "$ACCUMULO_CMD" =~ ^(gc|master|monitor|tserver|tracer)$ ]]; then
+ if [[ -f ${conf}/log4j.properties ]]; then
+ export JAVA_OPTS=("${JAVA_OPTS[@]}" "-Dlog4j.configuration=file:${conf}/log4j.properties")
else
- export ACCUMULO_OPTS="${ACCUMULO_OPTS} -Dlog4j.configuration=file:${ACCUMULO_CONF_DIR}/examples/log4j.properties"
+ export JAVA_OPTS=("${JAVA_OPTS[@]}" "-Dlog4j.configuration=file:${conf}/examples/log4j.properties")
fi
fi
- CLASSPATH="${ACCUMULO_LIB_DIR}/accumulo-start.jar:${ACCUMULO_CONF_DIR}:${SLF4J_JARS}:${LOG4J_JAR}:${CLASSPATH}"
-
- # This is default for hadoop 2.x;
- # for another distribution, specify (DY)LD_LIBRARY_PATH
- # explicitly in ${ACCUMULO_CONF_DIR}/accumulo-env.sh
- # usually something like:
- # ${HADOOP_PREFIX}/lib/native/${PLATFORM}
+ # This is default for hadoop 2.x; for another distribution, specify (DY)LD_LIBRARY_PATH explicitly above
if [ -e "${HADOOP_PREFIX}/lib/native/libhadoop.so" ]; then
- LIB_PATH="${HADOOP_PREFIX}/lib/native"
- LD_LIBRARY_PATH="${LIB_PATH}:${LD_LIBRARY_PATH}" # For Linux
- DYLD_LIBRARY_PATH="${LIB_PATH}:${DYLD_LIBRARY_PATH}" # For Mac
+ export LD_LIBRARY_PATH="${HADOOP_PREFIX}/lib/native:${LD_LIBRARY_PATH}" # For Linux
+ export DYLD_LIBRARY_PATH="${HADOOP_PREFIX}/lib/native:${DYLD_LIBRARY_PATH}" # For Mac
+ fi
+ # ACCUMULO_XTRAJARS is where all of the commandline -add items go into for reading by accumulo.
+ # It also holds the JAR run with the jar command and, if possible, any items in the JAR manifest's Class-Path.
+ if [[ "$ACCUMULO_CMD" = "-add" ]]; then
+ export ACCUMULO_XTRAJARS="$2"
+ shift 2
+ else
+ export ACCUMULO_XTRAJARS=""
+ fi
+ if [[ "$ACCUMULO_CMD" = "jar" && -f "$2" ]]; then
+ if [[ $2 =~ ^/ ]]; then
+ jardir="$(dirname "$2")"
+ jarfile="$2"
+ else
+ jardir="$(pwd)"
+ jarfile="${jardir}/${2}"
+ fi
+ if jar tf "$jarfile" | grep -q META-INF/MANIFEST.MF ; then
+ cp="$(unzip -p "$jarfile" META-INF/MANIFEST.MF | grep ^Class-Path: | sed 's/^Class-Path: *//')"
+ if [[ -n "$cp" ]] ; then
+ for j in $cp; do
+ if [[ "$j" != "Class-Path:" ]] ; then
+ ACCUMULO_XTRAJARS="${jardir}/${j},$ACCUMULO_XTRAJARS"
+ fi
+ done
+ fi
+ fi
+ ACCUMULO_XTRAJARS="${jarfile},$ACCUMULO_XTRAJARS"
fi
- # Export the variables just in case they are not exported
- # This makes them available to java
- export JAVA_HOME HADOOP_PREFIX ZOOKEEPER_HOME LD_LIBRARY_PATH DYLD_LIBRARY_PATH
-
- # Strip the instance from $1
- APP=$1
- # Avoid setting an instance unless it's necessary to ensure consistency in filenames
- INSTANCE=""
- # Avoid setting a pointless system property
- INSTANCE_OPT=""
- if [[ "$1" =~ ^tserver-[1-9][0-9]*$ ]]; then
- APP="$(echo "$1" | cut -d'-' -f1)"
- # Appending the trailing underscore to make single-tserver deploys look how they did
- INSTANCE="$(echo "$1" | cut -d'-' -f2)_"
-
- #Rewrite the input arguments
- set -- "$APP" "${@:2}"
-
- # The extra system property we'll pass to the java cmd
- INSTANCE_OPT="-Daccumulo.service.instance=${INSTANCE}"
+ if [ -x "$JAVA_HOME/bin/java" ]; then
+ JAVA="$JAVA_HOME/bin/java"
+ else
+ JAVA=$(which java)
+ fi
+ if [ ! -x "$JAVA" ]; then
+ echo "Could not find any executable java binary. Please set java on your PATH or set JAVA_HOME"
+ exit 1
fi
- # app isn't used anywhere, but it makes the process easier to spot when ps/top/snmp truncate the command line
- JAVA="${JAVA_HOME}/bin/java"
+ if [ -n "$ACCUMULO_JAVA_PREFIX" ]; then
+ JAVA=($ACCUMULO_JAVA_PREFIX $JAVA)
+ fi
+
+ CLASSPATH="${lib}/accumulo-start.jar:${conf}:${lib}/slf4j-api.jar:${lib}/slf4j-log4j12.jar:${lib}/log4j.jar:${CLASSPATH}"
- ACCUMULO_OPTS_ARRAY=(${ACCUMULO_OPTS})
- exec "$JAVA" "-Dapp=$1" \
- $INSTANCE_OPT \
- "${ACCUMULO_OPTS_ARRAY[@]}" \
+ exec "${JAVA[@]}" "-Dapp=$ACCUMULO_CMD" \
+ "${JAVA_OPTS[@]}" \
-classpath "${CLASSPATH}" \
- -XX:OnOutOfMemoryError="${ACCUMULO_KILL_CMD:-kill -9 %p}" \
- -XX:-OmitStackTraceInFastThrow \
- -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl \
-Dhadoop.home.dir="${HADOOP_PREFIX}" \
-Dzookeeper.home.dir="${ZOOKEEPER_HOME}" \
org.apache.accumulo.start.Main \
http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/bin/accumulo-cluster
----------------------------------------------------------------------
diff --git a/assemble/bin/accumulo-cluster b/assemble/bin/accumulo-cluster
index a8c0362..4f457a5 100755
--- a/assemble/bin/accumulo-cluster
+++ b/assemble/bin/accumulo-cluster
@@ -20,10 +20,14 @@ function print_usage {
Usage: accumulo-cluster <command> (<argument> ...)
Commands:
- create-config Creates cluster config
- start Starts Accumulo cluster
- stop Stops Accumulo cluster
-
+ create-config Creates cluster config
+ start Starts Accumulo cluster
+ stop Stops Accumulo cluster
+ start-non-tservers Starts all services except tservers
+ start-tservers Starts all tservers on cluster
+ stop-tservers Stops all tservers on cluster
+ start-here Starts all services on this node
+ stop-here Stops all services on this node
EOF
exit 1
}
@@ -34,28 +38,248 @@ function invalid_args {
exit 1
}
-function verify_dir {
- if [[ ! -d $1 ]]; then
- echo "Directory does not exist: $1"
+function verify_config {
+ if [[ -f ${conf}/slaves ]]; then
+ echo "ERROR: A 'slaves' file was found in ${conf}/"
+ echo "Accumulo now reads tablet server hosts from 'tservers' and requires that the 'slaves' file not be present to reduce confusion."
+ echo "Please rename the 'slaves' file to 'tservers' or remove it if both exist."
exit 1
fi
-}
-function verify_file {
- if [[ ! -f $1 ]]; then
- echo "File does not exist: $1"
+ if [[ ! -f ${conf}/tservers ]]; then
+ echo "ERROR: A 'tservers' file was not found at ${conf}/tservers"
+ echo "Please make sure it exists and is configured with tablet server hosts."
exit 1
fi
+
+ unset master1
+ if [[ -f "${conf}/masters" ]]; then
+ master1=$(egrep -v '(^#|^\s*$)' "${conf}/masters" | head -1)
+ fi
+
+ if [[ -z "${monitor}" ]] ; then
+ monitor=$master1
+ if [[ -f "${conf}/monitor" ]]; then
+ monitor=$(egrep -v '(^#|^\s*$)' "${conf}/monitor" | head -1)
+ fi
+ if [[ -z "${monitor}" ]] ; then
+ echo "Could not infer a Monitor role. You need to either define \"${conf}/monitor\","
+ echo "or make sure \"${conf}/masters\" is non-empty."
+ exit 1
+ fi
+ fi
+ if [[ ! -f "${conf}/tracers" ]]; then
+ if [[ -z "${master1}" ]] ; then
+ echo "Could not find a master node to use as a default for the tracer role."
+ echo "Either set up \"${conf}/tracers\" or make sure \"${conf}/masters\" is non-empty."
+ exit 1
+ else
+ echo "$master1" > "${conf}/tracers"
+ fi
+ fi
+ if [[ ! -f "${conf}/gc" ]]; then
+ if [[ -z "${master1}" ]] ; then
+ echo "Could not infer a GC role. You need to either set up \"${conf}/gc\" or make sure \"${conf}/masters\" is non-empty."
+ exit 1
+ else
+ echo "$master1" > "${conf}/gc"
+ fi
+ fi
+}
+
+function get_ip() {
+ ip_addr=$(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}' | cut -f1 -d'/')
+ if [[ $? != 0 ]]; then
+ ip_addr=$(getent ahosts "$(hostname -f)" | grep DGRAM | cut -f 1 -d ' ')
+ fi
+ echo "$ip_addr"
+}
+
+function start_service() {
+ host="$1"
+ service="$2"
+
+ if [[ $host == "localhost" || $host == $(hostname -f) || $host == $(hostname -s) || $host == $(get_ip) ]]; then
+ "${bin}/accumulo-service" "$service" start
+ else
+ $SSH "$host" "bash -c '${bin}/accumulo-service \"$service\" start'"
+ fi
}
-function copy_file {
- verify_file "$1"
- verify_dir "$2"
- if ! cp "$1" "$2"
+function start_tservers() {
+ echo -n "Starting tablet servers ..."
+ count=1
+ for server in $(egrep -v '(^#|^\s*$)' "${conf}/tservers"); do
+ echo -n "."
+ start_service "$server" tserver &
+ if (( ++count % 72 == 0 )) ;
+ then
+ echo
+ wait
+ fi
+ done
+ echo " done"
+}
+
+function start_all() {
+ unset DISPLAY
+
+ start_service "$monitor" monitor
+
+ if [ "$1" != "--no-tservers" ]; then
+ start_tservers
+ fi
+
+ for host in $(egrep -v '(^#|^\s*$)' "${conf}/masters"); do
+ start_service "$host" master
+ done
+
+ for host in $(egrep -v '(^#|^\s*$)' "${conf}/gc"); do
+ start_service "$host" gc
+ done
+
+ for host in $(egrep -v '(^#|^\s*$)' "${conf}/tracers"); do
+ start_service "$host" tracer
+ done
+}
+
+function start_here() {
+
+ local_hosts="$(hostname -a 2> /dev/null) $(hostname) localhost 127.0.0.1 $(get_ip)"
+ for host in $local_hosts; do
+ if grep -q "^${host}\$" "${conf}/tservers"; then
+ start_service "$host" tserver
+ break
+ fi
+ done
+
+ for host in $local_hosts; do
+ if grep -q "^${host}\$" "${conf}/masters"; then
+ start_service "$host" master
+ break
+ fi
+ done
+
+ for host in $local_hosts; do
+ if grep -q "^${host}\$" "${conf}/gc"; then
+ start_service "$host" gc
+ break
+ fi
+ done
+
+ for host in $local_hosts; do
+ if [ "$host" == "$monitor" ]; then
+ start_service "$host" monitor
+ break
+ fi
+ done
+
+ for host in $local_hosts; do
+ if grep -q "^${host}\$" "${conf}/tracers"; then
+ start_service "$host" tracer
+ break
+ fi
+ done
+}
+
+function end_service() {
+ host="$1"
+ service="$2"
+ end_cmd="$3"
+ if [[ $host == localhost || $host = "$(hostname -s)" || $host = "$(hostname -f)" || $host = $(get_ip) ]] ; then
+ "${bin}/accumulo-service" "$service" "$end_cmd"
+ else
+ $SSH "$host" "bash -c '${bin}/accumulo-service \"$service\" \"$end_cmd\"'"
+ fi
+}
+
+function stop_service() {
+ end_service "$1" "$2" "stop"
+}
+
+function kill_service() {
+ end_service "$1" "$2" "kill"
+}
+
+function stop_tservers() {
+ tserver_hosts=$(egrep -v '(^#|^\s*$)' "${conf}/tservers")
+
+ echo "Stopping unresponsive tablet servers (if any)..."
+ for host in ${tserver_hosts}; do
+ stop_service "$host" tserver &
+ done
+
+ sleep 10
+
+ echo "Stopping unresponsive tablet servers hard (if any)..."
+ for host in ${tserver_hosts}; do
+ kill_service "$host" tserver &
+ done
+
+ echo "Cleaning tablet server entries from zookeeper"
+ ${accumulo_cmd} org.apache.accumulo.server.util.ZooZap -tservers
+}
+
+function stop_all() {
+ echo "Stopping accumulo services..."
+ if ! ${accumulo_cmd} admin stopAll
then
- echo "Failed to copy file '$1' to '$2'"
- exit 1
+ echo "Invalid password or unable to connect to the master"
+ echo "Initiating forced shutdown in 15 seconds (Ctrl-C to abort)"
+ sleep 10
+ echo "Initiating forced shutdown in 5 seconds (Ctrl-C to abort)"
+ else
+ echo "Accumulo shut down cleanly"
+ echo "Utilities and unresponsive servers will shut down in 5 seconds (Ctrl-C to abort)"
+ fi
+
+ sleep 5
+
+ # Look for processes not killed by 'admin stopAll'
+ for end_cmd in "stop" "kill" ; do
+ for master in $(grep -v '^#' "${conf}/masters"); do
+ end_service "$master" master $end_cmd
+ done
+
+ for gc in $(grep -v '^#' "${conf}/gc"); do
+ end_service "$gc" gc $end_cmd
+ done
+
+ end_service "$monitor" monitor $end_cmd
+
+ for tracer in $(egrep -v '(^#|^\s*$)' "${conf}/tracers"); do
+ end_service "$tracer" tracer $end_cmd
+ done
+ done
+
+ # stop tserver still running
+ stop_tservers
+
+ echo "Cleaning all server entries in ZooKeeper"
+ ${accumulo_cmd} org.apache.accumulo.server.util.ZooZap -master -tservers -tracers --site-file "${conf}/accumulo-site.xml"
+}
+
+function stop_here() {
+ # Determine hostname without errors to user
+ hosts_to_check=($(hostname -a 2> /dev/null | head -1) $(hostname -f))
+
+ if egrep -q localhost\|127.0.0.1 "${conf}/tservers"; then
+ ${accumulo_cmd} admin stop localhost
+ else
+ for host in "${hosts_to_check[@]}"; do
+ if grep -q "$host" "${conf}/tservers"; then
+ ${accumulo_cmd} admin stop "$host"
+ fi
+ done
fi
+
+ for host in "${hosts_to_check[@]}"; do
+ for end_cmd in "stop" "kill"; do
+ for svc in tserver gc master monitor tracer; do
+ end_service "$host" $svc $end_cmd
+ done
+ done
+ done
}
function main() {
@@ -73,24 +297,46 @@ function main() {
done
bin="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
basedir=$( cd -P "${bin}"/.. && pwd )
+ conf="${basedir}/conf"
- source "$basedir"/libexec/load-env.sh
-
- verify_file "${ACCUMULO_LIBEXEC_DIR}/cluster.sh"
+ accumulo_cmd="${bin}/accumulo"
+ SSH='ssh -qnf -o ConnectTimeout=2'
case "$1" in
create-config)
- copy_file "${ACCUMULO_CONF_DIR}/examples/gc" "${ACCUMULO_CONF_DIR}"
- copy_file "${ACCUMULO_CONF_DIR}/examples/masters" "${ACCUMULO_CONF_DIR}"
- copy_file "${ACCUMULO_CONF_DIR}/examples/monitor" "${ACCUMULO_CONF_DIR}"
- copy_file "${ACCUMULO_CONF_DIR}/examples/tservers" "${ACCUMULO_CONF_DIR}"
- copy_file "${ACCUMULO_CONF_DIR}/examples/tracers" "${ACCUMULO_CONF_DIR}"
+ echo "localhost" > "$conf/gc"
+ echo "localhost" > "$conf/masters"
+ echo "localhost" > "$conf/monitor"
+ echo "localhost" > "$conf/tracers"
+ echo "localhost" > "$conf/tservers"
;;
start)
- "${ACCUMULO_LIBEXEC_DIR}"/cluster.sh start-all
+ verify_config
+ start_all
;;
stop)
- "${ACCUMULO_LIBEXEC_DIR}"/cluster.sh stop-all
+ verify_config
+ stop_all
+ ;;
+ start-non-tservers)
+ verify_config
+ start_all --no-tservers
+ ;;
+ start-tservers)
+ verify_config
+ start_tservers
+ ;;
+ start-here)
+ verify_config
+ start_here
+ ;;
+ stop-tservers)
+ verify_config
+ stop_tservers
+ ;;
+ stop-here)
+ verify_config
+ stop_here
;;
*)
invalid_args "'$1' is an invalid <command>"
http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/bin/accumulo-service
----------------------------------------------------------------------
diff --git a/assemble/bin/accumulo-service b/assemble/bin/accumulo-service
index bf01df9..c9b0a17 100755
--- a/assemble/bin/accumulo-service
+++ b/assemble/bin/accumulo-service
@@ -42,7 +42,10 @@ function invalid_args {
}
function get_host {
- host="$(hostname -s)"
+ host="$(hostname -f)"
+ if [ "$1" == "-s" ]; then
+ host="$(hostname -s)"
+ fi
if [[ -z "$host" ]]; then
host=$(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}' | cut -f1 -d'/')
if [[ $? != 0 ]]; then
@@ -52,8 +55,76 @@ function get_host {
echo "$host"
}
-function main() {
+function rotate_log () {
+ logfile="$1"
+ max_retained="5"
+ if [ -f "$logfile" ]; then
+ while [[ "$max_retained" -gt 1 ]]; do
+ prev=$(( max_retained - 1))
+ [ -f "$logfile.$prev" ] && mv -f "$logfile.$prev" "$logfile.$max_retained"
+ max_retained=$prev
+ done
+ mv -f "$logfile" "$logfile.$max_retained";
+ fi
+}
+
+function start_service() {
+ host="$(get_host -s)"
+
+ if [ -f "$pid_file" ]; then
+ pid=$(cat "$pid_file")
+ if kill -0 "$pid" 2>/dev/null; then
+ echo "$host : $service already running (${pid})"
+ exit 0
+ fi
+ fi
+ echo "Starting $service on $host"
+
+ if [[ $service == "master" ]]; then
+ "${bin}/accumulo" org.apache.accumulo.master.state.SetGoalState NORMAL
+ fi
+
+ outfile="${ACCUMULO_LOG_DIR}/${service}_${host}.out"
+ errfile="${ACCUMULO_LOG_DIR}/${service}_${host}.err"
+ rotate_log "$outfile"
+ rotate_log "$errfile"
+
+ export ACCUMULO_MONITOR_BIND_ALL=${ACCUMULO_MONITOR_BIND_ALL:-"true"}
+ extra_args=""
+ if [[ $service == "monitor" && $ACCUMULO_MONITOR_BIND_ALL == "true" ]]; then
+ extra_args=('--address' '0.0.0.0')
+ fi
+
+ nohup "${bin}/accumulo" "$service" "${extra_args[@]}" >"$outfile" 2>"$errfile" < /dev/null &
+ echo "$!" > "${pid_file}"
+ # Check the max open files limit and selectively warn
+ max_files_open=$(ulimit -n)
+ if [[ -n $max_files_open ]] ; then
+ max_files_recommended=32768
+ if (( max_files_open < max_files_recommended )); then
+ echo "WARN : Max open files on $host is $max_files_open, recommend $max_files_recommended" >&2
+ fi
+ fi
+}
+
+function stop_service() {
+ if [ -f "$pid_file" ]; then
+ echo "Stopping $service on $(get_host -s)";
+ kill -s TERM "$(cat "$pid_file")" 2>/dev/null
+ rm -f "${pid_file}" 2>/dev/null
+ fi
+}
+
+function kill_service() {
+ if [ -f "$pid_file" ]; then
+ echo "Killing $service on $(get_host -s)";
+ kill -s KILL "$(cat "$pid_file")" 2>/dev/null
+ rm -f "${pid_file}" 2>/dev/null
+ fi
+}
+
+function main() {
if [[ -z $1 ]]; then
invalid_args "<service> cannot be empty"
fi
@@ -67,10 +138,18 @@ function main() {
done
bin="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
basedir=$( cd -P "${bin}"/.. && pwd )
+ conf="${basedir}/conf"
+
+ if [ -f "${conf}/accumulo-env.sh" ]; then
+ source "${conf}/accumulo-env.sh"
+ fi
+ ACCUMULO_LOG_DIR="${ACCUMULO_LOG_DIR:-${basedir}/logs}"
- source "$basedir"/libexec/load-env.sh
+ mkdir -p "$ACCUMULO_LOG_DIR" 2>/dev/null
+ mkdir -p "${basedir}/run" 2>/dev/null
- service=$1
+ service="$1"
+ pid_file="${basedir}/run/accumulo-${service}.pid"
case "$service" in
gc|master|monitor|tserver|tracer)
if [[ -z $2 ]]; then
@@ -78,13 +157,13 @@ function main() {
fi
case "$2" in
start)
- "${ACCUMULO_LIBEXEC_DIR}/service.sh" start "$(get_host)" "$1"
+ start_service
;;
stop)
- "${ACCUMULO_LIBEXEC_DIR}/service.sh" stop "$(get_host)" "$service" TERM
+ stop_service
;;
kill)
- "${ACCUMULO_LIBEXEC_DIR}/service.sh" stop "$(get_host)" "$service" KILL
+ kill_service
;;
*)
invalid_args "'$2' is an invalid <command>"
@@ -92,7 +171,7 @@ function main() {
esac
;;
*)
- invalid_args "'$1' is an invalid <service>"
+ invalid_args "'$service' is an invalid <service>"
;;
esac
}
http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/bin/accumulo-util
----------------------------------------------------------------------
diff --git a/assemble/bin/accumulo-util b/assemble/bin/accumulo-util
new file mode 100755
index 0000000..55e89bd
--- /dev/null
+++ b/assemble/bin/accumulo-util
@@ -0,0 +1,686 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+function print_usage {
+ cat <<EOF
+Usage: accumulo-util <command> (<argument> ...)
+
+Commands:
+ create-config Creates Accumulo configuration
+ build-native Builds Accumulo native libraries
+ hadoop-jar Runs 'hadoop jar' command with Accumulo jars
+ gen-monitor-cert Generates Accumulo monitor certficate
+ load-jars-hdfs Loads Accumulo jars in lib/ to HDFS for VFS classloader
+
+EOF
+ exit 1
+}
+
+function create_config_usage() {
+ cat <<EOF
+Usage: accumulo-util create-config [-options]
+where options include (long options not available on all platforms):
+ -d, --dir Alternate directory to setup config files
+ -s, --size Supported sizes: '1GB' '2GB' '3GB' '512MB'
+ -n, --native Configure to use native libraries
+ -j, --jvm Configure to use the jvm
+ -o, --overwrite Overwrite the default config directory
+ -v, --version Specify the Apache Hadoop version supported versions: '1' '2'
+ -k, --kerberos Configure for use with Kerberos
+ -h, --help Print this help message
+EOF
+}
+
+function create_config() {
+ TEMPLATE_CONF_DIR="${conf}/examples/templates"
+ CONF_DIR="${conf}"
+ ACCUMULO_SITE=accumulo-site.xml
+ ACCUMULO_ENV=accumulo-env.sh
+
+ SIZE=
+ TYPE=
+ HADOOP_VERSION=
+ OVERWRITE="0"
+ BASE_DIR=
+ KERBEROS=
+
+ #Execute getopt
+ if [[ $(uname -s) == "Linux" ]]; then
+ args=$(getopt -o "b:d:s:njokv:h" -l "basedir:,dir:,size:,native,jvm,overwrite,kerberos,version:,help" -q -- "$@")
+ else # Darwin, BSD
+ args=$(getopt b:d:s:njokv:h "$@")
+ fi
+
+ #Bad arguments
+ if [[ $? != 0 ]]; then
+ create_config_usage 1>&2
+ exit 1
+ fi
+ eval set -- "${args[@]}"
+
+ for i
+ do
+ case "$i" in
+ -b|--basedir) #Hidden option used to set general.maven.project.basedir for developers
+ BASE_DIR=$2; shift
+ shift;;
+ -d|--dir)
+ CONF_DIR=$2; shift
+ shift;;
+ -s|--size)
+ SIZE=$2; shift
+ shift;;
+ -n|--native)
+ TYPE=native
+ shift;;
+ -j|--jvm)
+ TYPE=jvm
+ shift;;
+ -o|--overwrite)
+ OVERWRITE=1
+ shift;;
+ -v|--version)
+ HADOOP_VERSION=$2; shift
+ shift;;
+ -k|--kerberos)
+ KERBEROS="true"
+ shift;;
+ -h|--help)
+ create_config_usage
+ exit 0
+ shift;;
+ --)
+ shift
+ break;;
+ esac
+ done
+
+ while [[ "${OVERWRITE}" = "0" ]]; do
+ if [[ -e "${CONF_DIR}/${ACCUMULO_ENV}" || -e "${CONF_DIR}/${ACCUMULO_SITE}" ]]; then
+ echo "Warning your current config files in ${CONF_DIR} will be overwritten!"
+ echo
+ echo "How would you like to proceed?:"
+ select CHOICE in 'Continue with overwrite' 'Specify new conf dir'; do
+ if [[ "${CHOICE}" = 'Specify new conf dir' ]]; then
+ echo -n "Please specifiy new conf directory: "
+ read CONF_DIR
+ elif [[ "${CHOICE}" = 'Continue with overwrite' ]]; then
+ OVERWRITE=1
+ fi
+ break
+ done
+ else
+ OVERWRITE=1
+ fi
+ done
+ echo "Copying configuration files to: ${CONF_DIR}"
+
+ #Native 1GB
+ native_1GB_tServer="'-Xmx128m' '-Xms128m'"
+ _1GB_master="'-Xmx128m' '-Xms128m'"
+ _1GB_monitor="'-Xmx64m' '-Xms64m'"
+ _1GB_gc="'-Xmx64m' '-Xms64m'"
+ _1GB_other="'-Xmx128m' '-Xms64m'"
+ _1GB_shell="${_1GB_other}"
+
+ _1GB_memoryMapMax="256M"
+ native_1GB_nativeEnabled="true"
+ _1GB_cacheDataSize="15M"
+ _1GB_cacheIndexSize="40M"
+ _1GB_sortBufferSize="50M"
+ _1GB_waLogMaxSize="256M"
+
+ #Native 2GB
+ native_2GB_tServer="'-Xmx256m' '-Xms256m'"
+ _2GB_master="'-Xmx256m' '-Xms256m'"
+ _2GB_monitor="'-Xmx128m' '-Xms64m'"
+ _2GB_gc="'-Xmx128m' '-Xms128m'"
+ _2GB_other="'-Xmx256m' '-Xms64m'"
+ _2GB_shell="${_2GB_other}"
+
+ _2GB_memoryMapMax="512M"
+ native_2GB_nativeEnabled="true"
+ _2GB_cacheDataSize="30M"
+ _2GB_cacheIndexSize="80M"
+ _2GB_sortBufferSize="50M"
+ _2GB_waLogMaxSize="512M"
+
+ #Native 3GB
+ native_3GB_tServer="'-Xmx1g' '-Xms1g' '-XX:NewSize=500m' '-XX:MaxNewSize=500m'"
+ _3GB_master="'-Xmx1g' '-Xms1g'"
+ _3GB_monitor="'-Xmx1g' '-Xms256m'"
+ _3GB_gc="'-Xmx256m' '-Xms256m'"
+ _3GB_other="'-Xmx1g' '-Xms256m'"
+ _3GB_shell="${_3GB_other}"
+
+ _3GB_memoryMapMax="1G"
+ native_3GB_nativeEnabled="true"
+ _3GB_cacheDataSize="128M"
+ _3GB_cacheIndexSize="128M"
+ _3GB_sortBufferSize="200M"
+ _3GB_waLogMaxSize="1G"
+
+ #Native 512MB
+ native_512MB_tServer="'-Xmx48m' '-Xms48m'"
+ _512MB_master="'-Xmx128m' '-Xms128m'"
+ _512MB_monitor="'-Xmx64m' '-Xms64m'"
+ _512MB_gc="'-Xmx64m' '-Xms64m'"
+ _512MB_other="'-Xmx128m' '-Xms64m'"
+ _512MB_shell="${_512MB_other}"
+
+ _512MB_memoryMapMax="80M"
+ native_512MB_nativeEnabled="true"
+ _512MB_cacheDataSize="7M"
+ _512MB_cacheIndexSize="20M"
+ _512MB_sortBufferSize="50M"
+ _512MB_waLogMaxSize="100M"
+
+ #JVM 1GB
+ jvm_1GB_tServer="'-Xmx384m' '-Xms384m'"
+
+ jvm_1GB_nativeEnabled="false"
+
+ #JVM 2GB
+ jvm_2GB_tServer="'-Xmx768m' '-Xms768m'"
+
+ jvm_2GB_nativeEnabled="false"
+
+ #JVM 3GB
+ jvm_3GB_tServer="'-Xmx2g' '-Xms2g' '-XX:NewSize=1G' '-XX:MaxNewSize=1G'"
+
+ jvm_3GB_nativeEnabled="false"
+
+ #JVM 512MB
+ jvm_512MB_tServer="'-Xmx128m' '-Xms128m'"
+
+ jvm_512MB_nativeEnabled="false"
+
+
+ if [[ -z "${SIZE}" ]]; then
+ echo "Choose the heap configuration:"
+ select DIRNAME in 1GB 2GB 3GB 512MB; do
+ echo "Using '${DIRNAME}' configuration"
+ SIZE=${DIRNAME}
+ break
+ done
+ elif [[ "${SIZE}" != "1GB" && "${SIZE}" != "2GB" && "${SIZE}" != "3GB" && "${SIZE}" != "512MB" ]]; then
+ echo "Invalid memory size"
+ echo "Supported sizes: '1GB' '2GB' '3GB' '512MB'"
+ exit 1
+ fi
+
+ if [[ -z "${TYPE}" ]]; then
+ echo
+ echo "Choose the Accumulo memory-map type:"
+ select TYPENAME in Java Native; do
+ if [[ "${TYPENAME}" == "Native" ]]; then
+ TYPE="native"
+ echo "Don't forget to build the native libraries using the command 'accumulo-util build-native'"
+ elif [[ "${TYPENAME}" == "Java" ]]; then
+ TYPE="jvm"
+ fi
+ echo "Using '${TYPE}' configuration"
+ echo
+ break
+ done
+ fi
+
+ if [[ -z "${HADOOP_VERSION}" ]]; then
+ echo
+ echo "Choose the Apache Hadoop version:"
+ select HADOOP in 'Hadoop 2' 'HDP 2.0/2.1' 'HDP 2.2' 'IOP 4.1'; do
+ if [ "${HADOOP}" == "Hadoop 2" ]; then
+ HADOOP_VERSION="2"
+ elif [ "${HADOOP}" == "HDP 2.0/2.1" ]; then
+ HADOOP_VERSION="HDP2"
+ elif [ "${HADOOP}" == "HDP 2.2" ]; then
+ HADOOP_VERSION="HDP2.2"
+ elif [ "${HADOOP}" == "IOP 4.1" ]; then
+ HADOOP_VERSION="IOP4.1"
+ fi
+ echo "Using Hadoop version '${HADOOP_VERSION}' configuration"
+ echo
+ break
+ done
+ elif [[ "${HADOOP_VERSION}" != "2" && "${HADOOP_VERSION}" != "HDP2" && "${HADOOP_VERSION}" != "HDP2.2" ]]; then
+ echo "Invalid Hadoop version"
+ echo "Supported Hadoop versions: '2', 'HDP2', 'HDP2.2'"
+ exit 1
+ fi
+
+ TRACE_USER="root"
+
+ if [[ ! -z "${KERBEROS}" ]]; then
+ echo
+ read -p "Enter server's Kerberos principal: " PRINCIPAL
+ read -p "Enter server's Kerberos keytab: " KEYTAB
+ TRACE_USER="${PRINCIPAL}"
+ fi
+
+ for var in SIZE TYPE HADOOP_VERSION; do
+ if [[ -z ${!var} ]]; then
+ echo "Invalid $var configuration"
+ exit 1
+ fi
+ done
+
+ TSERVER="${TYPE}_${SIZE}_tServer"
+ MASTER="_${SIZE}_master"
+ MONITOR="_${SIZE}_monitor"
+ GC="_${SIZE}_gc"
+ SHELL="_${SIZE}_shell"
+ OTHER="_${SIZE}_other"
+
+ MEMORY_MAP_MAX="_${SIZE}_memoryMapMax"
+ NATIVE="${TYPE}_${SIZE}_nativeEnabled"
+ CACHE_DATA_SIZE="_${SIZE}_cacheDataSize"
+ CACHE_INDEX_SIZE="_${SIZE}_cacheIndexSize"
+ SORT_BUFFER_SIZE="_${SIZE}_sortBufferSize"
+ WAL_MAX_SIZE="_${SIZE}_waLogMaxSize"
+
+ MAVEN_PROJ_BASEDIR=""
+
+ if [[ ! -z "${BASE_DIR}" ]]; then
+ MAVEN_PROJ_BASEDIR="\n <property>\n <name>general.maven.project.basedir</name>\n <value>${BASE_DIR}</value>\n </property>\n"
+ fi
+
+ mkdir -p "${CONF_DIR}" && cp "${TEMPLATE_CONF_DIR}"/* "${CONF_DIR}"/
+
+ if [[ -f "${CONF_DIR}/examples/client.conf" ]]; then
+ cp "${CONF_DIR}"/examples/client.conf "${CONF_DIR}"/
+ fi
+
+ #Configure accumulo-env.sh
+ sed -e "s/\${tServerHigh_tServerLow}/${!TSERVER}/" \
+ -e "s/\${masterHigh_masterLow}/${!MASTER}/" \
+ -e "s/\${monitorHigh_monitorLow}/${!MONITOR}/" \
+ -e "s/\${gcHigh_gcLow}/${!GC}/" \
+ -e "s/\${shellHigh_shellLow}/${!SHELL}/" \
+ -e "s/\${otherHigh_otherLow}/${!OTHER}/" \
+ "${TEMPLATE_CONF_DIR}/$ACCUMULO_ENV" > "${CONF_DIR}/$ACCUMULO_ENV"
+
+ #Configure accumulo-site.xml
+ sed -e "s/\${memMapMax}/${!MEMORY_MAP_MAX}/" \
+ -e "s/\${nativeEnabled}/${!NATIVE}/" \
+ -e "s/\${cacheDataSize}/${!CACHE_DATA_SIZE}/" \
+ -e "s/\${cacheIndexSize}/${!CACHE_INDEX_SIZE}/" \
+ -e "s/\${sortBufferSize}/${!SORT_BUFFER_SIZE}/" \
+ -e "s/\${waLogMaxSize}/${!WAL_MAX_SIZE}/" \
+ -e "s=\${traceUser}=${TRACE_USER}=" \
+ -e "s=\${mvnProjBaseDir}=${MAVEN_PROJ_BASEDIR}=" "${TEMPLATE_CONF_DIR}/$ACCUMULO_SITE" > "${CONF_DIR}/$ACCUMULO_SITE"
+
+ # If we're not using kerberos, filter out the krb properties
+ if [[ -z "${KERBEROS}" ]]; then
+ sed -e 's/<!-- Kerberos requirements -->/<!-- Kerberos requirements --><!--/' \
+ -e 's/<!-- End Kerberos requirements -->/--><!-- End Kerberos requirements -->/' \
+ "${CONF_DIR}/$ACCUMULO_SITE" > temp
+ mv temp "${CONF_DIR}/$ACCUMULO_SITE"
+ else
+ # Make the substitutions
+ sed -e "s!\${keytab}!${KEYTAB}!" \
+ -e "s!\${principal}!${PRINCIPAL}!" \
+ "${CONF_DIR}/${ACCUMULO_SITE}" > temp
+ mv temp "${CONF_DIR}/${ACCUMULO_SITE}"
+ fi
+
+ # Configure hadoop version
+ if [[ "${HADOOP_VERSION}" == "2" ]]; then
+ sed -e 's/<!-- HDP 2.0 requirements -->/<!-- HDP 2.0 requirements --><!--/' \
+ -e 's/<!-- End HDP 2.0 requirements -->/--><!-- End HDP 2.0 requirements -->/' \
+ "${CONF_DIR}/$ACCUMULO_SITE" > temp
+ mv temp "${CONF_DIR}/$ACCUMULO_SITE"
+ sed -e 's/<!-- HDP 2.2 requirements -->/<!-- HDP 2.2 requirements --><!--/' \
+ -e 's/<!-- End HDP 2.2 requirements -->/--><!-- End HDP 2.2 requirements -->/' \
+ "${CONF_DIR}/$ACCUMULO_SITE" > temp
+ mv temp "${CONF_DIR}/$ACCUMULO_SITE"
+ sed -e 's/<!-- IOP 4.1 requirements -->/<!-- IOP 4.1 requirements --><!--/' \
+ -e 's/<!-- End IOP 4.1 requirements -->/--><!-- End IOP 4.1 requirements -->/' \
+ "${CONF_DIR}/$ACCUMULO_SITE" > temp
+ mv temp "${CONF_DIR}/$ACCUMULO_SITE"
+ elif [[ "${HADOOP_VERSION}" == "HDP2" ]]; then
+ sed -e 's/<!-- Hadoop 2 requirements -->/<!-- Hadoop 2 requirements --><!--/' \
+ -e 's/<!-- End Hadoop 2 requirements -->/--><!-- End Hadoop 2 requirements -->/' \
+ "${CONF_DIR}/$ACCUMULO_SITE" > temp
+ mv temp "${CONF_DIR}/$ACCUMULO_SITE"
+ sed -e 's/<!-- HDP 2.2 requirements -->/<!-- HDP 2.2 requirements --><!--/' \
+ -e 's/<!-- End HDP 2.2 requirements -->/--><!-- End HDP 2.2 requirements -->/' \
+ "${CONF_DIR}/$ACCUMULO_SITE" > temp
+ mv temp "${CONF_DIR}/$ACCUMULO_SITE"
+ sed -e 's/<!-- IOP 4.1 requirements -->/<!-- IOP 4.1 requirements --><!--/' \
+ -e 's/<!-- End IOP 4.1 requirements -->/--><!-- End IOP 4.1 requirements -->/' \
+ "${CONF_DIR}/$ACCUMULO_SITE" > temp
+ mv temp "${CONF_DIR}/$ACCUMULO_SITE"
+ elif [[ "${HADOOP_VERSION}" == "HDP2.2" ]]; then
+ sed -e 's/<!-- Hadoop 2 requirements -->/<!-- Hadoop 2 requirements --><!--/' \
+ -e 's/<!-- End Hadoop 2 requirements -->/--><!-- End Hadoop 2 requirements -->/' \
+ "${CONF_DIR}/$ACCUMULO_SITE" > temp
+ mv temp "${CONF_DIR}/$ACCUMULO_SITE"
+ sed -e 's/<!-- HDP 2.0 requirements -->/<!-- HDP 2.0 requirements --><!--/' \
+ -e 's/<!-- End HDP 2.0 requirements -->/--><!-- End HDP 2.0 requirements -->/' \
+ "${CONF_DIR}/$ACCUMULO_SITE" > temp
+ mv temp "${CONF_DIR}/$ACCUMULO_SITE"
+ sed -e 's/<!-- IOP 4.1 requirements -->/<!-- IOP 4.1 requirements --><!--/' \
+ -e 's/<!-- End IOP 4.1 requirements -->/--><!-- End IOP 4.1 requirements -->/' \
+ "${CONF_DIR}/$ACCUMULO_SITE" > temp
+ mv temp "${CONF_DIR}/$ACCUMULO_SITE"
+ elif [[ "${HADOOP_VERSION}" == "IOP4.1" ]]; then
+ sed -e 's/<!-- Hadoop 2 requirements -->/<!-- Hadoop 2 requirements --><!--/' \
+ -e 's/<!-- End Hadoop 2 requirements -->/--><!-- End Hadoop 2 requirements -->/' \
+ "${CONF_DIR}/$ACCUMULO_SITE" > temp
+ mv temp "${CONF_DIR}/$ACCUMULO_SITE"
+ sed -e 's/<!-- HDP 2.0 requirements -->/<!-- HDP 2.0 requirements --><!--/' \
+ -e 's/<!-- End HDP 2.0 requirements -->/--><!-- End HDP 2.0 requirements -->/' \
+ "${CONF_DIR}/$ACCUMULO_SITE" > temp
+ mv temp "${CONF_DIR}/$ACCUMULO_SITE"
+ sed -e 's/<!-- HDP 2.2 requirements -->/<!-- HDP 2.2 requirements --><!--/' \
+ -e 's/<!-- End HDP 2.2 requirements -->/--><!-- End HDP 2.2 requirements -->/' \
+ "${CONF_DIR}/$ACCUMULO_SITE" > temp
+ mv temp "${CONF_DIR}/$ACCUMULO_SITE"
+ fi
+
+ #Additional setup steps for native configuration.
+ if [[ ${TYPE} == native ]]; then
+ if [[ $(uname) == Linux ]]; then
+ if [[ -z $HADOOP_PREFIX ]]; then
+ echo "WARNING: HADOOP_PREFIX not set, cannot automatically configure LD_LIBRARY_PATH to include Hadoop native libraries"
+ else
+ NATIVE_LIB=$(readlink -ef "$(dirname "$(for x in $(find "$HADOOP_PREFIX" -name libhadoop.so); do ld "$x" 2>/dev/null && echo "$x" && break; done)" 2>>/dev/null)" 2>>/dev/null)
+ if [[ -z $NATIVE_LIB ]]; then
+ echo -e "WARNING: The Hadoop native libraries could not be found for your sytem in: $HADOOP_PREFIX"
+ else
+ sed "/# Should the monitor/ i export LD_LIBRARY_PATH=${NATIVE_LIB}:\${LD_LIBRARY_PATH}" "${CONF_DIR}/$ACCUMULO_ENV" > temp
+ mv temp "${CONF_DIR}/$ACCUMULO_ENV"
+ echo -e "Added ${NATIVE_LIB} to the LD_LIBRARY_PATH"
+ fi
+ fi
+ fi
+ echo -e "Please remember to compile the Accumulo native libraries using the command 'accumulo-util build-native' and to set the LD_LIBRARY_PATH variable in the ${CONF_DIR}/accumulo-env.sh if needed."
+ fi
+
+ echo "Setup complete"
+}
+
+function build_native() {
+ native_tarball="$basedir/lib/accumulo-native.tar.gz"
+ final_native_target="$basedir/lib/native"
+
+ if [[ ! -f $native_tarball ]]; then
+ echo "Could not find native code artifact: ${native_tarball}";
+ exit 1
+ fi
+
+ # Make the destination for the native library
+ mkdir -p "${final_native_target}" || exit 1
+
+ # Make a directory for us to unpack the native source into
+ TMP_DIR=$(mktemp -d /tmp/accumulo-native.XXXX) || exit 1
+
+ # Unpack the tarball to our temp directory
+ if ! tar xf "${native_tarball}" -C "${TMP_DIR}"
+ then
+ echo "Failed to unpack native tarball to ${TMP_DIR}"
+ exit 1
+ fi
+
+ # Move to the first (only) directory in our unpacked tarball
+ native_dir=$(find "${TMP_DIR}" -maxdepth 1 -mindepth 1 -type d)
+
+ cd "${native_dir}" || exit 1
+
+ # Make the native library
+ export USERFLAGS="$*"
+ make || { echo 'Make failed!'; exit 1; }
+
+ # "install" the artifact
+ cp libaccumulo.* "${final_native_target}" || exit 1
+
+ # Clean up our temp directory
+ rm -rf "${TMP_DIR}"
+
+ echo "Successfully installed native library"
+}
+
+function gen_monitor_cert() {
+ if [[ -z "$JAVA_HOME" || -d "$JAVA_HOME" ]]; then
+ echo "JAVA_HOME=${JAVA_HOME} must be set and exist"
+ exit 1
+ fi
+
+ ALIAS="default"
+ KEYPASS=$(LC_CTYPE=C tr -dc '#-~' < /dev/urandom | tr -d '<>&' | head -c 20)
+ STOREPASS=$(LC_CTYPE=C tr -dc '#-~' < /dev/urandom | tr -d '<>&' | head -c 20)
+ KEYSTOREPATH="${conf}/keystore.jks"
+ TRUSTSTOREPATH="${conf}/conf/cacerts.jks"
+ CERTPATH="${conf}/server.cer"
+
+ if [[ -e "$KEYSTOREPATH" ]]; then
+ rm -i "$KEYSTOREPATH"
+ if [[ -e "$KEYSTOREPATH" ]]; then
+ echo "KeyStore already exists, exiting"
+ exit 1
+ fi
+ fi
+ if [[ -e "$TRUSTSTOREPATH" ]]; then
+ rm -i "$TRUSTSTOREPATH"
+ if [[ -e "$TRUSTSTOREPATH" ]]; then
+ echo "TrustStore already exists, exiting"
+ exit 2
+ fi
+ fi
+ if [[ -e "$CERTPATH" ]]; then
+ rm -i "$CERTPATH"
+ if [[ -e "$CERTPATH" ]]; then
+ echo "Certificate already exists, exiting"
+ exit 3
+ fi
+ fi
+
+ "${JAVA_HOME}/bin/keytool" -genkey -alias "$ALIAS" -keyalg RSA -keypass "$KEYPASS" -storepass "$KEYPASS" -keystore "$KEYSTOREPATH"
+ "${JAVA_HOME}/bin/keytool" -export -alias "$ALIAS" -storepass "$KEYPASS" -file "$CERTPATH" -keystore "$KEYSTOREPATH"
+ "${JAVA_HOME}/bin/keytool" -import -v -trustcacerts -alias "$ALIAS" -file "$CERTPATH" -keystore "$TRUSTSTOREPATH" -storepass "$STOREPASS" <<< "yes"
+
+ echo
+ echo "keystore and truststore generated. now add the following to accumulo-site.xml:"
+ echo
+ echo " <property>"
+ echo " <name>monitor.ssl.keyStore</name>"
+ echo " <value>$KEYSTOREPATH</value>"
+ echo " </property>"
+ echo " <property>"
+ echo " <name>monitor.ssl.keyStorePassword</name>"
+ echo " <value>$KEYPASS</value>"
+ echo " </property>"
+ echo " <property>"
+ echo " <name>monitor.ssl.trustStore</name>"
+ echo " <value>$TRUSTSTOREPATH</value>"
+ echo " </property>"
+ echo " <property>"
+ echo " <name>monitor.ssl.trustStorePassword</name>"
+ echo " <value>$STOREPASS</value>"
+ echo " </property>"
+ echo
+}
+
+function load_jars_hdfs() {
+ export ACCUMULO_HOME="$basedir"
+
+ if [ -f "${conf}/accumulo-env.sh" ]; then
+ source "$conf/accumulo-env.sh"
+ fi
+ if [ -z "$HADOOP_PREFIX" ]; then
+ echo "HADOOP_PREFIX is not set. Please make sure it's set globally or in $conf/accumulo-env.sh"
+ exit 1
+ fi
+
+ # Find the system context directory in HDFS
+ SYSTEM_CONTEXT_HDFS_DIR=$(grep -A1 "general.vfs.classpaths" "$conf/accumulo-site.xml" | tail -1 | perl -pe 's/\s+<value>//; s/<\/value>//; s/,.+$//; s|[^/]+$||; print $ARGV[1]')
+
+ if [ -z "$SYSTEM_CONTEXT_HDFS_DIR" ]
+ then
+ echo "Your accumulo-site.xml file is not set up for the HDFS Classloader. Please add the following to your accumulo-site.xml file where ##CLASSPATH## is one of the following formats:"
+ echo "A single directory: hdfs://host:port/directory/"
+ echo "A single directory with a regex: hdfs://host:port/directory/.*.jar"
+ echo "Multiple directories: hdfs://host:port/directory/.*.jar,hdfs://host:port/directory2/"
+ echo ""
+ echo "<property>"
+ echo " <name>general.vfs.classpaths</name>"
+ echo " <value>##CLASSPATH##</value>"
+ echo " <description>location of the jars for the default (system) context</description>"
+ echo "</property>"
+ exit 1
+ fi
+
+ # Create the system context directy in HDFS if it does not exist
+ "$HADOOP_PREFIX/bin/hadoop" fs -ls "$SYSTEM_CONTEXT_HDFS_DIR" > /dev/null
+ if [[ $? != 0 ]]; then
+ "$HADOOP_PREFIX/bin/hadoop" fs -mkdir "$SYSTEM_CONTEXT_HDFS_DIR" > /dev/null
+ if [[ $? != 0 ]]; then
+ echo "Unable to create classpath directory at $SYSTEM_CONTEXT_HDFS_DIR"
+ exit 1
+ fi
+ fi
+
+ # Replicate to all tservers to avoid network contention on startup
+ TSERVERS=${conf}/tservers
+ NUM_TSERVERS=$(egrep -v '(^#|^\s*$)' "$TSERVERS" | wc -l)
+
+ #let each datanode service around 50 clients
+ REP=$(( NUM_TSERVERS / 50 ))
+ (( REP < 3 )) && REP=3
+
+ # Copy all jars in lib to the system context directory
+ "$HADOOP_PREFIX/bin/hadoop" fs -moveFromLocal "$lib"/*.jar "$SYSTEM_CONTEXT_HDFS_DIR" > /dev/null
+ "$HADOOP_PREFIX/bin/hadoop" fs -setrep -R $REP "$SYSTEM_CONTEXT_HDFS_DIR" > /dev/null
+
+ # We need some of the jars in lib, copy them back out and remove them from the system context dir
+ "$HADOOP_PREFIX/bin/hadoop" fs -copyToLocal "$SYSTEM_CONTEXT_HDFS_DIR/commons-vfs2.jar" "$lib/." > /dev/null
+ "$HADOOP_PREFIX/bin/hadoop" fs -rm "$SYSTEM_CONTEXT_HDFS_DIR/commons-vfs2.jar" > /dev/null
+ "$HADOOP_PREFIX/bin/hadoop" fs -copyToLocal "$SYSTEM_CONTEXT_HDFS_DIR/accumulo-start.jar" "$lib/." > /dev/null
+ "$HADOOP_PREFIX/bin/hadoop" fs -rm "$SYSTEM_CONTEXT_HDFS_DIR/accumulo-start.jar" > /dev/null
+ "$HADOOP_PREFIX/bin/hadoop" fs -copyToLocal "$SYSTEM_CONTEXT_HDFS_DIR/slf4j*.jar" "$lib/." > /dev/null
+ "$HADOOP_PREFIX/bin/hadoop" fs -rm "$SYSTEM_CONTEXT_HDFS_DIR/slf4j*.jar" > /dev/null
+ for f in $(grep -v '^#' "${conf}/tservers")
+ do
+ rsync -ra --delete "$ACCUMULO_HOME" "$(dirname "$ACCUMULO_HOME")"
+ done
+}
+
+function hadoop_jar() {
+ if [ -f "${conf}/accumulo-env.sh" ]; then
+ source "$conf/accumulo-env.sh"
+ fi
+ if [ -z "$HADOOP_PREFIX" ]; then
+ echo "HADOOP_PREFIX is not set. Please make sure it's set globally or in $conf/accumulo-env.sh"
+ exit 1
+ fi
+ if [ -z "$ZOOKEEPER_HOME" ]; then
+ echo "ZOOKEEPER_HOME is not set. Please make sure it's set globally or in $conf/accumulo-env.sh"
+ exit 1
+ fi
+
+ ZOOKEEPER_CMD="ls -1 $ZOOKEEPER_HOME/zookeeper-[0-9]*[^csn].jar "
+ if [[ $(eval "$ZOOKEEPER_CMD" | wc -l) -ne 1 ]] ; then
+ echo "Not exactly one zookeeper jar in $ZOOKEEPER_HOME"
+ exit 1
+ fi
+ ZOOKEEPER_LIB=$(eval "$ZOOKEEPER_CMD")
+
+ CORE_LIB="${lib}/accumulo-core.jar"
+ FATE_LIB="${lib}/accumulo-fate.jar"
+ THRIFT_LIB="${lib}/libthrift.jar"
+ JCOMMANDER_LIB="${lib}/jcommander.jar"
+ COMMONS_VFS_LIB="${lib}/commons-vfs2.jar"
+ GUAVA_LIB="${lib}/guava.jar"
+ HTRACE_LIB="${lib}/htrace-core.jar"
+
+ USERJARS=" "
+ for arg in "$@"; do
+ if [ "$arg" != "-libjars" -a -z "$TOOLJAR" ]; then
+ TOOLJAR="$arg"
+ shift
+ elif [ "$arg" != "-libjars" -a -z "$CLASSNAME" ]; then
+ CLASSNAME="$arg"
+ shift
+ elif [ -z "$USERJARS" ]; then
+ USERJARS=$(echo "$arg" | tr "," " ")
+ shift
+ elif [ "$arg" = "-libjars" ]; then
+ USERJARS=""
+ shift
+ else
+ break
+ fi
+ done
+
+ LIB_JARS="$THRIFT_LIB,$CORE_LIB,$FATE_LIB,$ZOOKEEPER_LIB,$JCOMMANDER_LIB,$COMMONS_VFS_LIB,$GUAVA_LIB,$HTRACE_LIB"
+ H_JARS="$THRIFT_LIB:$CORE_LIB:$FATE_LIB:$ZOOKEEPER_LIB:$JCOMMANDER_LIB:$COMMONS_VFS_LIB:$GUAVA_LIB:$HTRACE_LIB"
+
+ for jar in $USERJARS; do
+ LIB_JARS="$LIB_JARS,$jar"
+ H_JARS="$H_JARS:$jar"
+ done
+ export HADOOP_CLASSPATH="$H_JARS:$HADOOP_CLASSPATH"
+
+ if [[ -z "$CLASSNAME" || -z "$TOOLJAR" ]]; then
+ echo "Usage: accumulo-util hadoop-jar path/to/myTool.jar my.tool.class.Name [-libjars my1.jar,my2.jar]" 1>&2
+ exit 1
+ fi
+
+ #echo USERJARS=$USERJARS
+ #echo CLASSNAME=$CLASSNAME
+ #echo HADOOP_CLASSPATH=$HADOOP_CLASSPATH
+ #echo exec "$HADOOP_PREFIX/bin/hadoop" jar "$TOOLJAR" "$CLASSNAME" -libjars \"$LIB_JARS\" $ARGS
+ exec "$HADOOP_PREFIX/bin/hadoop" jar "$TOOLJAR" "$CLASSNAME" -libjars "$LIB_JARS" "$@"
+}
+
+function main() {
+ SOURCE="${BASH_SOURCE[0]}"
+ while [ -h "${SOURCE}" ]; do
+ bin="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${bin}/${SOURCE}"
+ done
+ bin="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ basedir=$( cd -P "${bin}"/.. && pwd )
+ conf="${basedir}/conf"
+ lib="${basedir}/lib"
+
+ case "$1" in
+ create-config)
+ create_config "${@:2}"
+ ;;
+ build-native)
+ build_native "${@:2}"
+ ;;
+ hadoop-jar)
+ hadoop_jar "${@:2}"
+ ;;
+ gen-monitor-cert)
+ gen_monitor_cert
+ ;;
+ load-jars-hdfs)
+ load_jars_hdfs
+ ;;
+ *)
+ echo -e "'$1' is an invalid <command>\n"
+ print_usage 1>&2
+ exit 1
+ ;;
+ esac
+}
+
+main "$@"
http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/conf/examples/gc
----------------------------------------------------------------------
diff --git a/assemble/conf/examples/gc b/assemble/conf/examples/gc
deleted file mode 100644
index 63fb8bb..0000000
--- a/assemble/conf/examples/gc
+++ /dev/null
@@ -1,16 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-localhost
http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/conf/examples/generic_logger.properties
----------------------------------------------------------------------
diff --git a/assemble/conf/examples/generic_logger.properties b/assemble/conf/examples/generic_logger.properties
index ff589d9..b4db23a 100644
--- a/assemble/conf/examples/generic_logger.properties
+++ b/assemble/conf/examples/generic_logger.properties
@@ -15,7 +15,7 @@
# Write out everything at the DEBUG level to the debug log
log4j.appender.A2=org.apache.log4j.RollingFileAppender
-log4j.appender.A2.File=${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}_${accumulo.service.instance}${org.apache.accumulo.core.ip.localhost.hostname}_fromprops.debug.log
+log4j.appender.A2.File=${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}${accumulo.service.instance}_${org.apache.accumulo.core.ip.localhost.hostname}_fromprops.debug.log
log4j.appender.A2.MaxFileSize=1000MB
log4j.appender.A2.MaxBackupIndex=10
log4j.appender.A2.Threshold=DEBUG
@@ -24,7 +24,7 @@ log4j.appender.A2.layout.ConversionPattern=%d{ISO8601} [%-8c{2}] %-5p: %m%n
# Write out INFO and higher to the regular log
log4j.appender.A3=org.apache.log4j.RollingFileAppender
-log4j.appender.A3.File=${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}_${accumulo.service.instance}${org.apache.accumulo.core.ip.localhost.hostname}_fromprops.log
+log4j.appender.A3.File=${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}${accumulo.service.instance}_${org.apache.accumulo.core.ip.localhost.hostname}_fromprops.log
log4j.appender.A3.MaxFileSize=1000MB
log4j.appender.A3.MaxBackupIndex=10
log4j.appender.A3.Threshold=INFO
@@ -36,7 +36,7 @@ log4j.appender.A3.layout.ConversionPattern=%d{ISO8601} [%-8c{2}] %-5p: %m%n
log4j.appender.ASYNC=org.apache.accumulo.core.util.AsyncSocketAppender
log4j.appender.ASYNC.RemoteHost=${org.apache.accumulo.core.host.log}
log4j.appender.ASYNC.Port=${org.apache.accumulo.core.host.log.port}
-log4j.appender.ASYNC.Application=${org.apache.accumulo.core.application}:${org.apache.accumulo.core.ip.localhost.hostname}
+log4j.appender.ASYNC.Application=${org.apache.accumulo.core.application}${accumulo.service.instance}:${org.apache.accumulo.core.ip.localhost.hostname}
log4j.appender.ASYNC.Threshold=WARN
# Log accumulo events to the debug, normal and remote logs.
http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/conf/examples/generic_logger.xml
----------------------------------------------------------------------
diff --git a/assemble/conf/examples/generic_logger.xml b/assemble/conf/examples/generic_logger.xml
index a71effe..23a1598 100644
--- a/assemble/conf/examples/generic_logger.xml
+++ b/assemble/conf/examples/generic_logger.xml
@@ -20,7 +20,7 @@
<!-- Write out everything at the DEBUG level to the debug log -->
<appender name="A2" class="org.apache.log4j.RollingFileAppender">
- <param name="File" value="${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}_${accumulo.service.instance}${org.apache.accumulo.core.ip.localhost.hostname}.debug.log"/>
+ <param name="File" value="${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}${accumulo.service.instance}_${org.apache.accumulo.core.ip.localhost.hostname}.debug.log"/>
<param name="MaxFileSize" value="1000MB"/>
<param name="MaxBackupIndex" value="10"/>
<param name="Threshold" value="DEBUG"/>
@@ -31,7 +31,7 @@
<!-- Write out INFO and higher to the regular log -->
<appender name="A3" class="org.apache.log4j.RollingFileAppender">
- <param name="File" value="${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}_${accumulo.service.instance}${org.apache.accumulo.core.ip.localhost.hostname}.log"/>
+ <param name="File" value="${org.apache.accumulo.core.dir.log}/${org.apache.accumulo.core.application}${accumulo.service.instance}_${org.apache.accumulo.core.ip.localhost.hostname}.log"/>
<param name="MaxFileSize" value="1000MB"/>
<param name="MaxBackupIndex" value="10"/>
<param name="Threshold" value="INFO"/>
@@ -44,7 +44,7 @@
<appender name="N1" class="org.apache.log4j.net.SocketAppender">
<param name="remoteHost" value="${org.apache.accumulo.core.host.log}"/>
<param name="port" value="${org.apache.accumulo.core.host.log.port}"/>
- <param name="application" value="${org.apache.accumulo.core.application}:${org.apache.accumulo.core.ip.localhost.hostname}"/>
+ <param name="application" value="${org.apache.accumulo.core.application}${accumulo.service.instance}:${org.apache.accumulo.core.ip.localhost.hostname}"/>
<param name="Threshold" value="WARN"/>
</appender>
http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/conf/examples/masters
----------------------------------------------------------------------
diff --git a/assemble/conf/examples/masters b/assemble/conf/examples/masters
deleted file mode 100644
index 63fb8bb..0000000
--- a/assemble/conf/examples/masters
+++ /dev/null
@@ -1,16 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-localhost
http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/conf/examples/monitor
----------------------------------------------------------------------
diff --git a/assemble/conf/examples/monitor b/assemble/conf/examples/monitor
deleted file mode 100644
index 63fb8bb..0000000
--- a/assemble/conf/examples/monitor
+++ /dev/null
@@ -1,16 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-localhost
http://git-wip-us.apache.org/repos/asf/accumulo/blob/9a3cc5c4/assemble/conf/examples/templates/accumulo-env.sh
----------------------------------------------------------------------
diff --git a/assemble/conf/examples/templates/accumulo-env.sh b/assemble/conf/examples/templates/accumulo-env.sh
new file mode 100644
index 0000000..e6a360c
--- /dev/null
+++ b/assemble/conf/examples/templates/accumulo-env.sh
@@ -0,0 +1,64 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+## Before accumulo-env.sh is loaded, these environment variables are set and can be used in this file:
+
+# ACCUMULO_CMD - Command that is being called such as tserver, master, etc.
+# ACCUMULO_HOME - Root directory of Accumulo installation
+# ACCUMULO_CONF_DIR - Directory containing Accumulo configuration
+
+############################
+# Variables that must be set
+############################
+
+## JVM options set for all processes. Extra options can be passed in by setting ACCUMULO_JAVA_OPTS to an array of options.
+JAVA_OPTS=("${ACCUMULO_JAVA_OPTS[@]}" '-XX:+UseConcMarkSweepGC' '-XX:CMSInitiatingOccupancyFraction=75' '-XX:+CMSClassUnloadingEnabled'
+'-XX:OnOutOfMemoryError=kill -9 %p' '-XX:-OmitStackTraceInFastThrow' '-Djava.net.preferIPv4Stack=true'
+'-Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl')
+
+## JVM options set for individual applications
+case "$ACCUMULO_CMD" in
+master) JAVA_OPTS=("${JAVA_OPTS[@]}" ${masterHigh_masterLow}) ;;
+gc) JAVA_OPTS=("${JAVA_OPTS[@]}" ${gcHigh_gcLow}) ;;
+tserver) JAVA_OPTS=("${JAVA_OPTS[@]}" ${tServerHigh_tServerLow}) ;;
+monitor) JAVA_OPTS=("${JAVA_OPTS[@]}" ${monitorHigh_monitorLow}) ;;
+shell) JAVA_OPTS=("${JAVA_OPTS[@]}" ${shellHigh_shellLow}) ;;
+*) JAVA_OPTS=("${JAVA_OPTS[@]}" ${otherHigh_otherLow}) ;;
+esac
+export JAVA_OPTS
+
+## Accumulo logs directory. Referenced by logger config.
+export ACCUMULO_LOG_DIR="${ACCUMULO_LOG_DIR:-$ACCUMULO_HOME/logs}"
+## Hadoop installation
+export HADOOP_PREFIX="${HADOOP_PREFIX:-/path/to/hadoop}"
+## Hadoop configuration
+export HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-${HADOOP_PREFIX}/etc/hadoop}"
+## Zookeeper installation
+export ZOOKEEPER_HOME="${ZOOKEEPER_HOME:-/path/to/zookeeper}"
+## See HADOOP-7154 and ACCUMULO-847
+export MALLOC_ARENA_MAX=${MALLOC_ARENA_MAX:-1}
+
+###############################################
+# Variables that are optional. Uncomment to set
+###############################################
+
+## Specifies command that will be placed before calls to Java in accumulo script
+# export ACCUMULO_JAVA_PREFIX=""
+## Optionally look for hadoop and accumulo native libraries for your platform in additional
+## directories. (Use DYLD_LIBRARY_PATH on Mac OS X.) May not be necessary for Hadoop 2.x or
+## using an RPM that installs to the correct system library directory.
+# export LD_LIBRARY_PATH=${HADOOP_PREFIX}/lib/native/${PLATFORM}:${LD_LIBRARY_PATH}