You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by bi...@apache.org on 2015/01/27 18:39:19 UTC
incubator-slider git commit: SLIDER-756 create accumulo-slider client
script
Repository: incubator-slider
Updated Branches:
refs/heads/develop f5b0940f0 -> 96f890b1b
SLIDER-756 create accumulo-slider client script
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/96f890b1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/96f890b1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/96f890b1
Branch: refs/heads/develop
Commit: 96f890b1b13985567505c10492326062f3e6dffd
Parents: f5b0940
Author: Billie Rinaldi <bi...@gmail.com>
Authored: Tue Jan 27 09:38:50 2015 -0800
Committer: Billie Rinaldi <bi...@gmail.com>
Committed: Tue Jan 27 09:38:59 2015 -0800
----------------------------------------------------------------------
app-packages/accumulo/README.md | 53 ++-
app-packages/accumulo/accumulo-slider | 53 +++
app-packages/accumulo/accumulo-slider.py | 378 +++++++++++++++++++
app-packages/accumulo/pom.xml | 2 +
app-packages/accumulo/src/assembly/accumulo.xml | 8 +
.../funtest/accumulo/AccumuloProxyIT.groovy | 2 +-
.../funtest/accumulo/AccumuloScriptIT.groovy | 88 +++++
.../funtest/accumulo/AccumuloSliderShell.groovy | 52 +++
.../slider/funtest/framework/ShellBase.groovy | 323 ++++++++++++++++
.../slider/funtest/framework/SliderShell.groovy | 305 +--------------
10 files changed, 950 insertions(+), 314 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96f890b1/app-packages/accumulo/README.md
----------------------------------------------------------------------
diff --git a/app-packages/accumulo/README.md b/app-packages/accumulo/README.md
index e44870e..9395c5a 100644
--- a/app-packages/accumulo/README.md
+++ b/app-packages/accumulo/README.md
@@ -15,20 +15,18 @@
limitations under the License.
-->
-# How to create a Slider package for Accumulo?
+# Creating a Slider app package for Accumulo
mvn clean package -DskipTests -Paccumulo-app-package-maven
-
+
OR
mvn clean package -DskipTests -Paccumulo-app-package -Dpkg.version=1.6.1 \
-Dpkg.name=accumulo-1.6.1-bin.tar.gz -Dpkg.src=/local/path/to/tarball
-App package can be found in
+The app package can be found in
app-packages/accumulo/target/slider-accumulo-app-package-*.zip
-
-
In the first case, the version number of the app package will match the
slider version, and in the second case it will match the `pkg.version`
@@ -38,26 +36,49 @@ Verify the content using
zip -Tv slider-accumulo-app-package*.zip
-`appConfig-default.json` and `resources-default.json` are not required to be packaged.
-These files are included as reference configuration for Slider apps and are suitable
-for a one-node cluster.
+`appConfig-default.json` and `resources-default.json` are not required to be
+packaged. These files are included as reference configuration for Slider apps
+and are suitable for a one-node cluster.
In the maven packaging case, the version of Accumulo used for the app package
can be adjusted by adding a flag such as
-Daccumulo.version=1.5.1
-**Note that the LICENSE.txt and NOTICE.txt that are bundled with the app
+**Note** that the LICENSE.txt and NOTICE.txt that are bundled with the app
package are designed for Accumulo 1.6.0 only and may need to be modified to be
applicable for other versions of the app package.
-Note also that the sample `appConfig-default.json` provided only works with Accumulo 1.6.
-For Accumulo 1.5 the instance.volumes property must be replaced with
-`instance.dfs.dir` (and it cannot use the provided variable `${DEFAULT_DATA_DIR}`
-which is an HDFS URI).
+Note also that the sample `appConfig-default.json` provided only works with
+Accumulo 1.6 or greater. For Accumulo 1.5 the instance.volumes property must be
+replaced with `instance.dfs.dir` (and it cannot use the provided variable
+`${DEFAULT_DATA_DIR}` which is an HDFS URI).
A less descriptive file name can be specified with
-`-Dapp.package.name=accumulo_160` which would create a file `accumulo_160.zip`
+`-Dapp.package.name=accumulo_160` which would create a file `accumulo_160.zip`.
+
+# Installing an Accumulo Client
+
+The Accumulo app package provides scripts to assist in client interactions with
+an Accumulo instance running on Slider. These can be extracted as follows.
+
+ unzip slider-accumulo-app-package*zip accumulo-slider
+ unzip slider-accumulo-app-package*zip accumulo-slider.py
+
+To install an Accumulo client, use the following command:
+
+ SLIDER_HOME=</path/to/slider> ./accumulo-slider --app <clusterName> install <dir>
+
+If the SLIDER_CONF_DIR is not at $SLIDER_HOME/conf, it should also be set.
+The dir specified will then contain an Accumulo installation that can be used
+to connect to the cluster.
+
+Examples of other commands that may be issued are:
+
+ SLIDER_HOME=</path/to/slider> ACCUMULO_HOME=</path/to/accumulo> ./accumulo-slider --app <clusterName> shell
+ SLIDER_HOME=</path/to/slider> ACCUMULO_HOME=</path/to/accumulo> ./accumulo-slider --app <clusterName> --appconf <accumulo_conf_dir> getconf
+ SLIDER_HOME=</path/to/slider> ACCUMULO_HOME=</path/to/accumulo> ./accumulo-slider --app <clusterName> quicklinks
+ SLIDER_HOME=</path/to/slider> ACCUMULO_HOME=</path/to/accumulo> ./accumulo-slider --app <clusterName> proxies
# Building Native Libraries
@@ -88,8 +109,8 @@ currently reside may have restrictions on the import, possession, use, and/or
re-export to another country, of encryption software. BEFORE using any
encryption software, please check your country's laws, regulations and
policies concerning the import, possession, or use, and re-export of encryption
-software, to see if this is permitted. See [http://www.wassenaar.org/](http://www.wassenaar.org/) for more
-information.
+software, to see if this is permitted. See
+[http://www.wassenaar.org/](http://www.wassenaar.org/) for more information.
The U.S. Government Department of Commerce, Bureau of Industry and Security
(BIS), has classified this software as Export Commodity Control Number (ECCN)
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96f890b1/app-packages/accumulo/accumulo-slider
----------------------------------------------------------------------
diff --git a/app-packages/accumulo/accumulo-slider b/app-packages/accumulo/accumulo-slider
new file mode 100644
index 0000000..8f9cb06
--- /dev/null
+++ b/app-packages/accumulo/accumulo-slider
@@ -0,0 +1,53 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Resolve links - $0 may be a softlink
+PRG="${0}"
+
+while [ -h "${PRG}" ]; do
+ ls=`ls -ld "${PRG}"`
+ link=`expr "$ls" : '.*-> \(.*\)$'`
+ if expr "$link" : '/.*' > /dev/null; then
+ PRG="$link"
+ else
+ PRG=`dirname "${PRG}"`/"$link"
+ fi
+done
+
+# find python >= 2.6
+if [ -a /usr/bin/python2.6 ]; then
+ PYTHON=/usr/bin/python2.6
+fi
+
+if [ -z "$PYTHON" ]; then
+ PYTHON=/usr/bin/python
+fi
+
+# check for version
+majversion=`$PYTHON -V 2>&1 | awk '{print $2}' | cut -d'.' -f1`
+minversion=`$PYTHON -V 2>&1 | awk '{print $2}' | cut -d'.' -f2`
+numversion=$(( 10 * $majversion + $minversion))
+if (( $numversion < 26 )); then
+ echo "Need python version > 2.6"
+ exit 1
+fi
+
+ACCUMULO_SLIDER_BIN_DIR=`dirname ${PRG}`
+
+$PYTHON ${ACCUMULO_SLIDER_BIN_DIR}/accumulo-slider.py $@
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96f890b1/app-packages/accumulo/accumulo-slider.py
----------------------------------------------------------------------
diff --git a/app-packages/accumulo/accumulo-slider.py b/app-packages/accumulo/accumulo-slider.py
new file mode 100644
index 0000000..1eccf59
--- /dev/null
+++ b/app-packages/accumulo/accumulo-slider.py
@@ -0,0 +1,378 @@
+#!/usr/bin/python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+import json
+import glob
+import tempfile
+import subprocess
+import shutil
+
+SLIDER_DIR = os.getenv('SLIDER_HOME', None)
+if SLIDER_DIR == None or (not os.path.exists(SLIDER_DIR)):
+ print "Unable to find SLIDER_HOME. Please configure SLIDER_HOME before running accumulo-slider"
+ sys.exit(1)
+SLIDER_CMD = os.path.join(SLIDER_DIR, 'bin', 'slider.py')
+
+TMP_DIR = os.path.join(tempfile.gettempdir(), "accumulo-slider-tmp."+str(os.getpid()))
+
+CMD_OPTS = {}
+
+def call(cmd):
+ print "Running: " + " ".join(cmd)
+ retcode = subprocess.call(cmd)
+ if retcode != 0:
+ raise Exception("return code from running %s was %d" % (cmd[0], retcode))
+
+def exec_accumulo_command(command, args=[]):
+ ACCUMULO_DIR = os.getenv('ACCUMULO_HOME', None)
+ if ACCUMULO_DIR == None or (not os.path.exists(ACCUMULO_DIR)):
+ print "Unable to find ACCUMULO_HOME. Please configure ACCUMULO_HOME before running accumulo-slider " + command
+ sys.exit(1)
+ ACCUMULO_CMD = os.path.join(ACCUMULO_DIR, 'bin', 'accumulo')
+
+ confdir = get_all_conf()
+ if command == 'shell' or command == 'admin':
+ cmd = [ACCUMULO_CMD, command, '--config-file', os.path.join(confdir, 'client.conf')] + list(args)
+ else:
+ cmd = [ACCUMULO_CMD, command] + list(args)
+ print "Setting ACCUMULO_CONF_DIR=" + confdir
+ os.putenv("ACCUMULO_CONF_DIR", confdir)
+ call(cmd)
+
+def exec_tool_command(jarfile, mainclass, args=[]):
+ ACCUMULO_DIR = os.getenv('ACCUMULO_HOME', None)
+ if ACCUMULO_DIR == None or (not os.path.exists(ACCUMULO_DIR)):
+ print "Unable to find ACCUMULO_HOME. Please configure ACCUMULO_HOME before running accumulo-slider tool"
+ sys.exit(1)
+ TOOL_CMD = os.path.join(ACCUMULO_DIR, 'bin', 'tool.sh')
+
+ confdir = get_all_conf()
+ cmd = [TOOL_CMD, jarfile, mainclass] + list(args)
+ print "Setting ACCUMULO_CONF_DIR=" + confdir
+ os.putenv("ACCUMULO_CONF_DIR", confdir)
+ call(cmd)
+
+def jar(jarfile, *args):
+ """Syntax: [accumulo-slider --app appname[ --appconf confdir] jar jarfile [mainclass]]
+ OR: [accumulo-slider --appconf confdir jar jarfile [mainclass]]
+ Runs a class from a specified jar
+ """
+ exec_accumulo_command("jar", jarfile, args=args)
+
+def classname(mainclass, *args):
+ """Syntax: [accumulo-slider --app appname[ --appconf confdir] classname mainclass]
+ OR: [accumulo-slider --appconf confdir classname mainclass]
+ Runs a specified class on the existing accumulo classpath
+ """
+ exec_accumulo_command(mainclass, args=args)
+
+def shell(*args):
+ """Syntax: [accumulo-slider --app appname[ --appconf confdir] shell]
+ OR: [accumulo-slider --appconf confdir shell]
+ Runs an accumulo shell
+ """
+ exec_accumulo_command("shell", args=args)
+
+def admin(*args):
+ """Syntax: [accumulo-slider --app appname[ --appconf confdir] admin cmd]
+ OR: [accumulo-slider --appconf confdir admin cmd]
+ Executes an admin command (run without cmd argument for a list of commands)
+ """
+ exec_accumulo_command("admin", args=args)
+
+def classpath(*args):
+ """Syntax: [accumulo-slider --app appname[ --appconf confdir] classpath]
+ OR: [accumulo-slider --appconf confdir classpath]
+ Prints the classpath of the accumulo client install
+ """
+ exec_accumulo_command("classpath", args=args)
+
+def info(*args):
+ """Syntax: [accumulo-slider --app appname[ --appconf confdir] info]
+ OR: [accumulo-slider --appconf confdir info]
+ Prints information about an accumulo instance (monitor, masters, and zookeepers)
+ """
+ exec_accumulo_command("info", args=args)
+
+def rfileinfo(*args):
+ """Syntax: [accumulo-slider --app appname[ --appconf confdir] rfile-info rfilename]
+ OR: [accumulo-slider --appconf confdir rfile-info rfilename]
+ Prints information about a specified RFile
+ """
+ exec_accumulo_command("rfile-info", args=args)
+
+def logininfo(*args):
+ """Syntax: [accumulo-slider --app appname[ --appconf confdir] login-info]
+ OR: [accumulo-slider --appconf confdir login-info]
+ Prints the supported authentication token types for the accumulo instance
+ """
+ exec_accumulo_command("login-info", args=args)
+
+def createtoken(*args):
+ """Syntax: [accumulo-slider --app appname[ --appconf confdir] create-token]
+ OR: [accumulo-slider --appconf confdir create-token]
+ Saves a given accumulo authentication token to a file
+ """
+ exec_accumulo_command("create-token", args=args)
+
+def version(*args):
+ """Syntax: [accumulo-slider --app appname[ --appconf confdir] version]
+ OR: [accumulo-slider --appconf confdir version]
+ Prints the version of the accumulo client install
+ """
+ exec_accumulo_command("version", args=args)
+
+def tool(jarfile, mainclass, *args):
+ """Syntax: [accumulo-slider --app appname[ --appconf confdir] tool jarfile classname]
+ OR: [accumulo-slider --appconf confdir tool jarfile classname]
+ Runs a mapreduce job using accumulo's tool.sh
+ """
+ exec_tool_command(jarfile, mainclass, args=args)
+
+def quicklinks():
+ """Syntax: [accumulo-slider --app appname quicklinks]
+ Prints the quicklinks information of accumulo-slider registry
+ """
+ global CMD_OPTS
+ if not 'app_name' in CMD_OPTS.keys():
+ print_usage()
+ sys.exit(1)
+
+ cmd = [SLIDER_CMD, "registry", "--getconf", "quicklinks", "--format", "json",
+ "--name", CMD_OPTS['app_name']]
+
+ if 'user' in CMD_OPTS.keys():
+ cmd.append( "--user "+CMD_OPTS['user'])
+
+ call(cmd)
+
+def proxies():
+ """Syntax: [accumulo-slider --app appname proxies]
+ Prints the componentinstancedata information of accumulo-slider registry
+ """
+ global CMD_OPTS
+ if not 'app_name' in CMD_OPTS.keys():
+ print_usage()
+ sys.exit(1)
+
+ cmd = [SLIDER_CMD, "registry", "--getconf", "componentinstancedata",
+ "--format", "json", "--name", CMD_OPTS['app_name']]
+
+ if 'user' in CMD_OPTS.keys():
+ cmd.append( "--user "+CMD_OPTS['user'])
+
+ call(cmd)
+
+def install(dir):
+ """Syntax: [accumulo-slider --app appname install dir]
+ Installs a fully configured accumulo client in the specified dir
+ The resulting client may be used on its own without accumulo-slider
+ """
+ global CMD_OPTS
+ if not 'app_name' in CMD_OPTS.keys():
+ print_usage()
+ sys.exit(1)
+ if os.path.exists(dir):
+ raise Exception("Install dir must not exist: " + dir)
+
+ global TMP_DIR
+ workdir = os.path.join(TMP_DIR, 'install-work-dir')
+
+ statusfile = os.path.join(workdir, 'status.json')
+ cmd = [SLIDER_CMD, "status", CMD_OPTS['app_name'], "--out", statusfile]
+ call(cmd)
+
+ infile = open(statusfile)
+ try:
+ content = json.load(infile)
+ finally:
+ infile.close()
+
+ appdef = content['options']['application.def']
+ appdeffile = appdef[appdef.rfind('/')+1:]
+ cmd = ["hadoop", "fs", "-copyToLocal", appdef, workdir]
+ call(cmd)
+
+ cmd = ["unzip", os.path.join(workdir, appdeffile), "-d", workdir]
+ call(cmd)
+
+ gzfile = glob.glob(os.path.join(workdir, 'package', 'files', 'accumulo*gz'))
+ if len(gzfile) != 1:
+ raise Exception("got " + gzfile + " from glob")
+ cmd = ["tar", "xvzf", gzfile[0], '-C', workdir]
+ call(cmd)
+
+ tmp_accumulo = glob.glob(os.path.join(workdir, 'accumulo-[0-9]*'))
+ if len(tmp_accumulo) != 1:
+ raise Exception("got " + tmp_accumulo + " from glob")
+ tmp_accumulo = tmp_accumulo[0]
+
+ confdir = os.path.join(tmp_accumulo, 'conf')
+ tmpconf = os.path.join(workdir, 'conf-tmp')
+ shutil.move(confdir, tmpconf)
+ make_conf(os.path.join(tmpconf, 'templates'), confdir)
+
+ libdir = os.path.join(tmp_accumulo, 'lib')
+ for jar in glob.glob(os.path.join(workdir, 'package', 'files', '*jar')):
+ shutil.move(jar, libdir)
+ shutil.move(tmp_accumulo, dir)
+
+def get_all_conf():
+ """Syntax: [accumulo-slider --app appname [--appconf confdir] getconf]
+ Downloads configuration for an accumulo instance
+ If --appconf is specified, creates the specified conf dir and populates it
+ """
+ ACCUMULO_CONF_DIR = os.getenv('ACCUMULO_CONF_DIR', None)
+ if ACCUMULO_CONF_DIR == None or (not os.path.exists(ACCUMULO_CONF_DIR)):
+ ACCUMULO_DIR = os.getenv('ACCUMULO_HOME', None)
+ if ACCUMULO_DIR == None or (not os.path.exists(ACCUMULO_DIR)):
+ print "Unable to find ACCUMULO_HOME. Please configure ACCUMULO_HOME before running this command"
+ sys.exit(1)
+ ACCUMULO_CONF_DIR = os.path.join(ACCUMULO_DIR, 'conf')
+
+ global CMD_OPTS
+ global TMP_DIR
+ confdir = os.path.join(TMP_DIR, 'conf')
+ if 'app_conf' in CMD_OPTS.keys():
+ confdir = CMD_OPTS['app_conf']
+ if os.path.exists(confdir):
+ print "Using existing app conf instead of downloading it again: " + confdir
+ return confdir
+
+ if not 'app_name' in CMD_OPTS.keys():
+ print_usage()
+ sys.exit(1)
+
+ make_conf(ACCUMULO_CONF_DIR, confdir)
+ return confdir
+
+def make_conf(oldconf, newconf):
+ client_file = os.path.join(newconf, 'client.conf')
+ site_file = os.path.join(newconf, 'accumulo-site.xml')
+ env_file = os.path.join(newconf, 'accumulo-env.sh')
+ env_json = os.path.join(newconf, 'accumulo-env.json')
+
+ print "Copying base conf from " + oldconf + " to " + newconf
+ shutil.copytree(oldconf, newconf)
+
+ try_remove(client_file)
+ try_remove(site_file)
+ try_remove(env_file)
+ try_remove(env_json)
+
+ get_conf("client", "properties", client_file)
+ get_conf("accumulo-site", "xml", site_file)
+ get_conf("accumulo-env", "json", env_json)
+
+ infile = open(env_json)
+ outfile = open(env_file, 'w')
+ try:
+ content = json.load(infile)
+ outfile.write(content['content'])
+ finally:
+ outfile.close()
+ infile.close()
+
+def try_remove(path):
+ try:
+ os.remove(path)
+ except:
+ if os.path.exists(path):
+ raise
+
+def get_conf(confname, fileformat, destfile):
+ if os.path.exists(destfile):
+ print "Conf file " + destfile + " already exists, remove it to re-download"
+ return
+
+ cmd = [SLIDER_CMD, "registry", "--getconf", confname, "--format",
+ fileformat, "--dest", destfile, "--name", CMD_OPTS['app_name']]
+ if 'user' in CMD_OPTS.keys():
+ cmd.append("--user " + CMD_OPTS['user'])
+
+ call(cmd)
+ if not os.path.exists(destfile):
+ raise Exception("Failed to read slider deployed accumulo config " + confname)
+
+def print_commands():
+ """Print all client commands and link to documentation"""
+ print "Commands:\n\t", "\n\t".join(sorted(COMMANDS.keys()))
+ print "\nHelp:", "\n\thelp", "\n\thelp <command>"
+
+def print_usage(command=None):
+ """Print one help message or list of available commands"""
+ if command != None:
+ if COMMANDS.has_key(command):
+ print (COMMANDS[command].__doc__ or
+ "No documentation provided for <%s>" % command)
+ else:
+ print "<%s> is not a valid command" % command
+ else:
+ print "Usage:"
+ print "accumulo-slider --app <name>[ --appconf <confdir> --user <username>] <command>"
+ print " The option --appconf creates a conf dir that can be reused;"
+ print " on subsequent calls to accumulo-slider, --app can be left off if"
+ print " --appconf is specified. If --appconf is not specified, a"
+ print " temporary conf dir is created each time accumulo-slider is run."
+ print_commands()
+
+def unknown_command(*args):
+ print "Unknown command: [accumulo-slider %s]" % ' '.join(sys.argv[1:])
+ print_usage()
+
+COMMANDS = {"shell": shell, "tool": tool, "admin": admin, "classpath": classpath,
+ "info": info, "version": version, "jar": jar, "classname": classname,
+ "quicklinks" : quicklinks, "proxies": proxies, "getconf": get_all_conf,
+ "rfile-info": rfileinfo, "login-info": logininfo, "create-token": createtoken,
+ "install": install, "help": print_usage}
+
+def parse_config_opts(args):
+ curr = args[:]
+ curr.reverse()
+ global CMD_OPTS
+ args_list = []
+ while len(curr) > 0:
+ token = curr.pop()
+ if token == "--app":
+ CMD_OPTS['app_name'] = curr.pop() if (len(curr) != 0) else None
+ elif token == "--user":
+ CMD_OPTS['user'] = curr.pop() if (len(curr) != 0) else None
+ elif token == "--appconf":
+ CMD_OPTS['app_conf'] = curr.pop() if (len(curr) != 0) else None
+ else:
+ args_list.append(token)
+ return args_list
+
+def main():
+ args = parse_config_opts(sys.argv[1:])
+ if len(args) < 1:
+ print_usage()
+ sys.exit(-1)
+ COMMAND = args[0]
+ ARGS = args[1:]
+ try:
+ (COMMANDS.get(COMMAND, unknown_command))(*ARGS)
+ finally:
+ global CMD_OPTS
+ if os.path.exists(TMP_DIR):
+ print "Cleaning up tmp dir " + TMP_DIR
+ shutil.rmtree(TMP_DIR)
+
+if __name__ == "__main__":
+ main()
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96f890b1/app-packages/accumulo/pom.xml
----------------------------------------------------------------------
diff --git a/app-packages/accumulo/pom.xml b/app-packages/accumulo/pom.xml
index 23ea3d6..4467294 100644
--- a/app-packages/accumulo/pom.xml
+++ b/app-packages/accumulo/pom.xml
@@ -38,6 +38,7 @@
<slider.bin.dir>../../slider-assembly/target/slider-${project.version}-all/slider-${project.version}</slider.bin.dir>
<test.app.pkg.dir>${project.build.directory}</test.app.pkg.dir>
<test.app.resources.dir>${project.build.directory}/test-config</test.app.resources.dir>
+ <test.app.scripts.dir>${project.build.directory}/${app.package.name}</test.app.scripts.dir>
<!-- these properties are used in the default and the test appConfigs -->
<hadoop.dir>/usr/lib/hadoop</hadoop.dir>
<hdfs.dir>/usr/lib/hadoop-hdfs</hdfs.dir>
@@ -214,6 +215,7 @@
<test.app.pkg.file>${app.package.name}.zip</test.app.pkg.file>
<test.app.pkg.name>ACCUMULO</test.app.pkg.name>
<test.app.resources.dir>${test.app.resources.dir}</test.app.resources.dir>
+ <test.app.scripts.dir>${test.app.scripts.dir}</test.app.scripts.dir>
</systemPropertyVariables>
</configuration>
</plugin>
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96f890b1/app-packages/accumulo/src/assembly/accumulo.xml
----------------------------------------------------------------------
diff --git a/app-packages/accumulo/src/assembly/accumulo.xml b/app-packages/accumulo/src/assembly/accumulo.xml
index 7be1942..3bc0e1f 100644
--- a/app-packages/accumulo/src/assembly/accumulo.xml
+++ b/app-packages/accumulo/src/assembly/accumulo.xml
@@ -24,6 +24,7 @@
<id>accumulo_v${accumulo.version}</id>
<formats>
<format>zip</format>
+ <format>dir</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
@@ -47,6 +48,12 @@
<fileMode>0755</fileMode>
</file>
<file>
+ <source>accumulo-slider</source>
+ <outputDirectory>/</outputDirectory>
+ <filtered>false</filtered>
+ <fileMode>0755</fileMode>
+ </file>
+ <file>
<source>${project.build.directory}/${work.dir}/${pkg.name}</source>
<outputDirectory>package/files</outputDirectory>
<filtered>false</filtered>
@@ -72,6 +79,7 @@
<exclude>appConfig-default.json</exclude>
<exclude>appConfig-secured-default.json</exclude>
<exclude>metainfo.xml</exclude>
+ <exclude>accumulo-slider</exclude>
</excludes>
<fileMode>0755</fileMode>
<directoryMode>0755</directoryMode>
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96f890b1/app-packages/accumulo/src/test/groovy/org/apache/slider/funtest/accumulo/AccumuloProxyIT.groovy
----------------------------------------------------------------------
diff --git a/app-packages/accumulo/src/test/groovy/org/apache/slider/funtest/accumulo/AccumuloProxyIT.groovy b/app-packages/accumulo/src/test/groovy/org/apache/slider/funtest/accumulo/AccumuloProxyIT.groovy
index 730355f..d25811f 100644
--- a/app-packages/accumulo/src/test/groovy/org/apache/slider/funtest/accumulo/AccumuloProxyIT.groovy
+++ b/app-packages/accumulo/src/test/groovy/org/apache/slider/funtest/accumulo/AccumuloProxyIT.groovy
@@ -71,7 +71,7 @@ class AccumuloProxyIT extends AccumuloBasicIT {
return proxies
} catch (Exception e) {
caught = e;
- log.info("Got exception trying to read quicklinks")
+ log.info("Got exception trying to read proxies")
if (tries-- == 0) {
break
}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96f890b1/app-packages/accumulo/src/test/groovy/org/apache/slider/funtest/accumulo/AccumuloScriptIT.groovy
----------------------------------------------------------------------
diff --git a/app-packages/accumulo/src/test/groovy/org/apache/slider/funtest/accumulo/AccumuloScriptIT.groovy b/app-packages/accumulo/src/test/groovy/org/apache/slider/funtest/accumulo/AccumuloScriptIT.groovy
new file mode 100644
index 0000000..2ba06af
--- /dev/null
+++ b/app-packages/accumulo/src/test/groovy/org/apache/slider/funtest/accumulo/AccumuloScriptIT.groovy
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.slider.funtest.accumulo
+
+import groovy.util.logging.Slf4j
+import org.apache.slider.api.ClusterDescription
+import org.apache.slider.client.SliderClient
+import org.apache.slider.funtest.framework.AccumuloSliderShell
+import org.junit.BeforeClass
+
+import java.nio.ByteBuffer
+
+@Slf4j
+class AccumuloScriptIT extends AccumuloBasicIT {
+ public static final String RESOURCES_DIR = sysprop("test.app.resources.dir")
+ public static final String ACCUMULO_HOME = RESOURCES_DIR + "/install"
+ public static final String ACCUMULO_CONF = sysprop("test.app.resources.dir") + "/conf"
+
+ @Override
+ public String getClusterName() {
+ return "test_script";
+ }
+
+ @Override
+ public String getDescription() {
+ return "Test accumulo-slider client script $clusterName"
+ }
+
+ @BeforeClass
+ public static void setShell() {
+ AccumuloSliderShell.scriptFile = new File(sysprop("test.app.scripts.dir"),
+ "accumulo-slider").canonicalFile
+ AccumuloSliderShell.setEnv("SLIDER_HOME", SLIDER_TAR_DIR)
+ AccumuloSliderShell.setEnv("SLIDER_CONF_DIR", SLIDER_CONF_DIR)
+ AccumuloSliderShell.setEnv("ACCUMULO_HOME", ACCUMULO_HOME)
+ }
+
+ @Override
+ public void clusterLoadOperations(ClusterDescription cd, SliderClient sliderClient) {
+ String clusterName = getClusterName()
+ AccumuloSliderShell.run(0, "--app $clusterName quicklinks")
+ AccumuloSliderShell.run(0, "--app $clusterName install $ACCUMULO_HOME")
+ AccumuloSliderShell.run(0, "--app $clusterName --appconf $ACCUMULO_CONF getconf")
+ runBoth("shell -u $USER -p $PASSWORD -e tables")
+ runBoth("login-info")
+
+ AccumuloSliderShell info = runOne("info")
+ String monitor = getMonitorUrl(sliderClient, getClusterName())
+ assert info.outputContains(monitor.substring(monitor.indexOf("://")+3)),
+ "accumulo info output did not contain monitor"
+
+ runOne("version")
+ runOne("classpath")
+ runOne("create-token -u $USER -p $PASSWORD -f $RESOURCES_DIR/token")
+ runOne("admin checkTablets")
+ runOne("admin listInstances")
+ runOne("admin ping")
+ runOne("admin dumpConfig -a -d $RESOURCES_DIR")
+ runOne("admin volumes")
+
+ // TODO: test tool, jar, classname, rfile-info
+ // runOne("shell -u $USER -p $PASSWORD -e \"createtable testtable\"")
+ }
+
+ public AccumuloSliderShell runOne(String cmd) {
+ return AccumuloSliderShell.run(0, "--appconf $ACCUMULO_CONF $cmd")
+ }
+
+ public void runBoth(String cmd) {
+ String clusterName = getClusterName()
+ AccumuloSliderShell.run(0, "--app $clusterName $cmd")
+ AccumuloSliderShell.run(0, "--appconf $ACCUMULO_CONF $cmd")
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96f890b1/app-packages/accumulo/src/test/groovy/org/apache/slider/funtest/accumulo/AccumuloSliderShell.groovy
----------------------------------------------------------------------
diff --git a/app-packages/accumulo/src/test/groovy/org/apache/slider/funtest/accumulo/AccumuloSliderShell.groovy b/app-packages/accumulo/src/test/groovy/org/apache/slider/funtest/accumulo/AccumuloSliderShell.groovy
new file mode 100644
index 0000000..09b8a39
--- /dev/null
+++ b/app-packages/accumulo/src/test/groovy/org/apache/slider/funtest/accumulo/AccumuloSliderShell.groovy
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.slider.funtest.framework
+
+import org.apache.bigtop.itest.shell.Shell
+import org.apache.slider.core.exceptions.SliderException
+import org.apache.slider.common.tools.SliderUtils
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+
+class AccumuloSliderShell extends ShellBase {
+
+ public static File scriptFile;
+
+ /**
+ * Build the command
+ * @param commands
+ */
+ AccumuloSliderShell(Collection<String> commands) {
+ super(scriptFile.absolutePath + " " + commands.join(" "))
+ }
+
+ /**
+ * Exec any slider command
+ * @param conf
+ * @param commands
+ * @return the shell
+ */
+ public static AccumuloSliderShell run(int exitCode, String command) {
+ AccumuloSliderShell shell = new AccumuloSliderShell([command])
+ shell.execute(exitCode);
+ return shell
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96f890b1/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/ShellBase.groovy
----------------------------------------------------------------------
diff --git a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/ShellBase.groovy b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/ShellBase.groovy
new file mode 100644
index 0000000..112486f
--- /dev/null
+++ b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/ShellBase.groovy
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.slider.funtest.framework
+
+import org.apache.bigtop.itest.shell.Shell
+import org.apache.slider.core.exceptions.SliderException
+import org.apache.slider.common.tools.SliderUtils
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+
+class ShellBase extends Shell {
+ private static final Logger log = LoggerFactory.getLogger(ShellBase.class);
+ private static final Logger LOG = log;
+
+ public static final String BASH = '/bin/bash -s'
+ public static final String CMD = 'cmd'
+
+ /**
+ * Environment variables
+ */
+ private static final Map<String, String> environment = [:]
+
+ private final String command
+
+ /**
+ * Build the command
+ * @param commands
+ */
+ ShellBase(String cmd) {
+ super(org.apache.hadoop.util.Shell.WINDOWS ? CMD : BASH)
+ command = cmd
+ }
+
+ /**
+ * Exec the command
+ * @return the script exit code
+ */
+ int execute() {
+ log.info(command)
+ List<String> commandLine = buildEnvCommands()
+
+ commandLine << command
+ if (org.apache.hadoop.util.Shell.WINDOWS) {
+ // Ensure the errorlevel returned by last call is set for the invoking shell
+ commandLine << "@echo ERRORLEVEL=%ERRORLEVEL%"
+ commandLine << "@exit %ERRORLEVEL%"
+ }
+ String script = commandLine.join("\n")
+ log.debug(script)
+ exec(script);
+ signCorrectReturnCode()
+ return ret;
+ }
+
+ public String getPathElementSeparator() {
+ File.pathSeparator
+ }
+
+ public static boolean isWindows() {
+ return org.apache.hadoop.util.Shell.WINDOWS
+ }
+
+ /**
+ * Set an environment variable
+ * @param var variable name
+ * @param val value
+ */
+ public static void setEnv(String var, Object val) {
+ environment[var] = val.toString()
+ }
+
+ /**
+ * Get an environment variable
+ * @param var variable name
+ * @return the value or null
+ */
+ public static String getEnv(String var) {
+ return environment[var]
+ }
+
+ /**
+ * Build up a list of environment variable setters from the
+ * env variables
+ * @return a list of commands to set up the env on the target system.
+ */
+ public static List<String> buildEnvCommands() {
+ List<String> commands = []
+ environment.each { String var, String val ->
+ commands << env(var, val)
+ }
+ return commands
+ }
+
+ /**
+ * Add an environment variable
+ * @param var variable
+ * @param val value (which will be stringified)
+ * @return an env variable command
+ */
+ static String env(String var, Object val) {
+ if (isWindows()) {
+ return "set " + var + "=${val.toString()}"
+ } else {
+ return "export " + var + "=${val.toString()};"
+ }
+ }
+
+ /**
+ * Fix up the return code so that a value of 255 is mapped back to -1
+ * @return twos complement return code from an unsigned byte
+ */
+ int signCorrectReturnCode() {
+ ret = signCorrect(ret)
+ }
+
+ /**
+ * Execute expecting a specific exit code
+ * @param expectedExitCode the expected exit code
+ */
+ void execute(int expectedExitCode) {
+ execute()
+ assertExitCode(expectedExitCode)
+ }
+
+ /**
+ * Sign-correct a process exit code
+ * @param exitCode the incoming exit code
+ * @return the sign-corrected version
+ */
+ public static int signCorrect(int exitCode) {
+ return (exitCode << 24) >> 24;
+ }
+
+ @Override
+ public String toString() {
+ return ret + " =>" + command
+ }
+
+ /**
+ * Dump the command, return code and outputs to the log.
+ * stdout is logged at info; stderr at error.
+ */
+ public void dumpOutput() {
+ log.error(toString())
+ log.error("return code = ${signCorrectReturnCode()}")
+ if (out.size() != 0) {
+ log.info("\n<stdout>\n${stdoutHistory}\n</stdout>");
+ }
+ if (err.size() != 0) {
+ log.error("\n<stderr>\n${stdErrHistory}\n</stderr>");
+ }
+ }
+
+ /**
+ * Get the stderr history
+ * @return the history
+ */
+ public String getStdErrHistory() {
+ return err.join('\n')
+ }
+
+ /**
+ * Get the stdout history
+ * @return the history
+ */
+ public String getStdoutHistory() {
+ return out.join('\n')
+ }
+
+ /**
+ * Assert the shell exited with a given error code
+ * if not the output is printed and an assertion is raised
+ * @param errorCode expected error code
+ */
+ public void assertExitCode(int errorCode, String extra="") {
+ if (this.ret != errorCode) {
+ dumpOutput()
+ throw new SliderException(ret,
+ "Expected exit code of command ${command} : ${errorCode} - actual=${ret} $extra")
+ }
+ }
+
+ /**
+ * Execute shell script consisting of as many Strings as we have arguments,
+ * NOTE: individual strings are concatenated into a single script as though
+ * they were delimited with new line character. All quoting rules are exactly
+ * what one would expect in standalone shell script.
+ *
+ * After executing the script its return code can be accessed as getRet(),
+ * stdout as getOut() and stderr as getErr(). The script itself can be accessed
+ * as getScript()
+ * WARNING: it isn't thread safe
+ * @param args shell script split into multiple Strings
+ * @return Shell object for chaining
+ */
+ Shell exec(Object... args) {
+ Process proc = "$shell".execute()
+ script = args.join("\n")
+ ByteArrayOutputStream baosErr = new ByteArrayOutputStream(4096);
+ ByteArrayOutputStream baosOut = new ByteArrayOutputStream(4096);
+ proc.consumeProcessOutput(baosOut, baosErr)
+
+ Thread.start {
+ def writer = new PrintWriter(new BufferedOutputStream(proc.out))
+ writer.println(script)
+ writer.flush()
+ writer.close()
+ }
+
+ proc.waitFor()
+ ret = proc.exitValue()
+
+ out = streamToLines(baosOut)
+ err = streamToLines(baosErr)
+
+ if (LOG.isTraceEnabled()) {
+ if (ret != 0) {
+ LOG.trace("return: $ret");
+ }
+ if (out.size() != 0) {
+ LOG.trace("\n<stdout>\n${out.join('\n')}\n</stdout>");
+ }
+
+ if (err.size() != 0) {
+ LOG.trace("\n<stderr>\n${err.join('\n')}\n</stderr>");
+ }
+ }
+ return this
+ }
+
+ /**
+ * Convert a stream to lines in an array
+ * @param out output stream
+ * @return the list of entries
+ */
+ protected List<String> streamToLines(ByteArrayOutputStream out) {
+ if (out.size() != 0) {
+ return out.toString().split('\n');
+ } else {
+ return [];
+ }
+ }
+
+ public String findLineEntry(String[] locaters) {
+ int index = 0;
+ def output = out +"\n"+ err
+ for (String str in output) {
+ if (str.contains("\"" + locaters[index] + "\"")) {
+ if (locaters.size() == index + 1) {
+ return str;
+ } else {
+ index++;
+ }
+ }
+ }
+
+ return null;
+ }
+
+ public boolean outputContains(
+ String lookThisUp,
+ int n = 1) {
+ int count = 0
+ def output = out + "\n" + err
+ for (String str in output) {
+ int subCount = countString(str, lookThisUp)
+ count = count + subCount
+ if (count == n) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ public static int countString(String str, String search) {
+ int count = 0
+ if (SliderUtils.isUnset(str) || SliderUtils.isUnset(search)) {
+ return count
+ }
+
+ int index = str.indexOf(search, 0)
+ while (index >= 0) {
+ ++count
+ index = str.indexOf(search, index + 1)
+ }
+ return count
+ }
+
+ public findLineEntryValue(String[] locaters) {
+ String line = findLineEntry(locaters);
+
+ if (line != null) {
+ log.info("Parsing {} for value.", line)
+ int dividerIndex = line.indexOf(":");
+ if (dividerIndex > 0) {
+ String value = line.substring(dividerIndex + 1).trim()
+ if (value.endsWith(",")) {
+ value = value.subSequence(0, value.length() - 1)
+ }
+ return value;
+ }
+ }
+ return null;
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/96f890b1/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/SliderShell.groovy
----------------------------------------------------------------------
diff --git a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/SliderShell.groovy b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/SliderShell.groovy
index ae40d6a..9270e8c 100644
--- a/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/SliderShell.groovy
+++ b/slider-funtest/src/main/groovy/org/apache/slider/funtest/framework/SliderShell.groovy
@@ -21,147 +21,40 @@ package org.apache.slider.funtest.framework
import org.apache.bigtop.itest.shell.Shell
import org.apache.slider.core.exceptions.SliderException
import org.apache.slider.common.tools.SliderUtils
+import org.junit.BeforeClass
import org.slf4j.Logger
import org.slf4j.LoggerFactory
-class SliderShell extends Shell {
- private static final Logger log = LoggerFactory.getLogger(SliderShell.class);
- private static final Logger LOG = log;
-
- public static final String BASH = '/bin/bash -s'
- public static final String CMD = 'cmd'
-
+class SliderShell extends ShellBase {
/**
* Configuration directory, shared across all instances. Not marked as volatile,
* assumed set up during @BeforeClass
*/
public static File confDir;
-
- public static File scriptFile;
-
- public File shellScript;
-
- public static final List<String> slider_classpath_extra = []
- /**
- * Environment varaibles
- */
- protected static final Map<String, String> environment = [:]
+ public static File scriptFile;
- final String command
+ public static final List<String> slider_classpath_extra = []
/**
* Build the command
* @param commands
*/
SliderShell(Collection<String> commands) {
- super(org.apache.hadoop.util.Shell.WINDOWS ? CMD : BASH)
+ super(scriptFile.absolutePath + " " + commands.join(" "))
assert confDir != null;
- assert scriptFile != null;
- shellScript = scriptFile;
- command = scriptFile.absolutePath + " " + commands.join(" ")
- }
- /**
- * Exec the command
- * @return the script exit code
- */
- int execute() {
- log.info(command)
setEnv(FuntestProperties.ENV_SLIDER_CONF_DIR, confDir)
if (!slider_classpath_extra.empty) {
setEnv(FuntestProperties.ENV_SLIDER_CLASSPATH_EXTRA,
- SliderUtils.join(slider_classpath_extra,
- pathElementSeparator,
- false))
- }
- List<String> commandLine = buildEnvCommands()
-
- commandLine << command
- if (org.apache.hadoop.util.Shell.WINDOWS) {
- // Ensure the errorlevel returned by last call is set for the invoking shell
- commandLine << "@echo ERRORLEVEL=%ERRORLEVEL%"
- commandLine << "@exit %ERRORLEVEL%"
- }
- String script = commandLine.join("\n")
- log.debug(script)
- exec(script);
- signCorrectReturnCode()
- return ret;
- }
-
- public String getPathElementSeparator() {
- File.pathSeparator
- }
-
- public static boolean isWindows() {
- return org.apache.hadoop.util.Shell.WINDOWS
- }
-
- /**
- * Set an environment variable
- * @param var variable name
- * @param val value
- */
- public static void setEnv(String var, Object val) {
- environment[var] = val.toString()
- }
-
- /**
- * Get an environment variable
- * @param var variable name
- * @return the value or null
- */
- public static String getEnv(String var) {
- return environment[var]
- }
-
- /**
- * Build up a list of environment variable setters from the
- * env variables
- * @return a list of commands to set up the env on the target system.
- */
- public static List<String> buildEnvCommands() {
- List<String> commands = []
- environment.each { String var, String val ->
- commands << env(var, val)
- }
- return commands
- }
-
- /**
- * Add an environment variable
- * @param var variable
- * @param val value (which will be stringified)
- * @return an env variable command
- */
- static String env(String var, Object val) {
- if (isWindows()) {
- return "set " + var + "=${val.toString()}"
- } else {
- return "export " + var + "=${val.toString()};"
+ SliderUtils.join(slider_classpath_extra,
+ pathElementSeparator,
+ false))
}
}
/**
- * Fix up the return code so that a value of 255 is mapped back to -1
- * @return twos complement return code from an unsigned byte
- */
- int signCorrectReturnCode() {
- ret = signCorrect(ret)
- }
-
- /**
- * Execute expecting a specific exit code
- * @param expectedExitCode the expected exit code
- */
- void execute(int expectedExitCode) {
- execute()
- assertExitCode(expectedExitCode)
- }
-
- /**
* Exec any slider command
* @param conf
* @param commands
@@ -172,186 +65,4 @@ class SliderShell extends Shell {
shell.execute(exitCode);
return shell
}
-
- /**
- * Sign-correct a process exit code
- * @param exitCode the incoming exit code
- * @return the sign-corrected version
- */
- public static int signCorrect(int exitCode) {
- return (exitCode << 24) >> 24;
- }
-
- @Override
- public String toString() {
- return ret + " =>" + command
- }
-
- /**
- * Dump the command, return code and outputs to the log.
- * stdout is logged at info; stderr at error.
- */
- public void dumpOutput() {
- log.error(toString())
- log.error("return code = ${signCorrectReturnCode()}")
- if (out.size() != 0) {
- log.info("\n<stdout>\n${stdoutHistory}\n</stdout>");
- }
- if (err.size() != 0) {
- log.error("\n<stderr>\n${stdErrHistory}\n</stderr>");
- }
- }
-
- /**
- * Get the stderr history
- * @return the history
- */
- public String getStdErrHistory() {
- return err.join('\n')
- }
-
- /**
- * Get the stdout history
- * @return the history
- */
- public String getStdoutHistory() {
- return out.join('\n')
- }
-
- /**
- * Assert the shell exited with a given error code
- * if not the output is printed and an assertion is raised
- * @param errorCode expected error code
- */
- public void assertExitCode(int errorCode, String extra="") {
- if (this.ret != errorCode) {
- dumpOutput()
- throw new SliderException(ret,
- "Expected exit code of command ${command} : ${errorCode} - actual=${ret} $extra")
- }
- }
-
- /**
- * Execute shell script consisting of as many Strings as we have arguments,
- * NOTE: individual strings are concatenated into a single script as though
- * they were delimited with new line character. All quoting rules are exactly
- * what one would expect in standalone shell script.
- *
- * After executing the script its return code can be accessed as getRet(),
- * stdout as getOut() and stderr as getErr(). The script itself can be accessed
- * as getScript()
- * WARNING: it isn't thread safe
- * @param args shell script split into multiple Strings
- * @return Shell object for chaining
- */
- Shell exec(Object... args) {
- Process proc = "$shell".execute()
- script = args.join("\n")
- ByteArrayOutputStream baosErr = new ByteArrayOutputStream(4096);
- ByteArrayOutputStream baosOut = new ByteArrayOutputStream(4096);
- proc.consumeProcessOutput(baosOut, baosErr)
-
- Thread.start {
- def writer = new PrintWriter(new BufferedOutputStream(proc.out))
- writer.println(script)
- writer.flush()
- writer.close()
- }
-
- proc.waitFor()
- ret = proc.exitValue()
-
- out = streamToLines(baosOut)
- err = streamToLines(baosErr)
-
- if (LOG.isTraceEnabled()) {
- if (ret != 0) {
- LOG.trace("return: $ret");
- }
- if (out.size() != 0) {
- LOG.trace("\n<stdout>\n${out.join('\n')}\n</stdout>");
- }
-
- if (err.size() != 0) {
- LOG.trace("\n<stderr>\n${err.join('\n')}\n</stderr>");
- }
- }
- return this
- }
-
- /**
- * Convert a stream to lines in an array
- * @param out output stream
- * @return the list of entries
- */
- protected List<String> streamToLines(ByteArrayOutputStream out) {
- if (out.size() != 0) {
- return out.toString().split('\n');
- } else {
- return [];
- }
- }
-
- public String findLineEntry(String[] locaters) {
- int index = 0;
- def output = out +"\n"+ err
- for (String str in output) {
- if (str.contains("\"" + locaters[index] + "\"")) {
- if (locaters.size() == index + 1) {
- return str;
- } else {
- index++;
- }
- }
- }
-
- return null;
- }
-
- public boolean outputContains(
- String lookThisUp,
- int n = 1) {
- int count = 0
- def output = out + "\n" + err
- for (String str in output) {
- int subCount = countString(str, lookThisUp)
- count = count + subCount
- if (count == n) {
- return true;
- }
- }
- return false;
- }
-
- public static int countString(String str, String search) {
- int count = 0
- if (SliderUtils.isUnset(str) || SliderUtils.isUnset(search)) {
- return count
- }
-
- int index = str.indexOf(search, 0)
- while (index >= 0) {
- ++count
- index = str.indexOf(search, index + 1)
- }
- return count
- }
-
- public findLineEntryValue(String[] locaters) {
- String line = findLineEntry(locaters);
-
- if (line != null) {
- log.info("Parsing {} for value.", line)
- int dividerIndex = line.indexOf(":");
- if (dividerIndex > 0) {
- String value = line.substring(dividerIndex + 1).trim()
- if (value.endsWith(",")) {
- value = value.subSequence(0, value.length() - 1)
- }
- return value;
- }
- }
- return null;
- }
-
}