You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by bh...@apache.org on 2014/02/19 17:12:05 UTC
[02/10] git commit: ACCUMULO-2379 Document running functional tests
under MR as a different user
ACCUMULO-2379 Document running functional tests under MR as a different user
Functional tests might not run under MapReduce in clusters where a different user from
the Accumulo user runs mappers. Needed configuration changes are now documented in
the functional test README.
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/ac18b569
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/ac18b569
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/ac18b569
Branch: refs/heads/1.5.1-SNAPSHOT
Commit: ac18b569738b6dd8f950ee5d747af5176bbc8d13
Parents: 3898317
Author: Bill Havanki <bh...@cloudera.com>
Authored: Wed Feb 19 11:05:12 2014 -0500
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Wed Feb 19 11:05:12 2014 -0500
----------------------------------------------------------------------
test/system/auto/README | 94 ++++++++++++++++++++++++++++++--------------
1 file changed, 65 insertions(+), 29 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/ac18b569/test/system/auto/README
----------------------------------------------------------------------
diff --git a/test/system/auto/README b/test/system/auto/README
index 4773393..45ed158 100644
--- a/test/system/auto/README
+++ b/test/system/auto/README
@@ -1,58 +1,94 @@
Apache Accumulo Functional Tests
-These scripts run a series of tests against a small local accumulo instance. To run these scripts,
-you must have hadoop and zookeeper installed and running. You will need a functioning C compiler to build
-a shared library needed for one of the tests. The test suite is known to run on Linux RedHat Enterprise
+These scripts run a series of tests against a small local accumulo instance.
+To run these scripts, you must have Hadoop and Zookeeper installed and running.
+You will need a functioning C compiler to build a shared library needed for
+one of the tests. The test suite is known to run on Linux RedHat Enterprise
version 5, and Mac OS X 10.5.
-The tests are shown as being run from the ACCUMULO_HOME directory, but they should run from any directory.
-Make sure to create "logs" and "walogs" directories in ACCUMULO_HOME. Also, ensure that accumulo-env.sh
-specifies its ACCUMULO_LOG_DIR in the following way:
+How to Run
+
+The tests are shown as being run from the ACCUMULO_HOME directory, but they
+should run from any directory. Make sure to create "logs" and "walogs"
+directories in ACCUMULO_HOME. Also, ensure that accumulo-env.sh specifies its
+ACCUMULO_LOG_DIR in the following way:
+
test -z "$ACCUMULO_LOG_DIR" && export ACCUMULO_LOG_DIR=$ACCUMULO_HOME/logs
$ ./test/system/auto/run.py -l
-
-Will list all the test names. You can run the suite like this:
-
+
+will list all the test names. You can run the suite like this:
+
$ ./test/system/auto/run.py
-
+
You can select tests using a case-insensitive regular expression:
-
+
$ ./test/system/auto/run.py -t simple
$ ./test/system/auto/run.py -t SunnyDay
-
+
To run tests repeatedly:
$ ./test/system/auto/run.py -r 3
-If you are attempting to debug what is causing a test to fail, you can run the tests in "verbose" mode:
+If you are attempting to debug what is causing a test to fail, you can run the
+tests in "verbose" mode:
$ python test/system/auto/run.py -t SunnyDay -v 10
-
-If a test is failing, and you would like to examine logs from the run, you can run the test in "dirty"
-mode which will keep the test from cleaning up all the logs at the end of the run:
-
+
+If a test is failing, and you would like to examine logs from the run, you can
+run the test in "dirty" mode which will keep the test from cleaning up all the
+logs at the end of the run:
+
$ ./test/system/auto/run.py -t some.failing.test -d
-
-If the test suite hangs, and you would like to re-run the tests starting with the last test that failed:
-
+
+If the test suite hangs, and you would like to re-run the tests starting with
+the last test that failed:
+
$ ./test/system/auto/run.py -s start.over.test
-
-If tests tend to time out (on slower hardware, for example), you can scale up the timeout values by a
-multiplier. This example triples timeouts:
+
+If tests tend to time out (on slower hardware, for example), you can scale up
+the timeout values by a multiplier. This example triples timeouts:
$ ./test/system/auto/run.py -f 3
-Test results are normally printed to the console, but you can send them to XML files compatible with
-Jenkins:
+Test results are normally printed to the console, but you can send them to XML
+files compatible with Jenkins:
$ ./test/system/auto/run.py -x
-The full test suite can take nearly an hour. If you have a larger hadoop cluster at your disposal, you can
-run the tests as a map-reduce job:
+Running under MapReduce
+
+The full test suite can take nearly an hour. If you have a larger Hadoop
+cluster at your disposal, you can run the tests as a MapReduce job:
$ python test/system/auto/run.py -l > tests
$ hadoop fs -put tests /user/hadoop/tests
- $ ./bin/accumulo org.apache.accumulo.server.test.functional.RunTests /user/hadoop/tests /user/hadoop/results
+ $ ./bin/accumulo org.apache.accumulo.server.test.functional.RunTests \
+ /user/hadoop/tests /user/hadoop/results
+
+The example above runs every test. You can trim the tests file to include
+only the tests you wish to run.
+
+In some clusters, the user under which MR jobs run is different from the user
+under which Accumulo is installed, and this can cause failures running the
+tests. Various configuration and permission changes can be made to help the
+tests run, including the following:
+
+* Opening up directory and file permissions on each cluster node so that the MR
+ user has the same read/write privileges as the Accumulo user. Adding the MR
+ user to a shared group is one easy way to accomplish this. Access is required
+ to the Accumulo installation, log, write-ahead log, and configuration
+ directories.
+* Creating a user directory in HDFS, named after and owned by the MR user,
+ e.g., /user/mruser.
+* Setting the ZOOKEEPER_HOME and HADOOP_CONF_DIR environment variables for the
+ MR user. These can be set using the mapred.child.env property in
+ mapred-site.xml, e.g.:
+
+ <property>
+ <name>mapred.child.env</name>
+ <value>ZOOKEEPER_HOME=/path/to/zookeeper,HADOOP_CONF_DIR=/path/to/hadoop/conf</value>
+ </property>
-
+Each functional test is run by a mapper, and so you can check the mapper logs
+to see any error messages tests produce.