You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by st...@apache.org on 2014/10/20 13:47:14 UTC
svn commit: r1633112 -
/incubator/slider/site/trunk/content/developing/functional_tests.md
Author: stevel
Date: Mon Oct 20 11:47:13 2014
New Revision: 1633112
URL: http://svn.apache.org/r1633112
Log:
SLIDER-540 Command Test Base to set up HADOOP_CONF_DIR
Modified:
incubator/slider/site/trunk/content/developing/functional_tests.md
Modified: incubator/slider/site/trunk/content/developing/functional_tests.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/developing/functional_tests.md?rev=1633112&r1=1633111&r2=1633112&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/developing/functional_tests.md (original)
+++ incubator/slider/site/trunk/content/developing/functional_tests.md Mon Oct 20 11:47:13 2014
@@ -27,19 +27,19 @@
The functional test suite is designed to test slider against
a live cluster.
-For these to work you need
+For these to work you MUST have
1. A YARN Cluster -secure or insecure
1. A `slider-client.xml` file configured to interact with the cluster
1. Agent tests and Accumulo Agent tests: nothing additional
-1. HBase provider tests: HBase `.tar.gz` uploaded to HDFS, and a local or remote hbase conf
+1. (deprecated) HBase provider tests: HBase aa`.tar.gz` uploaded to HDFS, and a local or remote hbase conf
directory
-1. Accumulo provider tests: Accumulo `.tar.gz` uploaded to HDFS, and a local or remote accumulo conf
+1. (deprecated) Accumulo provider tests: Accumulo `.tar.gz` uploaded to HDFS, and a local or remote accumulo conf
directory
## Configuration of functional tests
-Maven needs to be given
+Maven MUST have to be given
1. A path to the expanded test archive
1. A path to a slider configuration directory for the cluster
@@ -64,11 +64,67 @@ This file is loaded whenever a slider bu
The `slider-client.xml` must have extra configuration options for both the HBase and
Accumulo tests, as well as a common set for actually talking to a YARN cluster.
+### How to pick up `core-site.xml`, `hdfs-site.xml` and `yarn-site.xml` values
+
+The `slider-client.xml` file can declare a `HADOOP_CONF_DIR` value
+for use by both the test runner AND the command line `bin\slider` application
+invoked.
+
+
+ <property>
+ <name>HADOOP_CONF_DIR</name>
+ <value>/home/tester/sites/production/hadoop-conf</value>
+ </property>
+
+If set:
+
+1. The standard `-site.xml` files are loaded by the JUnit test runner, to bond
+the test classes to the YARN cluster.
+1. The property is used to set the environment variable `HADOOP_CONF_DIR`
+before the `bin/slider` or bin\slider.py` script is executed.
+
+
+**Note 1:** a path can be set relative to ${SLIDER_CONF_DIR}
+
+ <property>
+ <name>HADOOP_CONF_DIR</name>
+ <value>${SLIDER_CONF_DIR}/../hadoop-conf</value>
+ </property>
+
+or, on Windows
+
+ <property>
+ <name>HADOOP_CONF_DIR</name>
+ <value>${SLIDER_CONF_DIR}\..\hadoop-conf</value>
+ </property>
+
+**Note 2:** To test on the local cluster, use either an absolute path,
+or refer to the environment variable `env.HADOOP_CONF_DIR`:
+
+ <property>
+ <name>HADOOP_CONF_DIR</name>
+ <value>${env.HADOOP_CONF_DIR}</value>
+ </property>
+
+Be aware that expansion of the `env.` environment variables and `SLIDER_CONF_DIR`
+are only performed on the test runs. If the same `slider-client.xml` value
+is used directly from the CLI, the environment variable `HADOOP_CONF_DIR`
+must be set to the absolute path of the hadoop configuration directory.
+
+## How to validate the configuration
+
+A test case executing the `slider diagnostics` command can be used
+to print out the configuration as seen by the `bin/slider` process
+
+ mvn integration-test \
+ -Dslider.conf.dir=/home/tester/sites/production/slider \
+ -Dit.test=DiagnosticsCommandIT
+
## Disabling the functional tests entirely
All functional tests which require a live YARN cluster
-are run during the integration-test phase. They are executed with the command
-`mvn verify`.
+are run during the integration-test phase. They are executed with the commands
+`mvn verify` or `mvn integration-test` .
If you do not wish to run the functional tests, simply use the `mvn package` command
and only those tests which do not require a live YARN cluster will run.
@@ -142,71 +198,25 @@ To enable the test ensure that *slider.t
**Test setup**
-Edit config file src/test/clusters/remote/slider/slider-client.xml and ensure that the host names are accurate for the test cluster.
+Edit config file `src/test/clusters/remote/slider/slider-client.xml` or
+ your chosen equivalent and ensure that the host names are accurate for the test cluster.
**User setup**
-Ensure that the user, running the test, is present on the cluster against which you are running the tests. The user must be a member of the hadoop group.
+Ensure that the user running the test, is present on the cluster against which you are running the tests. The user must be a member of the hadoop group.
-E.g. adduser **testuser** -d /home/**testuser** -G hadoop -m
+E.g:
+
+ adduser testuser -d /home/testuser -G hadoop -m
**HDFS Setup**
Set up hdfs folders for test user
-* su hdfs
-* hdfs dfs -mkdir /user/testuser
-* hdfs dfs -chown testuser:hdfs /user/testuser
+ su hdfs
+ hdfs dfs -mkdir /user/testuser
+ hdfs dfs -chown testuser:hdfs /user/testuser
-### HBase Tests
-
-The HBase tests can be enabled or disabled
-
- <property>
- <name>slider.test.hbase.enabled</name>
- <description>Flag to enable/disable HBase tests</description>
- <value>true</value>
- </property>
-
-Mandatory test parameters must be added to `slider-client.xml`
-
- <property>
- <name>slider.test.hbase.tar</name>
- <description>Path to the HBase Tar file in HDFS</description>
- <value>hdfs://sandbox:8020/user/slider/hbase.tar.gz</value>
- </property>
-
- <property>
- <name>slider.test.hbase.appconf</name>
- <description>Path to the directory containing the HBase application config</description>
- <value>file://${user.dir}/src/test/clusters/sandbox/hbase</value>
- </property>
-
-Optional parameters:
-
- <property>
- <name>slider.test.hbase.launch.wait.seconds</name>
- <description>Time to wait in seconds for HBase to start</description>
- <value>1800</value>
- </property>
-
-#### Accumulo configuration options
-
-Enable/disable the tests
-
- <property>
- <name>slider.test.accumulo.enabled</name>
- <description>Flag to enable/disable Accumulo tests</description>
- <value>true</value>
- </property>
-
-Optional parameters
-
- <property>
- <name>slider.test.accumulo.launch.wait.seconds</name>
- <description>Time to wait in seconds for Accumulo to start</description>
- <value>1800</value>
- </property>
### Configuring the YARN cluster for tests
@@ -244,6 +254,17 @@ to keep the log files around after an ap
<value>30000</value>
</property>
+
+ <!-- registry-->
+ <property>
+ <name>hadoop.registry.rm.enabled</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hadoop.registry.zk.quorum</name>
+ <value>${hostname}:2181</value>
+ </property>
## Testing against a secure cluster
@@ -271,19 +292,19 @@ There's two techniques
1. Keep the data in HDFS and refer to it there. This works if there is a shared,
persistent HDFS cluster.
-1. Use the special property `slider.test.conf.dir` that is set to the path
+1. Use the special property `SLIDER_CONF_DIR` that is set to the path
of the directory, and which can then be used to create an absolute path
from paths relative to the configuration dir:
<property>
- <name>slider.test.hbase.appconf</name>
- <description>Path to the directory containing the HBase application config</description>
- <value>file://${slider.test.conf.dir}/../hbase</value>
- </property>
+ <name>slider.test.hbase.appconf</name>
+ <description>Path to the directory containing the HBase application config</description>
+ <value>file://${SLIDER_CONF_DIR}/../hbase</value>
+ </property>
-If the actual XML file path is required, a similar property
-`slider.test.conf.xml` is set.
+If the actual XML file path is required, use the property
+`slider.test.conf.xml`
## Parallel execution
@@ -399,3 +420,52 @@ For example, to run all Agent IT tests:
mvn integration-test -Dslider.conf.dir=${your-config-dir} -Dit.test=Agent\*IT
+### HBase Tests
+
+The HBase tests can be enabled or disabled
+
+ <property>
+ <name>slider.test.hbase.enabled</name>
+ <description>Flag to enable/disable HBase tests</description>
+ <value>true</value>
+ </property>
+
+Mandatory test parameters must be added to `slider-client.xml`
+
+ <property>
+ <name>slider.test.hbase.tar</name>
+ <description>Path to the HBase Tar file in HDFS</description>
+ <value>hdfs://sandbox:8020/user/slider/hbase.tar.gz</value>
+ </property>
+
+ <property>
+ <name>slider.test.hbase.appconf</name>
+ <description>Path to the directory containing the HBase application config</description>
+ <value>file://${${SLIDER_CONF_DIR}}/../hbase</value>
+ </property>
+
+Optional parameters:
+
+ <property>
+ <name>slider.test.hbase.launch.wait.seconds</name>
+ <description>Time to wait in seconds for HBase to start</description>
+ <value>1800</value>
+ </property>
+
+#### Accumulo configuration options
+
+Enable/disable the tests
+
+ <property>
+ <name>slider.test.accumulo.enabled</name>
+ <description>Flag to enable/disable Accumulo tests</description>
+ <value>true</value>
+ </property>
+
+Optional parameters
+
+ <property>
+ <name>slider.test.accumulo.launch.wait.seconds</name>
+ <description>Time to wait in seconds for Accumulo to start</description>
+ <value>1800</value>
+ </property>