You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bigtop.apache.org by co...@apache.org on 2016/01/13 00:47:14 UTC
bigtop git commit: BIGTOP-2227. smoke-tests should be a part of the
top-level project, not separated
Repository: bigtop
Updated Branches:
refs/heads/master 6c9829e8b -> 5d63481d5
BIGTOP-2227. smoke-tests should be a part of the top-level project, not separated
Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo
Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/5d63481d
Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/5d63481d
Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/5d63481d
Branch: refs/heads/master
Commit: 5d63481d533e79f89f2be56dede58c7c2bb29fc5
Parents: 6c9829e
Author: Konstantin Boudnik <co...@apache.org>
Authored: Fri Jan 8 21:29:46 2016 -0800
Committer: Konstantin Boudnik <co...@apache.org>
Committed: Tue Jan 12 15:44:20 2016 -0800
----------------------------------------------------------------------
README.md | 5 +-
bigtop-deploy/vm/utils/smoke-tests.sh | 8 ++--
bigtop-tests/smoke-tests/README | 25 +++++++---
bigtop-tests/smoke-tests/build.gradle | 22 +++------
bigtop-tests/smoke-tests/hive/build.gradle | 8 ++--
.../smoke-tests/ignite-hadoop/build.gradle | 4 +-
bigtop-tests/smoke-tests/mahout/build.gradle | 4 +-
bigtop-tests/smoke-tests/mapreduce/build.gradle | 2 +-
bigtop-tests/smoke-tests/pig/build.gradle | 2 +-
bigtop-tests/smoke-tests/settings.gradle | 50 --------------------
build.gradle | 21 +++++++-
settings.gradle | 14 +++++-
12 files changed, 73 insertions(+), 92 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 8c08616..e772809 100644
--- a/README.md
+++ b/README.md
@@ -97,10 +97,7 @@ For specific questions it's always a good idea to ping the mailing list at dev-s
For Users: Running the smoke tests.
-----------------------------------
-The simplest way to test bigtop is to:
-
-* Step 1: cd bigtop-tests/smoke-tests/
-* Step 2: Follow the instructions in the smoke-tests/README file.
+The simplest way to test bigtop is described in bigtop-tests/smoke-tests/README file
For integration (API level) testing with maven, read on.
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-deploy/vm/utils/smoke-tests.sh
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/utils/smoke-tests.sh b/bigtop-deploy/vm/utils/smoke-tests.sh
index b27c082..1d83687 100755
--- a/bigtop-deploy/vm/utils/smoke-tests.sh
+++ b/bigtop-deploy/vm/utils/smoke-tests.sh
@@ -28,7 +28,6 @@ fi
echo -e "\n===== START TO RUN SMOKE TESTS: $SMOKE_TESTS =====\n"
export HADOOP_CONF_DIR=/etc/hadoop/conf/
-export BIGTOP_HOME=/bigtop-home/
export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce/
export HIVE_HOME=/usr/lib/hive/
export PIG_HOME=/usr/lib/pig/
@@ -36,7 +35,6 @@ export FLUME_HOME=/usr/lib/flume/
export SQOOP_HOME=/usr/lib/sqoop/
export HIVE_CONF_DIR=/etc/hive/conf/
export MAHOUT_HOME="/usr/lib/mahout"
-export ITEST="1.0.0"
su -s /bin/bash $HCFS_USER -c '/usr/bin/hadoop fs -mkdir /user/vagrant /user/root'
su -s /bin/bash $HCFS_USER -c 'hadoop fs -chmod 777 /user/vagrant'
@@ -47,4 +45,8 @@ if [ -f /etc/debian_version ] ; then
else
yum install -y pig hive flume mahout sqoop
fi
-cd /bigtop-home && ./gradlew -b bigtop-tests/smoke-tests/build.gradle clean test -Dsmoke.tests=$SMOKE_TESTS --info
+ALL_SMOKE_TASKS=""
+for s in `echo $SMOKE_TESTS | sed -e 's#,# #g'`; do
+ ALL_SMOKE_TASKS="$ALL_SMOKE_TASKS bigtop-tests:smoke-tests:$s:test"
+done
+cd /bigtop-home && ./gradlew clean $ALL_SMOKE_TASKS -Psmoke.tests --info
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/README
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/README b/bigtop-tests/smoke-tests/README
index 992f49c..430d03b 100644
--- a/bigtop-tests/smoke-tests/README
+++ b/bigtop-tests/smoke-tests/README
@@ -18,7 +18,6 @@ So overall, this framework makes testing with apache bigtop easy as well as fun!
You need to export several environment variables. For example,
```
export HADOOP_CONF_DIR=/etc/hadoop/conf/
-export BIGTOP_HOME=/bigtop-home/
export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce/
export HIVE_HOME=/usr/lib/hive/
export PIG_HOME=/usr/lib/pig/
@@ -28,19 +27,33 @@ export JAVA_HOME="/usr/lib/jvm/java-openjdk/"
export MAHOUT_HOME="/usr/lib/mahout"
export KITE_HOME="/usr/lib/kite"
export SPARK_HOME="/usr/lib/spark"
-export ITEST="1.0.0"
```
-Then, simply invoke the tests including the smoke.tests you want to run, as described below.
+Then, simply invoke the tests you want to run, as described below.
Each directory is a gradle "subproject" mapping to an ecosystem component.
-If you specify, for example, "flume", then the flume test runs. To test hive as well,
-you would specify "flume,hive", and so on. To run the tests from the top-level
+Invoking test task of flume subproject will execute flume tests. To test hive as well,
+you would add the invocation of hive:test, and so on. To run the tests from the top-level
Bigtop folder, follow the example below:
```
- ./gradlew -b bigtop-tests/smoke-tests/build.gradle clean test -Dsmoke.tests=flume,hive --info
+ ./gradlew bigtop-tests:smoke-tests:flume:test bigtop-tests:smoke-tests:hive:test -Psmoke.tests --info
```
+You also can run smoke tests from bigtop-tests/smoke-tests directory,
+or from any subproject of it. E.g.
+```
+ cd bigtop-tests/smoke-tests
+ ../../gradlew hive:test -Psmoke.tests
+```
+or
+```
+ cd bigtop-tests/smoke-tests/hive
+ ../../../gradlew test -Psmoke.tests
+```
+Instead of using top-level Gradle wrapper one can also use system-wide
+Gradle, if installed and available.
+Setting up project property smoke.tests is important, cause this is
+required to to enable test tasks for smoke projects.
The --info option is a nice feature to have here: You can easily debug your tests this way,
just make sure you have log4j.properties in your conf/ directory locally.
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/build.gradle
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/build.gradle b/bigtop-tests/smoke-tests/build.gradle
index 1e0da58..6842c51 100644
--- a/bigtop-tests/smoke-tests/build.gradle
+++ b/bigtop-tests/smoke-tests/build.gradle
@@ -15,14 +15,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-allprojects {
- apply plugin: 'groovy'
-
- repositories {
- mavenCentral()
- }
-}
-
subprojects {
/**
* Utility function for tests to use to confirm EVN Variables.
@@ -32,21 +24,20 @@ subprojects {
def value = System.getenv("${it}")
if (value == null || value == "null")
throw new GradleException("undeclared env variable: ${it}")
- else
- println("ENV VARIABLE: ${it} = $value");
}
}
/**
* Two important environment variables.
* java_home should be declared in all cases.
- * itest can be greater than or equal to = 0.7.0, 0.8.0
*/
- checkEnv(["JAVA_HOME","ITEST","BIGTOP_HOME"])
+ checkEnv(["JAVA_HOME"])
- ext.itestVersion = System.getenv("ITEST")
ext.groovyVersion = '1.8.0'
ext.hadoopVersion = '2.6.0'
+ // itest needs be greater than or equal to = 1.0.0
+ ext.itestVersion = '1.0.0' // Might need to be able to read an input for alternate version?
+ ext.BIGTOP_HOME = rootDir
dependencies {
//needed to avoid groovy not on classpath error.
@@ -81,9 +72,8 @@ subprojects {
return !keep_this_test;
}
- println("Now testing...");
- test {
-
+ test << {
+ println("Now testing...");
//todo, add back in 'basic' after BIGTOP-1392 .
testLogging {
events "passed", "skipped", "failed"
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/hive/build.gradle
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/hive/build.gradle b/bigtop-tests/smoke-tests/hive/build.gradle
index 67dcb71..9b06c3c 100644
--- a/bigtop-tests/smoke-tests/hive/build.gradle
+++ b/bigtop-tests/smoke-tests/hive/build.gradle
@@ -26,8 +26,8 @@ def tests_to_include() {
sourceSets {
test {
groovy {
- srcDirs = ["${System.env.BIGTOP_HOME}/bigtop-tests/test-artifacts/hive/",
- "${System.env.BIGTOP_HOME}/bigtop-tests/smoke-tests/hive/"]
+ srcDirs = ["${BIGTOP_HOME}/bigtop-tests/test-artifacts/hive/",
+ "${BIGTOP_HOME}/bigtop-tests/smoke-tests/hive/"]
exclude { FileTreeElement elem -> (doExclude(elem.getName())) }
}
}
@@ -39,7 +39,5 @@ test {
}
test.doFirst {
- // BIGTOP_HOME is necessary here, since we use it to read in from
- // test-artifacts.
- checkEnv(["BIGTOP_HOME", "HIVE_HOME", "HIVE_CONF_DIR", "BIGTOP_HOME"])
+ checkEnv(["HIVE_HOME", "HIVE_CONF_DIR"])
}
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle b/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle
index b3ea841..46e6364 100644
--- a/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle
+++ b/bigtop-tests/smoke-tests/ignite-hadoop/build.gradle
@@ -24,11 +24,11 @@ def tests_to_include() {
sourceSets {
test {
groovy {
- srcDirs = ["${System.env.BIGTOP_HOME}/bigtop-tests/smoke-tests/ignite-hadoop/"]
+ srcDirs = ["${BIGTOP_HOME}/bigtop-tests/smoke-tests/ignite-hadoop/"]
}
}
}
test.doFirst {
- checkEnv(["BIGTOP_HOME", "HADOOP_MAPRED_HOME"])
+ checkEnv(["HADOOP_MAPRED_HOME"])
}
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/mahout/build.gradle
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/mahout/build.gradle b/bigtop-tests/smoke-tests/mahout/build.gradle
index b6048d6..d79c09e 100644
--- a/bigtop-tests/smoke-tests/mahout/build.gradle
+++ b/bigtop-tests/smoke-tests/mahout/build.gradle
@@ -24,12 +24,12 @@ def tests_to_include() {
sourceSets {
test {
groovy {
- srcDirs = ["${System.env.BIGTOP_HOME}/bigtop-tests/test-artifacts/mahout/"]
+ srcDirs = ["${BIGTOP_HOME}/bigtop-tests/test-artifacts/mahout/"]
exclude { FileTreeElement elem -> (doExclude(elem.getName())) }
}
}
}
test.doFirst {
- checkEnv(["BIGTOP_HOME", "MAHOUT_HOME"])
+ checkEnv(["MAHOUT_HOME"])
}
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/mapreduce/build.gradle
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/mapreduce/build.gradle b/bigtop-tests/smoke-tests/mapreduce/build.gradle
index 4aad5f4..0d5e4df 100644
--- a/bigtop-tests/smoke-tests/mapreduce/build.gradle
+++ b/bigtop-tests/smoke-tests/mapreduce/build.gradle
@@ -25,7 +25,7 @@ sourceSets {
test {
groovy {
srcDirs = [
- "${System.env.BIGTOP_HOME}/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/mapreduce"
+ "${BIGTOP_HOME}/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/mapreduce"
]
exclude {
FileTreeElement elem -> (doExclude(elem.getName()))
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/pig/build.gradle
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/pig/build.gradle b/bigtop-tests/smoke-tests/pig/build.gradle
index 81c0d68..cc09ea8 100644
--- a/bigtop-tests/smoke-tests/pig/build.gradle
+++ b/bigtop-tests/smoke-tests/pig/build.gradle
@@ -26,7 +26,7 @@ def tests_to_include() {
sourceSets {
test {
groovy {
- srcDirs = ["./"]
+ srcDirs = [projectDir]
exclude 'src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/**'
exclude { FileTreeElement elem -> (doExclude(elem.getName())) }
}
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/bigtop-tests/smoke-tests/settings.gradle
----------------------------------------------------------------------
diff --git a/bigtop-tests/smoke-tests/settings.gradle b/bigtop-tests/smoke-tests/settings.gradle
deleted file mode 100644
index 464988c..0000000
--- a/bigtop-tests/smoke-tests/settings.gradle
+++ /dev/null
@@ -1,50 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- * <p/>
- * http://www.apache.org/licenses/LICENSE-2.0
- * <p/>
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-def tests_comma_str = System.properties['smoke.tests']
-
-if(tests_comma_str != null) {
- def tests = tests_comma_str.split(",")
- tests.each() { t ->
- println("including $t")
- include(t)
- }
-}
-else {
- println("""
-
- ******************************************************************************
- Welcome to the bigtop smoke tests.
-
- This is an extensible test module, with one directory for every test component.
-
- To run it, just use the following command, adding components (directories) into
-
- the smoke.tests system property.
-
- For example, to run the hive/ and pig/ smoke tests:
-
- gradle clean compileGroovy test -Dsmoke.tests=hive,pig
-
- NOTE: You must run clean. Otherwise gradle might skip some of the tests.
-
- ... Exiting now ...
- ******************************************************************************
- """)
- System.exit(1)
-}
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/build.gradle
----------------------------------------------------------------------
diff --git a/build.gradle b/build.gradle
index 56accb9..e7cd663 100644
--- a/build.gradle
+++ b/build.gradle
@@ -172,6 +172,24 @@ project(':itest-common') {
}
}
+/**
+ * checkSmokeTestProjects() configures smoke-tests:<component>:test task only
+ * in if the project has smoke.tests properties set.
+ * This is done to avoid running cluster smoke tests during the normal life-cycle
+ * of the project development
+ */
+def checkSmokeTestProjects = {
+ FileTree fTree = fileTree(dir: 'bigtop-tests/smoke-tests', include: '*/build.gradle')
+ fTree.each() { smokeProject ->
+ def parent = smokeProject.getParentFile().name
+ project (":bigtop-tests:smoke-tests:$parent") {
+ test {
+ onlyIf {project.hasProperty('smoke.tests')}
+ }
+ }
+ }
+}
+
def TESTARTIFACTS_GROUP = 'test artifacts'
def DEVENV_GROUP = 'development tools'
def DEPLOY_GROUP = 'deployment'
@@ -389,7 +407,8 @@ task "bigtop-slaves"(dependsOn: 'gen-gradle-home', type:Exec,
commandLine command
}
-project.afterEvaluate{
+project.afterEvaluate {
+ checkSmokeTestProjects()
artifactToInstall(dependsOn: [installTopLevel, installCommon, installConf, installiTest])
}
http://git-wip-us.apache.org/repos/asf/bigtop/blob/5d63481d/settings.gradle
----------------------------------------------------------------------
diff --git a/settings.gradle b/settings.gradle
index 30e6924..ac071f1 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -18,5 +18,17 @@
rootProject.name = 'bigtop'
include 'bigtop-test-framework', 'bigtop-tests:smoke-tests'
-
project(":bigtop-test-framework").name = 'itest-common'
+
+/**
+ * The following code will does include all subprojects of smoke-tests
+ * so they are lined-up properly in the structure of the Bigtop project.
+ * The inclusion is done dynamically, so when new smoke test projects
+ * there's no extra effort to get it running.
+ */
+FileTree fTree = fileTree(dir: 'bigtop-tests/smoke-tests', include: '*/build.gradle')
+fTree.each() { buildFile ->
+ def parent = buildFile.getParentFile().name
+ include("bigtop-tests:smoke-tests:$parent")
+}
+