You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by ct...@apache.org on 2020/02/18 22:59:46 UTC
[accumulo] branch master updated: Remove some test components from
tarball/pom (#1517)
This is an automated email from the ASF dual-hosted git repository.
ctubbsii pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/master by this push:
new adbe766 Remove some test components from tarball/pom (#1517)
adbe766 is described below
commit adbe7663d445148e4fd78605fdf8ac22b4fc3de3
Author: Christopher Tubbs <ct...@apache.org>
AuthorDate: Tue Feb 18 17:59:39 2020 -0500
Remove some test components from tarball/pom (#1517)
Remove unmaintained "MRIT" testing feature to run ITs in MapReduce.
ITs can be run in parallel using more reliable mechanisms, like matrix
build jobs in a CI service, or by scripting Docker or some other
mechanism. The "MRIT" feature is not currently maintained and hasn't
been supported for some time.
Also remove accumulo-test jar from binary tarball, because it isn't
needed by downstream users, and it can pollute the class path with
metrics configuration files that only apply during integration testing.
---
TESTING.md | 22 ---
assemble/pom.xml | 22 ---
pom.xml | 6 -
test/pom.xml | 58 ------
.../accumulo/test/functional/DurabilityIT.java | 8 -
.../test/mrit/IntegrationTestMapReduce.java | 219 ---------------------
6 files changed, 335 deletions(-)
diff --git a/TESTING.md b/TESTING.md
index 51d4e16..ab1e1e5 100644
--- a/TESTING.md
+++ b/TESTING.md
@@ -129,27 +129,6 @@ Each of the above properties can be set on the commandline (-Daccumulo.it.cluste
collection can be placed into a properties file and referenced using "accumulo.it.cluster.properties". Properties
specified on the command line override properties set in a file.
-## MapReduce job for Integration tests
-
-[ACCUMULO-3871][issue] (re)introduced the ability to parallelize the execution of the Integration Test suite by the use
-of MapReduce/YARN. When a YARN cluster is available, this can drastically reduce the amount of time to run all tests.
-
-To run the tests, you first need a list of the tests. A simple way to get a list, is to scan the accumulo-test jar file for them.
-
-`jar -tf lib/accumulo-test.jar | grep IT.class | tr / . | sed -e 's/.class$//' >accumulo-integration-tests.txt`
-
-Then, put the list of files into HDFS:
-
-`hdfs dfs -mkdir /tmp`
-`hdfs dfs -put accumulo-integration-tests.txt /tmp/tests`
-
-Finally, launch the job, providing the list of tests to run and a location to store the test results. Optionally, a built
-native library shared object can be provided to the Mapper's classpath to enable MiniAccumuloCluster to use the native maps
-instead of the Java-based implementation. (Note that the below paths are the JAR and shared object are based on an installation.
-These files do exist in the build tree, but at different locations)
-
-`yarn jar lib/accumulo-test.jar org.apache.accumulo.test.mrit.IntegrationtestMapReduce -libjars lib/native/libaccumulo.so /tmp/accumulo-integration-tests.txt /tmp/accumulo-integration-test-results`
-
# Manual Distributed Testing
Apache Accumulo has a number of tests which are suitable for running against large clusters for hours to days at a time.
@@ -157,5 +136,4 @@ These test suites exist in the [accumulo-testing repo][testing].
[testing]: https://github.com/apache/accumulo-testing
[surefire]: http://maven.apache.org/surefire/maven-surefire-plugin/
-[issue]: https://issues.apache.org/jira/browse/ACCUMULO-3871
[SpotBugs]: https://spotbugs.github.io/
diff --git a/assemble/pom.xml b/assemble/pom.xml
index 8c35d10..f76648d 100644
--- a/assemble/pom.xml
+++ b/assemble/pom.xml
@@ -224,11 +224,6 @@
</dependency>
<dependency>
<groupId>org.apache.accumulo</groupId>
- <artifactId>accumulo-test</artifactId>
- <optional>true</optional>
- </dependency>
- <dependency>
- <groupId>org.apache.accumulo</groupId>
<artifactId>accumulo-tracer</artifactId>
<optional>true</optional>
</dependency>
@@ -552,22 +547,5 @@
</plugins>
</build>
</profile>
- <profile>
- <!-- create shaded test jar appropriate for running ITs on MapReduce -->
- <id>mrit</id>
- <activation>
- <property>
- <name>mrit</name>
- </property>
- </activation>
- <dependencies>
- <dependency>
- <groupId>org.apache.accumulo</groupId>
- <artifactId>accumulo-test</artifactId>
- <classifier>mrit</classifier>
- <optional>true</optional>
- </dependency>
- </dependencies>
- </profile>
</profiles>
</project>
diff --git a/pom.xml b/pom.xml
index 785ddc8..ec322cf 100644
--- a/pom.xml
+++ b/pom.xml
@@ -380,12 +380,6 @@
<groupId>org.apache.accumulo</groupId>
<artifactId>accumulo-test</artifactId>
<version>${project.version}</version>
- <classifier>mrit</classifier>
- </dependency>
- <dependency>
- <groupId>org.apache.accumulo</groupId>
- <artifactId>accumulo-test</artifactId>
- <version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.accumulo</groupId>
diff --git a/test/pom.xml b/test/pom.xml
index 15d683c..0dbae0e 100644
--- a/test/pom.xml
+++ b/test/pom.xml
@@ -304,63 +304,5 @@
</plugins>
</build>
</profile>
- <profile>
- <!-- create shaded test jar appropriate for running ITs on MapReduce -->
- <id>mrit</id>
- <activation>
- <property>
- <name>mrit</name>
- </property>
- </activation>
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-shade-plugin</artifactId>
- <configuration>
- <artifactSet>
- <excludes>
- <exclude>com.google.auto.service</exclude>
- <exclude>com.google.auto</exclude>
- <exclude>javax.servlet:servlet-api</exclude>
- <exclude>org.apache.accumulo:accumulo-native</exclude>
- <exclude>org.slf4j:slf4j-log4j12</exclude>
- </excludes>
- </artifactSet>
- <shadedArtifactAttached>true</shadedArtifactAttached>
- <shadedClassifierName>mrit</shadedClassifierName>
- <createDependencyReducedPom>false</createDependencyReducedPom>
- <filters>
- <filter>
- <artifact>*:*</artifact>
- <excludes>
- <exclude>META-INF/*.DSA</exclude>
- <exclude>META-INF/*.RSA</exclude>
- <exclude>META-INF/*.SF</exclude>
- <exclude>META-INF/DEPENDENCIES</exclude>
- </excludes>
- </filter>
- </filters>
- <transformers>
- <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
- <manifestEntries>
- <Sealed>false</Sealed>
- <Main-Class>org.apache.accumulo.test.mrit.IntegrationTestMapReduce</Main-Class>
- </manifestEntries>
- </transformer>
- </transformers>
- </configuration>
- <executions>
- <execution>
- <id>create-shaded-mrit</id>
- <goals>
- <goal>shade</goal>
- </goals>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
- </profile>
</profiles>
</project>
diff --git a/test/src/main/java/org/apache/accumulo/test/functional/DurabilityIT.java b/test/src/main/java/org/apache/accumulo/test/functional/DurabilityIT.java
index 9728a9b..b186d75 100644
--- a/test/src/main/java/org/apache/accumulo/test/functional/DurabilityIT.java
+++ b/test/src/main/java/org/apache/accumulo/test/functional/DurabilityIT.java
@@ -20,7 +20,6 @@ package org.apache.accumulo.test.functional;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
-import static org.junit.Assume.assumeFalse;
import java.util.HashMap;
import java.util.Map;
@@ -38,10 +37,8 @@ import org.apache.accumulo.minicluster.ServerType;
import org.apache.accumulo.miniclusterImpl.MiniAccumuloConfigImpl;
import org.apache.accumulo.miniclusterImpl.ProcessReference;
import org.apache.accumulo.test.categories.MiniClusterOnlyTests;
-import org.apache.accumulo.test.mrit.IntegrationTestMapReduce;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.RawLocalFileSystem;
-import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.experimental.categories.Category;
@@ -62,11 +59,6 @@ public class DurabilityIT extends ConfigurableMacBase {
return 4 * 60;
}
- @BeforeClass
- public static void checkMR() {
- assumeFalse(IntegrationTestMapReduce.isMapReduce());
- }
-
static final long N = 100000;
private String[] init(AccumuloClient c) throws Exception {
diff --git a/test/src/main/java/org/apache/accumulo/test/mrit/IntegrationTestMapReduce.java b/test/src/main/java/org/apache/accumulo/test/mrit/IntegrationTestMapReduce.java
deleted file mode 100644
index 7df953a..0000000
--- a/test/src/main/java/org/apache/accumulo/test/mrit/IntegrationTestMapReduce.java
+++ /dev/null
@@ -1,219 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.accumulo.test.mrit;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
-
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.conf.Configured;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.io.LongWritable;
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.mapreduce.MRJobConfig;
-import org.apache.hadoop.mapreduce.Mapper;
-import org.apache.hadoop.mapreduce.Reducer;
-import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
-import org.apache.hadoop.mapreduce.lib.input.NLineInputFormat;
-import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
-import org.apache.hadoop.util.Tool;
-import org.apache.hadoop.util.ToolRunner;
-import org.junit.runner.Description;
-import org.junit.runner.JUnitCore;
-import org.junit.runner.Result;
-import org.junit.runner.notification.Failure;
-import org.junit.runner.notification.RunListener;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * Run the Integration Tests as a Map-Reduce job.
- * <p>
- * Each of the Integration tests takes 30s to 20m to run. Using a larger cluster, all the tests can
- * be run in parallel and finish much faster.
- * <p>
- * To run the tests, you first need a list of the tests. A simple way to get a list, is to scan the
- * accumulo-test jar file for them.
- *
- * <pre>
- * $ jar -tf lib/accumulo-test.jar | grep IT.class | tr / . | sed -e 's/.class$//' >tests
- * </pre>
- *
- * Put the list of tests into HDFS:
- *
- * <pre>
- * $ hadoop fs -mkdir /tmp
- * $ hadoop fs -put tests /tmp/tests
- * </pre>
- *
- * Run the class below as a map-reduce job, giving it the lists of tests, and a place to store the
- * results.
- *
- * <pre>
- * $ yarn jar lib/accumulo-test-mrit.jar -libjars lib/native/libaccumulo.so /tmp/tests /tmp/results
- * </pre>
- *
- * The result is a list of IT classes that pass or fail. Those classes that fail will be annotated
- * with the particular test that failed within the class.
- */
-public class IntegrationTestMapReduce extends Configured implements Tool {
-
- private static final Logger log = LoggerFactory.getLogger(IntegrationTestMapReduce.class);
-
- private static boolean isMapReduce = false;
-
- public static boolean isMapReduce() {
- return isMapReduce;
- }
-
- public static class TestMapper extends Mapper<LongWritable,Text,Text,Text> {
-
- static final Text FAIL = new Text("FAIL");
- static final Text PASS = new Text("PASS");
- static final Text ERROR = new Text("ERROR");
-
- public static enum TestCounts {
- PASS, FAIL, ERROR
- }
-
- @Override
- protected void map(LongWritable key, Text value,
- final Mapper<LongWritable,Text,Text,Text>.Context context)
- throws IOException, InterruptedException {
- isMapReduce = true;
- String className = value.toString();
- if (className.trim().isEmpty()) {
- return;
- }
- final List<String> failures = new ArrayList<>();
- Class<? extends Object> test = null;
- try {
- test = Class.forName(className);
- } catch (ClassNotFoundException e) {
- log.debug("Error finding class {}", className, e);
- context.getCounter(TestCounts.ERROR).increment(1);
- context.write(ERROR, new Text(e.toString()));
- return;
- }
- log.info("Running test {}", className);
- JUnitCore core = new JUnitCore();
- core.addListener(new RunListener() {
-
- @Override
- public void testStarted(Description description) {
- log.info("Starting {}", description);
- context.progress();
- }
-
- @Override
- public void testFinished(Description description) {
- log.info("Finished {}", description);
- context.progress();
- }
-
- @Override
- public void testFailure(Failure failure) {
- log.info("Test failed: {}", failure.getDescription(), failure.getException());
- failures.add(failure.getDescription().getMethodName());
- context.progress();
- }
-
- });
- context.setStatus(test.getSimpleName());
- try {
- Result result = core.run(test);
- if (result.wasSuccessful()) {
- log.info("{} was successful", className);
- context.getCounter(TestCounts.PASS).increment(1);
- context.write(PASS, value);
- } else {
- log.info("{} failed", className);
- context.getCounter(TestCounts.FAIL).increment(1);
- context.write(FAIL, new Text(className + "(" + String.join(", ", failures) + ")"));
- }
- } catch (Exception e) {
- // most likely JUnit issues, like no tests to run
- log.info("Test failed: {}", className, e);
- }
- }
- }
-
- public static class TestReducer extends Reducer<Text,Text,Text,Text> {
-
- @Override
- protected void reduce(Text code, Iterable<Text> tests,
- Reducer<Text,Text,Text,Text>.Context context) throws IOException, InterruptedException {
- StringBuilder result = new StringBuilder("\n");
- for (Text test : tests) {
- result.append(" ");
- result.append(test);
- result.append("\n");
- }
- context.write(code, new Text(result.toString()));
- }
- }
-
- @Override
- public int run(String[] args) throws Exception {
- // read a list of tests from the input, and print out the results
- if (args.length != 2) {
- System.err.println("Wrong number of args: <input> <output>");
- return 1;
- }
- Configuration conf = getConf();
- Job job = Job.getInstance(conf, "accumulo integration test runner");
- conf = job.getConfiguration();
-
- // some tests take more than 10 minutes
- conf.setLong(MRJobConfig.TASK_TIMEOUT, 20 * 60 * 1000);
-
- // minicluster uses a lot of ram
- conf.setInt(MRJobConfig.MAP_MEMORY_MB, 4000);
-
- // hadoop puts an ancient version of jline on the classpath
- conf.setBoolean(MRJobConfig.MAPREDUCE_JOB_USER_CLASSPATH_FIRST, true);
-
- // no need to run a test multiple times
- job.setSpeculativeExecution(false);
-
- // read one line at a time
- job.setInputFormatClass(NLineInputFormat.class);
- NLineInputFormat.setNumLinesPerSplit(job, 1);
-
- // run the test
- job.setJarByClass(IntegrationTestMapReduce.class);
- job.setMapperClass(TestMapper.class);
-
- // group test by result code
- job.setReducerClass(TestReducer.class);
- job.setOutputKeyClass(Text.class);
- job.setOutputValueClass(Text.class);
-
- FileInputFormat.addInputPath(job, new Path(args[0]));
- FileOutputFormat.setOutputPath(job, new Path(args[1]));
- return job.waitForCompletion(true) ? 0 : 1;
- }
-
- public static void main(String[] args) throws Exception {
- System.exit(ToolRunner.run(new IntegrationTestMapReduce(), args));
- }
-
-}