You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by bh...@apache.org on 2014/01/07 15:23:00 UTC
[01/10] git commit: ACCUMULO-2042 Get scalability test running
Updated Branches:
refs/heads/1.4.5-SNAPSHOT 8f9fe4175 -> a3d77e4ff
refs/heads/1.5.1-SNAPSHOT 21b1b1108 -> 833ac3533
refs/heads/1.6.0-SNAPSHOT 98211f581 -> c4cddf97f
refs/heads/master a2652c3c4 -> a72e80ca4
ACCUMULO-2042 Get scalability test running
Two problems prevented scalability tests from being run properly.
1. The Run class was looking for test classes under the "accumulo"
package instead of "org.apache.accumulo".
2. A results subdirectory on the local filesystem was not being
created, and so result data from HDFS could not be copied out.
This commit also updates the scalability test README with more
detail.
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/a3d77e4f
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/a3d77e4f
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/a3d77e4f
Branch: refs/heads/1.4.5-SNAPSHOT
Commit: a3d77e4ff2a12f161f1343490f42aa91d5b109d3
Parents: 8f9fe41
Author: Bill Havanki <bh...@cloudera.com>
Authored: Thu Dec 19 11:15:44 2013 -0500
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Tue Jan 7 09:10:04 2014 -0500
----------------------------------------------------------------------
.../accumulo/server/test/scalability/Run.java | 2 +-
test/system/scalability/README | 35 ++++++++++++++------
test/system/scalability/run.py | 2 ++
3 files changed, 28 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3d77e4f/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
----------------------------------------------------------------------
diff --git a/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java b/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
index 63861f8..7fb78c7 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
@@ -59,7 +59,7 @@ public class Run {
e.printStackTrace();
}
- ScaleTest test = (ScaleTest) Class.forName(String.format("accumulo.server.test.scalability.%s", testId)).newInstance();
+ ScaleTest test = (ScaleTest) Class.forName(String.format("org.apache.accumulo.server.test.scalability.%s", testId)).newInstance();
test.init(scaleProps, testProps, numTabletServers);
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3d77e4f/test/system/scalability/README
----------------------------------------------------------------------
diff --git a/test/system/scalability/README b/test/system/scalability/README
index 7206daa..cd7440c 100644
--- a/test/system/scalability/README
+++ b/test/system/scalability/README
@@ -1,23 +1,38 @@
-1. To run scalability tests, set up the site configuration (used by all tests)
+Apache Accumulo Scalability Tests
- cp conf/site.conf.example conf/site.conf
+The scalability test framework needs to be configured for your Accumulo
+instance by performing the following steps.
+
+WARNING: Each scalability test rewrites your conf/slaves file and reinitializes
+your Accumulo instance. Do not run these tests on a cluster holding essential
+data.
+
+1. Make sure you have both ACCUMULO_HOME and HADOOP_HOME set in your
+ $ACCUMULO_CONF_DIR/accumulo-env.sh.
-2. Modify site.conf to the correct configuration parameters for your Accumulo instance
+2. Create a 'site.conf' file in the conf directory containing settings
+ needed by test nodes to connect to Accumulo, and to guide the tests.
+
+ cp conf/site.conf.example conf/site.conf
-3. Set up the configuration file for the test you want to run
+3. Create an 'Ingest.conf' file in the conf directory containing performance
+ settings for the Ingest test. (This test is currently the only scalability
+ test available.)
cp conf/Ingest.conf.example conf/Ingest.conf
-4. Each test has a unique ID which correlates with its test code in:
+ Each test has a unique ID (e.g., "Ingest") which correlates with its test
+ code in:
org.apache.accumulo.server.test.scalability.tests.<ID>
- This ID correlates with a the following config file:
+ This ID correlates with a config file:
- ./conf/<ID>.conf
+ conf/<ID>.conf
-5. To run the test, just specifying its ID to the run.py script
+To run the test, specify its ID to the run.py script.
- export ACCUMULO_HOME=/opt/accumulo; nohup ./run.py Ingest > test1.log 2>&1 &
+ nohup ./run.py Ingest > test1.log 2>&1 &
-5. A directory will be create where logs and results are placed
+A timestamped directory will be created, and results are placed in it as each
+test completes.
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3d77e4f/test/system/scalability/run.py
----------------------------------------------------------------------
diff --git a/test/system/scalability/run.py b/test/system/scalability/run.py
index ac45aa2..938805a 100755
--- a/test/system/scalability/run.py
+++ b/test/system/scalability/run.py
@@ -189,8 +189,10 @@ def main():
log('Creating test directory structure')
testDir = 'test-%d' % time.time()
nodesDir = testDir+'/nodes'
+ resultsDir = testDir+'/results'
syscall('mkdir %s' % testDir)
syscall('mkdir %s' % nodesDir)
+ syscall('mkdir %s' % resultsDir)
log('Removing current /accumulo-scale directory')
syscall('hadoop fs -rmr /accumulo-scale')
[06/10] git commit: ACCUMULO-2042 Merge branch '1.4.5-SNAPSHOT' into
1.5.1-SNAPSHOT
Posted by bh...@apache.org.
ACCUMULO-2042 Merge branch '1.4.5-SNAPSHOT' into 1.5.1-SNAPSHOT
Conflicts:
test/src/main/java/org/apache/accumulo/test/scalability/Run.java
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/833ac353
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/833ac353
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/833ac353
Branch: refs/heads/1.5.1-SNAPSHOT
Commit: 833ac35339e91893063aba0f5cde8ff4886b2337
Parents: 21b1b11 a3d77e4
Author: Bill Havanki <bh...@cloudera.com>
Authored: Tue Jan 7 09:19:04 2014 -0500
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Tue Jan 7 09:19:04 2014 -0500
----------------------------------------------------------------------
.../apache/accumulo/test/scalability/Run.java | 2 +-
test/system/scalability/README | 35 ++++++++++++++------
test/system/scalability/run.py | 2 ++
3 files changed, 28 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/833ac353/test/src/main/java/org/apache/accumulo/test/scalability/Run.java
----------------------------------------------------------------------
diff --cc test/src/main/java/org/apache/accumulo/test/scalability/Run.java
index 6622d7c,0000000..0c598c0
mode 100644,000000..100644
--- a/test/src/main/java/org/apache/accumulo/test/scalability/Run.java
+++ b/test/src/main/java/org/apache/accumulo/test/scalability/Run.java
@@@ -1,88 -1,0 +1,88 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.test.scalability;
+
+import java.io.FileInputStream;
+import java.util.Properties;
+import java.net.InetAddress;
+
+import org.apache.accumulo.core.cli.Help;
+import org.apache.accumulo.core.util.CachedConfiguration;
+import org.apache.accumulo.test.scalability.ScaleTest;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import com.beust.jcommander.Parameter;
+
+public class Run {
+
+ static class Opts extends Help {
+ @Parameter(names="--testId", required=true)
+ String testId;
+ @Parameter(names="--action", required=true, description="one of 'setup', 'teardown' or 'client'")
+ String action;
+ @Parameter(names="--count", description="number of tablet servers", required=true)
+ int numTabletServers;
+ }
+
+ public static void main(String[] args) throws Exception {
+
+ final String sitePath = "/tmp/scale-site.conf";
+ final String testPath = "/tmp/scale-test.conf";
+ Opts opts = new Opts();
+ opts.parseArgs(Run.class.getName(), args);
+
+ Configuration conf = CachedConfiguration.getInstance();
+ FileSystem fs;
+ fs = FileSystem.get(conf);
+
+ fs.copyToLocalFile(new Path("/accumulo-scale/conf/site.conf"), new Path(sitePath));
+ fs.copyToLocalFile(new Path(String.format("/accumulo-scale/conf/%s.conf", opts.testId)), new Path(testPath));
+
+ // load configuration file properties
+ Properties scaleProps = new Properties();
+ Properties testProps = new Properties();
+ try {
+ FileInputStream fis = new FileInputStream(sitePath);
+ scaleProps.load(fis);
+ fis.close();
+ fis = new FileInputStream(testPath);
+ testProps.load(fis);
+ } catch (Exception e) {
+ System.out.println("Problem loading config file");
+ e.printStackTrace();
+ }
+
- ScaleTest test = (ScaleTest) Class.forName(String.format("accumulo.test.scalability.%s", opts.testId)).newInstance();
++ ScaleTest test = (ScaleTest) Class.forName(String.format("org.apache.accumulo.test.scalability.%s", opts.testId)).newInstance();
+
+ test.init(scaleProps, testProps, opts.numTabletServers);
+
+ if (opts.action.equalsIgnoreCase("setup")) {
+ test.setup();
+ } else if (opts.action.equalsIgnoreCase("client")) {
+ InetAddress addr = InetAddress.getLocalHost();
+ String host = addr.getHostName();
+ fs.createNewFile(new Path("/accumulo-scale/clients/" + host));
+ test.client();
+ fs.copyFromLocalFile(new Path("/tmp/scale.out"), new Path("/accumulo-scale/results/" + host));
+ } else if (opts.action.equalsIgnoreCase("teardown")) {
+ test.teardown();
+ }
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/accumulo/blob/833ac353/test/system/scalability/README
----------------------------------------------------------------------
diff --cc test/system/scalability/README
index f678061,cd7440c..2e4514d
--- a/test/system/scalability/README
+++ b/test/system/scalability/README
@@@ -8,16 -21,18 +21,18 @@@ data
cp conf/Ingest.conf.example conf/Ingest.conf
- 4. Each test has a unique ID which correlates with its test code in:
+ Each test has a unique ID (e.g., "Ingest") which correlates with its test
+ code in:
- org.apache.accumulo.server.test.scalability.tests.<ID>
+ org.apache.accumulo.test.scalability.tests.<ID>
- This ID correlates with a the following config file:
+ This ID correlates with a config file:
- ./conf/<ID>.conf
+ conf/<ID>.conf
- 5. To run the test, just specifying its ID to the run.py script
+ To run the test, specify its ID to the run.py script.
- export ACCUMULO_HOME=/opt/accumulo; nohup ./run.py Ingest > test1.log 2>&1 &
+ nohup ./run.py Ingest > test1.log 2>&1 &
- 5. A directory will be create where logs and results are placed
+ A timestamped directory will be created, and results are placed in it as each
+ test completes.
http://git-wip-us.apache.org/repos/asf/accumulo/blob/833ac353/test/system/scalability/run.py
----------------------------------------------------------------------
[08/10] git commit: ACCUMULO-2042 Merge branch '1.5.1-SNAPSHOT' into
1.6.0-SNAPSHOT
Posted by bh...@apache.org.
ACCUMULO-2042 Merge branch '1.5.1-SNAPSHOT' into 1.6.0-SNAPSHOT
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/c4cddf97
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/c4cddf97
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/c4cddf97
Branch: refs/heads/1.6.0-SNAPSHOT
Commit: c4cddf97f86d3a20a945c417eb5546f8a0c2e54a
Parents: 98211f5 833ac35
Author: Bill Havanki <bh...@cloudera.com>
Authored: Tue Jan 7 09:19:56 2014 -0500
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Tue Jan 7 09:19:56 2014 -0500
----------------------------------------------------------------------
.../apache/accumulo/test/scalability/Run.java | 2 +-
test/system/scalability/README | 35 ++++++++++++++------
test/system/scalability/run.py | 2 ++
3 files changed, 28 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
[07/10] git commit: ACCUMULO-2042 Merge branch '1.4.5-SNAPSHOT' into
1.5.1-SNAPSHOT
Posted by bh...@apache.org.
ACCUMULO-2042 Merge branch '1.4.5-SNAPSHOT' into 1.5.1-SNAPSHOT
Conflicts:
test/src/main/java/org/apache/accumulo/test/scalability/Run.java
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/833ac353
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/833ac353
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/833ac353
Branch: refs/heads/master
Commit: 833ac35339e91893063aba0f5cde8ff4886b2337
Parents: 21b1b11 a3d77e4
Author: Bill Havanki <bh...@cloudera.com>
Authored: Tue Jan 7 09:19:04 2014 -0500
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Tue Jan 7 09:19:04 2014 -0500
----------------------------------------------------------------------
.../apache/accumulo/test/scalability/Run.java | 2 +-
test/system/scalability/README | 35 ++++++++++++++------
test/system/scalability/run.py | 2 ++
3 files changed, 28 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/833ac353/test/src/main/java/org/apache/accumulo/test/scalability/Run.java
----------------------------------------------------------------------
diff --cc test/src/main/java/org/apache/accumulo/test/scalability/Run.java
index 6622d7c,0000000..0c598c0
mode 100644,000000..100644
--- a/test/src/main/java/org/apache/accumulo/test/scalability/Run.java
+++ b/test/src/main/java/org/apache/accumulo/test/scalability/Run.java
@@@ -1,88 -1,0 +1,88 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.test.scalability;
+
+import java.io.FileInputStream;
+import java.util.Properties;
+import java.net.InetAddress;
+
+import org.apache.accumulo.core.cli.Help;
+import org.apache.accumulo.core.util.CachedConfiguration;
+import org.apache.accumulo.test.scalability.ScaleTest;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import com.beust.jcommander.Parameter;
+
+public class Run {
+
+ static class Opts extends Help {
+ @Parameter(names="--testId", required=true)
+ String testId;
+ @Parameter(names="--action", required=true, description="one of 'setup', 'teardown' or 'client'")
+ String action;
+ @Parameter(names="--count", description="number of tablet servers", required=true)
+ int numTabletServers;
+ }
+
+ public static void main(String[] args) throws Exception {
+
+ final String sitePath = "/tmp/scale-site.conf";
+ final String testPath = "/tmp/scale-test.conf";
+ Opts opts = new Opts();
+ opts.parseArgs(Run.class.getName(), args);
+
+ Configuration conf = CachedConfiguration.getInstance();
+ FileSystem fs;
+ fs = FileSystem.get(conf);
+
+ fs.copyToLocalFile(new Path("/accumulo-scale/conf/site.conf"), new Path(sitePath));
+ fs.copyToLocalFile(new Path(String.format("/accumulo-scale/conf/%s.conf", opts.testId)), new Path(testPath));
+
+ // load configuration file properties
+ Properties scaleProps = new Properties();
+ Properties testProps = new Properties();
+ try {
+ FileInputStream fis = new FileInputStream(sitePath);
+ scaleProps.load(fis);
+ fis.close();
+ fis = new FileInputStream(testPath);
+ testProps.load(fis);
+ } catch (Exception e) {
+ System.out.println("Problem loading config file");
+ e.printStackTrace();
+ }
+
- ScaleTest test = (ScaleTest) Class.forName(String.format("accumulo.test.scalability.%s", opts.testId)).newInstance();
++ ScaleTest test = (ScaleTest) Class.forName(String.format("org.apache.accumulo.test.scalability.%s", opts.testId)).newInstance();
+
+ test.init(scaleProps, testProps, opts.numTabletServers);
+
+ if (opts.action.equalsIgnoreCase("setup")) {
+ test.setup();
+ } else if (opts.action.equalsIgnoreCase("client")) {
+ InetAddress addr = InetAddress.getLocalHost();
+ String host = addr.getHostName();
+ fs.createNewFile(new Path("/accumulo-scale/clients/" + host));
+ test.client();
+ fs.copyFromLocalFile(new Path("/tmp/scale.out"), new Path("/accumulo-scale/results/" + host));
+ } else if (opts.action.equalsIgnoreCase("teardown")) {
+ test.teardown();
+ }
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/accumulo/blob/833ac353/test/system/scalability/README
----------------------------------------------------------------------
diff --cc test/system/scalability/README
index f678061,cd7440c..2e4514d
--- a/test/system/scalability/README
+++ b/test/system/scalability/README
@@@ -8,16 -21,18 +21,18 @@@ data
cp conf/Ingest.conf.example conf/Ingest.conf
- 4. Each test has a unique ID which correlates with its test code in:
+ Each test has a unique ID (e.g., "Ingest") which correlates with its test
+ code in:
- org.apache.accumulo.server.test.scalability.tests.<ID>
+ org.apache.accumulo.test.scalability.tests.<ID>
- This ID correlates with a the following config file:
+ This ID correlates with a config file:
- ./conf/<ID>.conf
+ conf/<ID>.conf
- 5. To run the test, just specifying its ID to the run.py script
+ To run the test, specify its ID to the run.py script.
- export ACCUMULO_HOME=/opt/accumulo; nohup ./run.py Ingest > test1.log 2>&1 &
+ nohup ./run.py Ingest > test1.log 2>&1 &
- 5. A directory will be create where logs and results are placed
+ A timestamped directory will be created, and results are placed in it as each
+ test completes.
http://git-wip-us.apache.org/repos/asf/accumulo/blob/833ac353/test/system/scalability/run.py
----------------------------------------------------------------------
[02/10] git commit: ACCUMULO-2042 Get scalability test running
Posted by bh...@apache.org.
ACCUMULO-2042 Get scalability test running
Two problems prevented scalability tests from being run properly.
1. The Run class was looking for test classes under the "accumulo"
package instead of "org.apache.accumulo".
2. A results subdirectory on the local filesystem was not being
created, and so result data from HDFS could not be copied out.
This commit also updates the scalability test README with more
detail.
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/a3d77e4f
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/a3d77e4f
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/a3d77e4f
Branch: refs/heads/1.5.1-SNAPSHOT
Commit: a3d77e4ff2a12f161f1343490f42aa91d5b109d3
Parents: 8f9fe41
Author: Bill Havanki <bh...@cloudera.com>
Authored: Thu Dec 19 11:15:44 2013 -0500
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Tue Jan 7 09:10:04 2014 -0500
----------------------------------------------------------------------
.../accumulo/server/test/scalability/Run.java | 2 +-
test/system/scalability/README | 35 ++++++++++++++------
test/system/scalability/run.py | 2 ++
3 files changed, 28 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3d77e4f/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
----------------------------------------------------------------------
diff --git a/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java b/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
index 63861f8..7fb78c7 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
@@ -59,7 +59,7 @@ public class Run {
e.printStackTrace();
}
- ScaleTest test = (ScaleTest) Class.forName(String.format("accumulo.server.test.scalability.%s", testId)).newInstance();
+ ScaleTest test = (ScaleTest) Class.forName(String.format("org.apache.accumulo.server.test.scalability.%s", testId)).newInstance();
test.init(scaleProps, testProps, numTabletServers);
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3d77e4f/test/system/scalability/README
----------------------------------------------------------------------
diff --git a/test/system/scalability/README b/test/system/scalability/README
index 7206daa..cd7440c 100644
--- a/test/system/scalability/README
+++ b/test/system/scalability/README
@@ -1,23 +1,38 @@
-1. To run scalability tests, set up the site configuration (used by all tests)
+Apache Accumulo Scalability Tests
- cp conf/site.conf.example conf/site.conf
+The scalability test framework needs to be configured for your Accumulo
+instance by performing the following steps.
+
+WARNING: Each scalability test rewrites your conf/slaves file and reinitializes
+your Accumulo instance. Do not run these tests on a cluster holding essential
+data.
+
+1. Make sure you have both ACCUMULO_HOME and HADOOP_HOME set in your
+ $ACCUMULO_CONF_DIR/accumulo-env.sh.
-2. Modify site.conf to the correct configuration parameters for your Accumulo instance
+2. Create a 'site.conf' file in the conf directory containing settings
+ needed by test nodes to connect to Accumulo, and to guide the tests.
+
+ cp conf/site.conf.example conf/site.conf
-3. Set up the configuration file for the test you want to run
+3. Create an 'Ingest.conf' file in the conf directory containing performance
+ settings for the Ingest test. (This test is currently the only scalability
+ test available.)
cp conf/Ingest.conf.example conf/Ingest.conf
-4. Each test has a unique ID which correlates with its test code in:
+ Each test has a unique ID (e.g., "Ingest") which correlates with its test
+ code in:
org.apache.accumulo.server.test.scalability.tests.<ID>
- This ID correlates with a the following config file:
+ This ID correlates with a config file:
- ./conf/<ID>.conf
+ conf/<ID>.conf
-5. To run the test, just specifying its ID to the run.py script
+To run the test, specify its ID to the run.py script.
- export ACCUMULO_HOME=/opt/accumulo; nohup ./run.py Ingest > test1.log 2>&1 &
+ nohup ./run.py Ingest > test1.log 2>&1 &
-5. A directory will be create where logs and results are placed
+A timestamped directory will be created, and results are placed in it as each
+test completes.
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3d77e4f/test/system/scalability/run.py
----------------------------------------------------------------------
diff --git a/test/system/scalability/run.py b/test/system/scalability/run.py
index ac45aa2..938805a 100755
--- a/test/system/scalability/run.py
+++ b/test/system/scalability/run.py
@@ -189,8 +189,10 @@ def main():
log('Creating test directory structure')
testDir = 'test-%d' % time.time()
nodesDir = testDir+'/nodes'
+ resultsDir = testDir+'/results'
syscall('mkdir %s' % testDir)
syscall('mkdir %s' % nodesDir)
+ syscall('mkdir %s' % resultsDir)
log('Removing current /accumulo-scale directory')
syscall('hadoop fs -rmr /accumulo-scale')
[09/10] git commit: ACCUMULO-2042 Merge branch '1.5.1-SNAPSHOT' into
1.6.0-SNAPSHOT
Posted by bh...@apache.org.
ACCUMULO-2042 Merge branch '1.5.1-SNAPSHOT' into 1.6.0-SNAPSHOT
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/c4cddf97
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/c4cddf97
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/c4cddf97
Branch: refs/heads/master
Commit: c4cddf97f86d3a20a945c417eb5546f8a0c2e54a
Parents: 98211f5 833ac35
Author: Bill Havanki <bh...@cloudera.com>
Authored: Tue Jan 7 09:19:56 2014 -0500
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Tue Jan 7 09:19:56 2014 -0500
----------------------------------------------------------------------
.../apache/accumulo/test/scalability/Run.java | 2 +-
test/system/scalability/README | 35 ++++++++++++++------
test/system/scalability/run.py | 2 ++
3 files changed, 28 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
[03/10] git commit: ACCUMULO-2042 Get scalability test running
Posted by bh...@apache.org.
ACCUMULO-2042 Get scalability test running
Two problems prevented scalability tests from being run properly.
1. The Run class was looking for test classes under the "accumulo"
package instead of "org.apache.accumulo".
2. A results subdirectory on the local filesystem was not being
created, and so result data from HDFS could not be copied out.
This commit also updates the scalability test README with more
detail.
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/a3d77e4f
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/a3d77e4f
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/a3d77e4f
Branch: refs/heads/1.6.0-SNAPSHOT
Commit: a3d77e4ff2a12f161f1343490f42aa91d5b109d3
Parents: 8f9fe41
Author: Bill Havanki <bh...@cloudera.com>
Authored: Thu Dec 19 11:15:44 2013 -0500
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Tue Jan 7 09:10:04 2014 -0500
----------------------------------------------------------------------
.../accumulo/server/test/scalability/Run.java | 2 +-
test/system/scalability/README | 35 ++++++++++++++------
test/system/scalability/run.py | 2 ++
3 files changed, 28 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3d77e4f/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
----------------------------------------------------------------------
diff --git a/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java b/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
index 63861f8..7fb78c7 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
@@ -59,7 +59,7 @@ public class Run {
e.printStackTrace();
}
- ScaleTest test = (ScaleTest) Class.forName(String.format("accumulo.server.test.scalability.%s", testId)).newInstance();
+ ScaleTest test = (ScaleTest) Class.forName(String.format("org.apache.accumulo.server.test.scalability.%s", testId)).newInstance();
test.init(scaleProps, testProps, numTabletServers);
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3d77e4f/test/system/scalability/README
----------------------------------------------------------------------
diff --git a/test/system/scalability/README b/test/system/scalability/README
index 7206daa..cd7440c 100644
--- a/test/system/scalability/README
+++ b/test/system/scalability/README
@@ -1,23 +1,38 @@
-1. To run scalability tests, set up the site configuration (used by all tests)
+Apache Accumulo Scalability Tests
- cp conf/site.conf.example conf/site.conf
+The scalability test framework needs to be configured for your Accumulo
+instance by performing the following steps.
+
+WARNING: Each scalability test rewrites your conf/slaves file and reinitializes
+your Accumulo instance. Do not run these tests on a cluster holding essential
+data.
+
+1. Make sure you have both ACCUMULO_HOME and HADOOP_HOME set in your
+ $ACCUMULO_CONF_DIR/accumulo-env.sh.
-2. Modify site.conf to the correct configuration parameters for your Accumulo instance
+2. Create a 'site.conf' file in the conf directory containing settings
+ needed by test nodes to connect to Accumulo, and to guide the tests.
+
+ cp conf/site.conf.example conf/site.conf
-3. Set up the configuration file for the test you want to run
+3. Create an 'Ingest.conf' file in the conf directory containing performance
+ settings for the Ingest test. (This test is currently the only scalability
+ test available.)
cp conf/Ingest.conf.example conf/Ingest.conf
-4. Each test has a unique ID which correlates with its test code in:
+ Each test has a unique ID (e.g., "Ingest") which correlates with its test
+ code in:
org.apache.accumulo.server.test.scalability.tests.<ID>
- This ID correlates with a the following config file:
+ This ID correlates with a config file:
- ./conf/<ID>.conf
+ conf/<ID>.conf
-5. To run the test, just specifying its ID to the run.py script
+To run the test, specify its ID to the run.py script.
- export ACCUMULO_HOME=/opt/accumulo; nohup ./run.py Ingest > test1.log 2>&1 &
+ nohup ./run.py Ingest > test1.log 2>&1 &
-5. A directory will be create where logs and results are placed
+A timestamped directory will be created, and results are placed in it as each
+test completes.
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3d77e4f/test/system/scalability/run.py
----------------------------------------------------------------------
diff --git a/test/system/scalability/run.py b/test/system/scalability/run.py
index ac45aa2..938805a 100755
--- a/test/system/scalability/run.py
+++ b/test/system/scalability/run.py
@@ -189,8 +189,10 @@ def main():
log('Creating test directory structure')
testDir = 'test-%d' % time.time()
nodesDir = testDir+'/nodes'
+ resultsDir = testDir+'/results'
syscall('mkdir %s' % testDir)
syscall('mkdir %s' % nodesDir)
+ syscall('mkdir %s' % resultsDir)
log('Removing current /accumulo-scale directory')
syscall('hadoop fs -rmr /accumulo-scale')
[05/10] git commit: ACCUMULO-2042 Merge branch '1.4.5-SNAPSHOT' into
1.5.1-SNAPSHOT
Posted by bh...@apache.org.
ACCUMULO-2042 Merge branch '1.4.5-SNAPSHOT' into 1.5.1-SNAPSHOT
Conflicts:
test/src/main/java/org/apache/accumulo/test/scalability/Run.java
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/833ac353
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/833ac353
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/833ac353
Branch: refs/heads/1.6.0-SNAPSHOT
Commit: 833ac35339e91893063aba0f5cde8ff4886b2337
Parents: 21b1b11 a3d77e4
Author: Bill Havanki <bh...@cloudera.com>
Authored: Tue Jan 7 09:19:04 2014 -0500
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Tue Jan 7 09:19:04 2014 -0500
----------------------------------------------------------------------
.../apache/accumulo/test/scalability/Run.java | 2 +-
test/system/scalability/README | 35 ++++++++++++++------
test/system/scalability/run.py | 2 ++
3 files changed, 28 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/833ac353/test/src/main/java/org/apache/accumulo/test/scalability/Run.java
----------------------------------------------------------------------
diff --cc test/src/main/java/org/apache/accumulo/test/scalability/Run.java
index 6622d7c,0000000..0c598c0
mode 100644,000000..100644
--- a/test/src/main/java/org/apache/accumulo/test/scalability/Run.java
+++ b/test/src/main/java/org/apache/accumulo/test/scalability/Run.java
@@@ -1,88 -1,0 +1,88 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.test.scalability;
+
+import java.io.FileInputStream;
+import java.util.Properties;
+import java.net.InetAddress;
+
+import org.apache.accumulo.core.cli.Help;
+import org.apache.accumulo.core.util.CachedConfiguration;
+import org.apache.accumulo.test.scalability.ScaleTest;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import com.beust.jcommander.Parameter;
+
+public class Run {
+
+ static class Opts extends Help {
+ @Parameter(names="--testId", required=true)
+ String testId;
+ @Parameter(names="--action", required=true, description="one of 'setup', 'teardown' or 'client'")
+ String action;
+ @Parameter(names="--count", description="number of tablet servers", required=true)
+ int numTabletServers;
+ }
+
+ public static void main(String[] args) throws Exception {
+
+ final String sitePath = "/tmp/scale-site.conf";
+ final String testPath = "/tmp/scale-test.conf";
+ Opts opts = new Opts();
+ opts.parseArgs(Run.class.getName(), args);
+
+ Configuration conf = CachedConfiguration.getInstance();
+ FileSystem fs;
+ fs = FileSystem.get(conf);
+
+ fs.copyToLocalFile(new Path("/accumulo-scale/conf/site.conf"), new Path(sitePath));
+ fs.copyToLocalFile(new Path(String.format("/accumulo-scale/conf/%s.conf", opts.testId)), new Path(testPath));
+
+ // load configuration file properties
+ Properties scaleProps = new Properties();
+ Properties testProps = new Properties();
+ try {
+ FileInputStream fis = new FileInputStream(sitePath);
+ scaleProps.load(fis);
+ fis.close();
+ fis = new FileInputStream(testPath);
+ testProps.load(fis);
+ } catch (Exception e) {
+ System.out.println("Problem loading config file");
+ e.printStackTrace();
+ }
+
- ScaleTest test = (ScaleTest) Class.forName(String.format("accumulo.test.scalability.%s", opts.testId)).newInstance();
++ ScaleTest test = (ScaleTest) Class.forName(String.format("org.apache.accumulo.test.scalability.%s", opts.testId)).newInstance();
+
+ test.init(scaleProps, testProps, opts.numTabletServers);
+
+ if (opts.action.equalsIgnoreCase("setup")) {
+ test.setup();
+ } else if (opts.action.equalsIgnoreCase("client")) {
+ InetAddress addr = InetAddress.getLocalHost();
+ String host = addr.getHostName();
+ fs.createNewFile(new Path("/accumulo-scale/clients/" + host));
+ test.client();
+ fs.copyFromLocalFile(new Path("/tmp/scale.out"), new Path("/accumulo-scale/results/" + host));
+ } else if (opts.action.equalsIgnoreCase("teardown")) {
+ test.teardown();
+ }
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/accumulo/blob/833ac353/test/system/scalability/README
----------------------------------------------------------------------
diff --cc test/system/scalability/README
index f678061,cd7440c..2e4514d
--- a/test/system/scalability/README
+++ b/test/system/scalability/README
@@@ -8,16 -21,18 +21,18 @@@ data
cp conf/Ingest.conf.example conf/Ingest.conf
- 4. Each test has a unique ID which correlates with its test code in:
+ Each test has a unique ID (e.g., "Ingest") which correlates with its test
+ code in:
- org.apache.accumulo.server.test.scalability.tests.<ID>
+ org.apache.accumulo.test.scalability.tests.<ID>
- This ID correlates with a the following config file:
+ This ID correlates with a config file:
- ./conf/<ID>.conf
+ conf/<ID>.conf
- 5. To run the test, just specifying its ID to the run.py script
+ To run the test, specify its ID to the run.py script.
- export ACCUMULO_HOME=/opt/accumulo; nohup ./run.py Ingest > test1.log 2>&1 &
+ nohup ./run.py Ingest > test1.log 2>&1 &
- 5. A directory will be create where logs and results are placed
+ A timestamped directory will be created, and results are placed in it as each
+ test completes.
http://git-wip-us.apache.org/repos/asf/accumulo/blob/833ac353/test/system/scalability/run.py
----------------------------------------------------------------------
[10/10] git commit: ACCUMULO-2042 Merge branch '1.6.0-SNAPSHOT'
Posted by bh...@apache.org.
ACCUMULO-2042 Merge branch '1.6.0-SNAPSHOT'
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/a72e80ca
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/a72e80ca
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/a72e80ca
Branch: refs/heads/master
Commit: a72e80ca45962e807b5195e5f619e8da5b537c43
Parents: a2652c3 c4cddf9
Author: Bill Havanki <bh...@cloudera.com>
Authored: Tue Jan 7 09:21:47 2014 -0500
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Tue Jan 7 09:21:47 2014 -0500
----------------------------------------------------------------------
.../apache/accumulo/test/scalability/Run.java | 2 +-
test/system/scalability/README | 35 ++++++++++++++------
test/system/scalability/run.py | 2 ++
3 files changed, 28 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
[04/10] git commit: ACCUMULO-2042 Get scalability test running
Posted by bh...@apache.org.
ACCUMULO-2042 Get scalability test running
Two problems prevented scalability tests from being run properly.
1. The Run class was looking for test classes under the "accumulo"
package instead of "org.apache.accumulo".
2. A results subdirectory on the local filesystem was not being
created, and so result data from HDFS could not be copied out.
This commit also updates the scalability test README with more
detail.
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/a3d77e4f
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/a3d77e4f
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/a3d77e4f
Branch: refs/heads/master
Commit: a3d77e4ff2a12f161f1343490f42aa91d5b109d3
Parents: 8f9fe41
Author: Bill Havanki <bh...@cloudera.com>
Authored: Thu Dec 19 11:15:44 2013 -0500
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Tue Jan 7 09:10:04 2014 -0500
----------------------------------------------------------------------
.../accumulo/server/test/scalability/Run.java | 2 +-
test/system/scalability/README | 35 ++++++++++++++------
test/system/scalability/run.py | 2 ++
3 files changed, 28 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3d77e4f/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
----------------------------------------------------------------------
diff --git a/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java b/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
index 63861f8..7fb78c7 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/test/scalability/Run.java
@@ -59,7 +59,7 @@ public class Run {
e.printStackTrace();
}
- ScaleTest test = (ScaleTest) Class.forName(String.format("accumulo.server.test.scalability.%s", testId)).newInstance();
+ ScaleTest test = (ScaleTest) Class.forName(String.format("org.apache.accumulo.server.test.scalability.%s", testId)).newInstance();
test.init(scaleProps, testProps, numTabletServers);
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3d77e4f/test/system/scalability/README
----------------------------------------------------------------------
diff --git a/test/system/scalability/README b/test/system/scalability/README
index 7206daa..cd7440c 100644
--- a/test/system/scalability/README
+++ b/test/system/scalability/README
@@ -1,23 +1,38 @@
-1. To run scalability tests, set up the site configuration (used by all tests)
+Apache Accumulo Scalability Tests
- cp conf/site.conf.example conf/site.conf
+The scalability test framework needs to be configured for your Accumulo
+instance by performing the following steps.
+
+WARNING: Each scalability test rewrites your conf/slaves file and reinitializes
+your Accumulo instance. Do not run these tests on a cluster holding essential
+data.
+
+1. Make sure you have both ACCUMULO_HOME and HADOOP_HOME set in your
+ $ACCUMULO_CONF_DIR/accumulo-env.sh.
-2. Modify site.conf to the correct configuration parameters for your Accumulo instance
+2. Create a 'site.conf' file in the conf directory containing settings
+ needed by test nodes to connect to Accumulo, and to guide the tests.
+
+ cp conf/site.conf.example conf/site.conf
-3. Set up the configuration file for the test you want to run
+3. Create an 'Ingest.conf' file in the conf directory containing performance
+ settings for the Ingest test. (This test is currently the only scalability
+ test available.)
cp conf/Ingest.conf.example conf/Ingest.conf
-4. Each test has a unique ID which correlates with its test code in:
+ Each test has a unique ID (e.g., "Ingest") which correlates with its test
+ code in:
org.apache.accumulo.server.test.scalability.tests.<ID>
- This ID correlates with a the following config file:
+ This ID correlates with a config file:
- ./conf/<ID>.conf
+ conf/<ID>.conf
-5. To run the test, just specifying its ID to the run.py script
+To run the test, specify its ID to the run.py script.
- export ACCUMULO_HOME=/opt/accumulo; nohup ./run.py Ingest > test1.log 2>&1 &
+ nohup ./run.py Ingest > test1.log 2>&1 &
-5. A directory will be create where logs and results are placed
+A timestamped directory will be created, and results are placed in it as each
+test completes.
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3d77e4f/test/system/scalability/run.py
----------------------------------------------------------------------
diff --git a/test/system/scalability/run.py b/test/system/scalability/run.py
index ac45aa2..938805a 100755
--- a/test/system/scalability/run.py
+++ b/test/system/scalability/run.py
@@ -189,8 +189,10 @@ def main():
log('Creating test directory structure')
testDir = 'test-%d' % time.time()
nodesDir = testDir+'/nodes'
+ resultsDir = testDir+'/results'
syscall('mkdir %s' % testDir)
syscall('mkdir %s' % nodesDir)
+ syscall('mkdir %s' % resultsDir)
log('Removing current /accumulo-scale directory')
syscall('hadoop fs -rmr /accumulo-scale')