You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by mikewalch <gi...@git.apache.org> on 2017/01/23 22:05:54 UTC
[GitHub] accumulo-testing pull request #2: ACCUMULO-4510 Refactored Continous Ingest ...
GitHub user mikewalch opened a pull request:
https://github.com/apache/accumulo-testing/pull/2
ACCUMULO-4510 Refactored Continous Ingest tests
* Continuous ingest applications can be launched locally or in YARN and
are now configured by accumulo-testing.properties file
* Applications are now launched using the accumulo-testing command
rather than custom bash scripts
* Removed continuous stats collector as its uses Accumulo
internals and corresponding report generation code.
* Agitator was seperated from continuous ingests. It is
run by accumulo-testing command and configured by
accumulo-testing-env.sh
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mikewalch/accumulo-testing ci-update
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/accumulo-testing/pull/2.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2
----
commit 1ff2e801d7564e75fb2ff8ccc57fc277ca0941b2
Author: Mike Walch <mw...@apache.org>
Date: 2017-01-12T16:03:41Z
ACCUMULO-4510 Refactored Continous Ingest tests
* Continuous ingest applications can be launched locally or in YARN and
are now configured by accumulo-testing.properties file
* Applications are now launched using the accumulo-testing command
rather than custom bash scripts
* Removed continuous stats collector as its uses Accumulo
internals and corresponding report generation code.
* Agitator was seperated from continuous ingests. It is
run by accumulo-testing command and configured by
accumulo-testing-env.sh
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #2: ACCUMULO-4510 Refactored Continous Ingest ...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo-testing/pull/2#discussion_r97820053
--- Diff: README.md ---
@@ -46,9 +46,88 @@ walkers in 5 containers in YARN using the Image.xml module.
./bin/accumulo-testing rw-yarn 5 Image.xml
-This command will create an application in YARN and exit when all containers for the test have started.
-While its running, you can view logs for each random walker using the YARN resource manager. The YARN
-application can be killed at any time using the YARN resource manager or command line tool.
+This command will create an application in YARN and exit when all containers for the test have
+started. While its running, you can view logs for each random walker using the YARN resource manager.
+The YARN application can be killed at any time using the YARN resource manager or command line tool.
+
+## Continuous Ingest & Query
+
+The Continuous Ingest test consists of several applications that place a continuous query and ingest
+load on Accumulo to see if it breaks.
--- End diff --
i will update
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #2: ACCUMULO-4510 Refactored Continous Ingest ...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo-testing/pull/2#discussion_r97819994
--- Diff: README.md ---
@@ -32,9 +32,9 @@ on your machine as well as an Accumulo instance to use for testing.
The random walk test generates client behavior on an Apache Accumulo instance by randomly walking a
graph of client operations.
-Before running random walk, review the `test.common.*` and `test.randomwalk.*` properties in
-`accumulo-testing.properties` file. A test module must also be specified. See the [modules][modules]
-directory for a list of available ones.
+Before running random walk, review the `test.common.*` properties in `accumulo-testing.properties`
+file. A test module must also be specified. See the [modules][modules] directory for a list of
--- End diff --
i will fix
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #2: ACCUMULO-4510 Refactored Continous Ingest ...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo-testing/pull/2#discussion_r97805152
--- Diff: README.md ---
@@ -46,9 +46,88 @@ walkers in 5 containers in YARN using the Image.xml module.
./bin/accumulo-testing rw-yarn 5 Image.xml
-This command will create an application in YARN and exit when all containers for the test have started.
-While its running, you can view logs for each random walker using the YARN resource manager. The YARN
-application can be killed at any time using the YARN resource manager or command line tool.
+This command will create an application in YARN and exit when all containers for the test have
+started. While its running, you can view logs for each random walker using the YARN resource manager.
+The YARN application can be killed at any time using the YARN resource manager or command line tool.
+
+## Continuous Ingest & Query
+
+The Continuous Ingest test consists of several applications that place a continuous query and ingest
+load on Accumulo to see if it breaks.
--- End diff --
I think it would be useful to state that CI creates lots of large linked list and the test suite can look for holes in these linked list to see if Accumulo lost data. Could pull from [goraci]'s background section.
[goraci]: https://github.com/keith-turner/goraci
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #2: ACCUMULO-4510 Refactored Continous Ingest ...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo-testing/pull/2#discussion_r97820864
--- Diff: core/src/main/java/org/apache/accumulo/testing/core/continuous/ContinuousIngest.java ---
@@ -183,8 +177,9 @@ private static long flush(BatchWriter bw, long count, final int flushInterval, l
return lastFlushTime;
}
- public static Mutation genMutation(long rowLong, int cfInt, int cqInt, ColumnVisibility cv, byte[] ingestInstanceId, long count, byte[] prevRow, Random r,
- boolean checksum) {
+ static Mutation genMutation(long rowLong, int cfInt, int cqInt, ColumnVisibility cv,
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #2: ACCUMULO-4510 Refactored Continous Ingest ...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo-testing/pull/2#discussion_r97820268
--- Diff: core/src/main/java/org/apache/accumulo/testing/core/continuous/ContinuousIngest.java ---
@@ -183,8 +177,9 @@ private static long flush(BatchWriter bw, long count, final int flushInterval, l
return lastFlushTime;
}
- public static Mutation genMutation(long rowLong, int cfInt, int cqInt, ColumnVisibility cv, byte[] ingestInstanceId, long count, byte[] prevRow, Random r,
- boolean checksum) {
+ static Mutation genMutation(long rowLong, int cfInt, int cqInt, ColumnVisibility cv,
--- End diff --
I plan on adding the java formatter plugin in a later pull request.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #2: ACCUMULO-4510 Refactored Continous Ingest ...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo-testing/pull/2#discussion_r97806863
--- Diff: core/src/main/java/org/apache/accumulo/testing/core/continuous/ContinuousIngest.java ---
@@ -51,49 +46,44 @@
private static List<ColumnVisibility> visibilities;
- private static void initVisibilities(ContinuousOpts opts) throws Exception {
- if (opts.visFile == null) {
- visibilities = Collections.singletonList(new ColumnVisibility());
- return;
- }
-
- visibilities = new ArrayList<>();
-
- FileSystem fs = FileSystem.get(new Configuration());
- BufferedReader in = new BufferedReader(new InputStreamReader(fs.open(new Path(opts.visFile)), UTF_8));
-
- String line;
-
- while ((line = in.readLine()) != null) {
- visibilities.add(new ColumnVisibility(line));
- }
-
- in.close();
- }
-
private static ColumnVisibility getVisibility(Random rand) {
return visibilities.get(rand.nextInt(visibilities.size()));
}
public static void main(String[] args) throws Exception {
- ContinuousOpts opts = new ContinuousOpts();
- BatchWriterOpts bwOpts = new BatchWriterOpts();
- ClientOnDefaultTable clientOpts = new ClientOnDefaultTable("ci");
- clientOpts.parseArgs(ContinuousIngest.class.getName(), args, bwOpts, opts);
+ if (args.length != 1) {
+ System.err.println("Usage: ContinuousIngest <propsPath>");
--- End diff --
this is nice, replacing all of the command line options with a props file
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #2: ACCUMULO-4510 Refactored Continous Ingest ...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo-testing/pull/2#discussion_r97805921
--- Diff: bin/accumulo-testing ---
@@ -25,8 +25,16 @@ function print_usage() {
Usage: accumulo-testing <command> (<argument>)
Possible commands:
- rw-local <module> Runs randomwalk <module> in local java process
- rw-yarn <num> <module> Runs randomwalk <module> in <num> containers on YARN
+ agitator <command> Runs agitator <command>. Available commands: start, stop
+ ci-createtable Creates Accumulo table with splits for continuous ingest tests
+ ci-local <application> Runs continuous ingest <application> in local java process
--- End diff --
this is nice
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #2: ACCUMULO-4510 Refactored Continous Ingest ...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo-testing/pull/2#discussion_r97808927
--- Diff: core/src/main/java/org/apache/accumulo/testing/core/continuous/ContinuousIngest.java ---
@@ -183,8 +177,9 @@ private static long flush(BatchWriter bw, long count, final int flushInterval, l
return lastFlushTime;
}
- public static Mutation genMutation(long rowLong, int cfInt, int cqInt, ColumnVisibility cv, byte[] ingestInstanceId, long count, byte[] prevRow, Random r,
- boolean checksum) {
+ static Mutation genMutation(long rowLong, int cfInt, int cqInt, ColumnVisibility cv,
--- End diff --
Why was this reformatted? Does maven format?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #2: ACCUMULO-4510 Refactored Continous Ingest ...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo-testing/pull/2#discussion_r97809185
--- Diff: core/src/main/java/org/apache/accumulo/testing/core/continuous/ContinuousScanner.java ---
@@ -20,49 +20,44 @@
import java.util.Iterator;
import java.util.Map.Entry;
+import java.util.Properties;
import java.util.Random;
import java.util.concurrent.TimeUnit;
-import org.apache.accumulo.core.cli.ClientOnDefaultTable;
-import org.apache.accumulo.core.cli.ScannerOpts;
import org.apache.accumulo.core.client.Connector;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.testing.core.TestProps;
import org.apache.hadoop.io.Text;
-import com.beust.jcommander.Parameter;
-import com.beust.jcommander.validators.PositiveInteger;
import static com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly;
public class ContinuousScanner {
- static class Opts extends ContinuousWalk.Opts {
- @Parameter(names = "--numToScan", description = "Number rows to scan between sleeps", required = true, validateWith = PositiveInteger.class)
- long numToScan = 0;
- }
-
public static void main(String[] args) throws Exception {
- Opts opts = new Opts();
- ScannerOpts scanOpts = new ScannerOpts();
- ClientOnDefaultTable clientOpts = new ClientOnDefaultTable("ci");
- clientOpts.parseArgs(ContinuousScanner.class.getName(), args, scanOpts, opts);
+
+ Properties props = TestProps.loadFromFile(args[0]);
+ ContinuousEnv env = new ContinuousEnv(props);
--- End diff --
I like ContinuousEnv
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #2: ACCUMULO-4510 Refactored Continous Ingest ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/accumulo-testing/pull/2
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #2: ACCUMULO-4510 Refactored Continous Ingest ...
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo-testing/pull/2#discussion_r97804309
--- Diff: README.md ---
@@ -32,9 +32,9 @@ on your machine as well as an Accumulo instance to use for testing.
The random walk test generates client behavior on an Apache Accumulo instance by randomly walking a
graph of client operations.
-Before running random walk, review the `test.common.*` and `test.randomwalk.*` properties in
-`accumulo-testing.properties` file. A test module must also be specified. See the [modules][modules]
-directory for a list of available ones.
+Before running random walk, review the `test.common.*` properties in `accumulo-testing.properties`
+file. A test module must also be specified. See the [modules][modules] directory for a list of
--- End diff --
I think you can replace `[modules][modules]` with `[modules]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---