You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by bh...@apache.org on 2014/06/26 23:29:25 UTC

[2/6] git commit: ACCUMULO-2850 Add enhanced memory stress test scripts

ACCUMULO-2850 Add enhanced memory stress test scripts

New scripts for the memory stress test make it easy to start and stop multiple
reader and writer processes, in a manner similar to continuous ingest. See the
updated README.md for information on how the scripts are used.


Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/e01a7206
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/e01a7206
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/e01a7206

Branch: refs/heads/1.6.1-SNAPSHOT
Commit: e01a7206220a0ed143815416f28730fbdeb8c818
Parents: 2422d49
Author: Bill Havanki <bh...@cloudera.com>
Authored: Wed Jun 25 13:30:33 2014 -0400
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Thu Jun 26 17:26:40 2014 -0400

----------------------------------------------------------------------
 test/system/stress/README.md             | 89 +++++++++++++++++++++++----
 test/system/stress/reader.sh             | 42 ++++++-------
 test/system/stress/readers               | 17 +++++
 test/system/stress/start-readers.sh      | 40 ++++++++++++
 test/system/stress/start-writers.sh      | 40 ++++++++++++
 test/system/stress/stop-readers.sh       | 36 +++++++++++
 test/system/stress/stop-writers.sh       | 36 +++++++++++
 test/system/stress/stress-env.sh.example | 56 +++++++++++++++++
 test/system/stress/writer.sh             | 44 +++++++------
 test/system/stress/writers               | 17 +++++
 10 files changed, 362 insertions(+), 55 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/README.md
----------------------------------------------------------------------
diff --git a/test/system/stress/README.md b/test/system/stress/README.md
index 1a0bbae..e400d4b 100644
--- a/test/system/stress/README.md
+++ b/test/system/stress/README.md
@@ -4,10 +4,10 @@ contributor license agreements.  See the NOTICE file distributed with
 this work for additional information regarding copyright ownership.
 The ASF licenses this file to You under the Apache License, Version 2.0
 (the "License"); you may not use this file except in compliance with
-the License.  You may obtain a copy of the License at 
- 
+the License.  You may obtain a copy of the License at
+
     http://www.apache.org/licenses/LICENSE-2.0
- 
+
 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@@ -15,19 +15,86 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
-
 Accumulo Stress Test
---------------------
+====================
+
+This is an attempt to observe the behavior Accumulo displays when compacting and
+reading cells. There are two components to this package:
+`org.apache.accumulo.test.stress.random.Write` and
+`org.apache.accumulo.test.stress.random.Scan`.
+
+Since this test is meant to stress the system, there is no pre-defined test
+period and no success criteria that we can programmatically check. During a
+successful test period, the Accumulo cluster should remain stable and
+functional. Possible failure conditions include component failures, such as
+tablet servers running out of memory.
+
+The `org.apache.accumulo.test.stress.random.Write` utility provides facilities
+for writing random sized cells. Users can configure minimum and maximum sized
+portions of a cell. The portions users can configure are the row, column family,
+column qualifier and value. Note that the sizes are uniformly distributed
+between the minimum and maximum values.
+
+The `org.apache.accumulo.test.stress.random.Scan` utility provides users with
+the ability to query tables generated by the Write. It will pick a tablet at
+random and scan the entire range. The amount of times this process is performed
+is user configurable. By default, it happens 1,024 times. Users can also specify
+whether or not the scan should be isolated.
+
+There is no shared state intended by either of these services. This allows
+multiple clients to be run in parallel, either on the same host or distributed
+across hosts.
+
+## Prerequisites
+
+The utilities are run on a normal, initialized Accumulo cluster. By default,
+they only work with a table named "stress_test".
+
+The start and stop scripts rely on pssh. Before running them, you will need
+to install pssh on the machines that will be controlled by them.
+
+## Running
+
+Copy `stress-env.sh.example` to `stress-env.sh` and edit it, supplying at least
+correct values for the USERPASS and INSTANCE variables. The remainder of this
+file describes and provides defaults for many of the configuration options
+for the stress utilities.
+
+### Individual Execution
+
+Before starting a fresh write test, clear out the test table using the Accumulo
+shell.
+
+    > deletetable -f stress_test
+    > createtable stress_test
+
+To run a writer:
+
+    $ ./writer.sh
+
+The writer will begin logging into a "logs" subdirectory, using timestamped
+filenames. You can stop the writer by killing it or using Control-C.
+
+To run a reader:
+
+    $ ./reader.sh
 
-This is an attempt to observe the behavior Accumulo displays when compacting and reading these cells. There are two components to this package: `org.apache.accumulo.test.stress.random.Write` and `org.apache.accumulo.test.stress.random.Scan`.
+The reader logs like the writer. By default, the reader reads forever, but
+it can be configured for a fixed number of scan iterations.
 
-Since this test is meant to stress the system, there is no pre-defined test period and no success criteria that we can programmatically check. During a successful test period, the Accumulo cluster should remain stable and functional. Possible failure conditions include component failures, such as tablet servers running out of memory.
+### Group Execution
 
-The `org.apache.accumulo.test.stress.random.Write` utility provides facilities for writing random sized cells. Users can configure minimum and maximum sized portions of a cell. The portions users can configure are the row, column family, column qualifier and value. Note that the sizes are uniformly distributed between the minimum and maximum values
+To run writers across multiple machines, create a writers file and list each
+host that will run a writer, one per line.
 
-The `org.apache.accumulo.test.stress.random.Scan` utility provides users with the ability to query tables generated by the Write. It will pick a tablet at random and scan the entire range. The amount of times this process is performed is user configurable. By default, it happens 1,024 times. Users can also specify whether or not the scan should be isolated or not.
+Finally, to start and stop the writers:
 
-There is no shared state intended by either of these services. This allows multiple clients to be run in parallel, either on the same host or distributedacross hosts.
+    $ ./start-writers.sh
+    $ ./stop-writers.sh
 
-See the provided `test/system/stress/writer.sh` and `test/system/stress/reader.sh` scripts for example configuration options and how to launch the processes.
+For readers, make a similar readers file, and use `start-readers.sh` and
+`stop-readers.sh`.
 
+The start scripts copy out stress-env.sh to each of the writers and readers,
+overwriting what is there, so you only need to edit the file on the host from
+which you run those scripts.

http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/reader.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/reader.sh b/test/system/stress/reader.sh
index 0b796f0..1138685 100755
--- a/test/system/stress/reader.sh
+++ b/test/system/stress/reader.sh
@@ -15,24 +15,24 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Edit the credentials to match your system
-USERPASS='-u root -p secret'
-INSTANCE='-z localhost -i inst'
-
-# This is the seed for the range picking logic used by the scanner.
-SCAN_SEED='--scan-seed 1337'
-
-# Controls the number of random tablets the scanner will read sequentially
-#SCAN_ITERATIONS='--num-iterations 1024'
-
-# Alternatively, we can just continously scan
-CONTINUOUS_SCAN='--continuous'
-
-# Controls whether or not the scan will be using an isolated scanner. Add this to the execution 
-#SCAN_ISOLATION='--isolate'
-
-# Sets the batch size for the scanner, use a lower number for large rows / cells
-#SCAN_BATCH_SIZE='--scan-batch-size -1'
-
-../../../bin/accumulo org.apache.accumulo.test.stress.random.Scan $INSTANCE $USERPASS $SCAN_SEED $CONTINUOUS_SCAN \
-  $SCAN_BATCH_SIZE
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+   DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+   SOURCE="$(readlink "${SOURCE}")"
+   [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+LOG_DIR=${DIR}/logs
+mkdir -p $LOG_DIR
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+ts=$(date +%Y%m%d%H%M%S)
+host=$(hostname)
+${ACCUMULO_HOME}/bin/accumulo org.apache.accumulo.test.stress.random.Scan $INSTANCE $USERPASS $SCAN_SEED $CONTINUOUS_SCAN $SCAN_BATCH_SIZE \
+    > $LOG_DIR/${ts}_${host}_reader.out \
+    2> $LOG_DIR/${ts}_${host}_reader.err

http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/readers
----------------------------------------------------------------------
diff --git a/test/system/stress/readers b/test/system/stress/readers
new file mode 100644
index 0000000..80a4224
--- /dev/null
+++ b/test/system/stress/readers
@@ -0,0 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+readerhost1
+readerhost2

http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/start-readers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/start-readers.sh b/test/system/stress/start-readers.sh
new file mode 100755
index 0000000..709c46e
--- /dev/null
+++ b/test/system/stress/start-readers.sh
@@ -0,0 +1,40 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+   DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+   SOURCE="$(readlink "${SOURCE}")"
+   [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/readers ]]; then
+    echo readers file is missing
+    exit 1
+fi
+
+# Copy environment out
+pscp -h ${DIR}/readers ${DIR}/stress-env.sh ${DIR}
+
+pssh -h ${DIR}/readers "nohup ${DIR}/reader.sh >${DIR}/reader.out 2>${DIR}/reader.err < /dev/null &"

http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/start-writers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/start-writers.sh b/test/system/stress/start-writers.sh
new file mode 100755
index 0000000..b9464cc
--- /dev/null
+++ b/test/system/stress/start-writers.sh
@@ -0,0 +1,40 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+   DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+   SOURCE="$(readlink "${SOURCE}")"
+   [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/writers ]]; then
+    echo writers file is missing
+    exit 1
+fi
+
+# Copy environment out
+pscp -h ${DIR}/writers ${DIR}/stress-env.sh ${DIR}
+
+pssh -h ${DIR}/writers "nohup ${DIR}/writer.sh >${DIR}/writer.out 2>${DIR}/writer.err < /dev/null &"

http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/stop-readers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/stop-readers.sh b/test/system/stress/stop-readers.sh
new file mode 100755
index 0000000..1c59971
--- /dev/null
+++ b/test/system/stress/stop-readers.sh
@@ -0,0 +1,36 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+   DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+   SOURCE="$(readlink "${SOURCE}")"
+   [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/readers ]]; then
+    echo readers file is missing
+    exit 1
+fi
+pssh -h ${DIR}/readers "pkill -f '[o]rg.apache.accumulo.test.stress.random.Scan'" < /dev/null

http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/stop-writers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/stop-writers.sh b/test/system/stress/stop-writers.sh
new file mode 100755
index 0000000..d925cb3
--- /dev/null
+++ b/test/system/stress/stop-writers.sh
@@ -0,0 +1,36 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+   DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+   SOURCE="$(readlink "${SOURCE}")"
+   [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/writers ]]; then
+    echo writers file is missing
+    exit 1
+fi
+pssh -h ${DIR}/writers "pkill -f '[o]rg.apache.accumulo.test.stress.random.Write'" < /dev/null

http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/stress-env.sh.example
----------------------------------------------------------------------
diff --git a/test/system/stress/stress-env.sh.example b/test/system/stress/stress-env.sh.example
new file mode 100644
index 0000000..1360c67
--- /dev/null
+++ b/test/system/stress/stress-env.sh.example
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ACCUMULO_HOME=${ACCUMULO_HOME:-/opt/accumulo}
+
+# Edit the credentials to match your system
+USERPASS='-u root -p secret'
+INSTANCE='-z localhost -i inst'
+
+### Read configuration ###
+
+# This is the seed for the range picking logic used by the scanner.
+SCAN_SEED='--scan-seed 1337'
+
+# Controls the number of random tablets the scanner will read sequentially
+#SCAN_ITERATIONS='--num-iterations 1024'
+
+# Alternatively, we can just continously scan
+CONTINUOUS_SCAN='--continuous'
+
+# Controls whether or not the scan will be using an isolated scanner. Add this to the execution
+#SCAN_ISOLATION='--isolate'
+
+# Sets the batch size for the scanner, use a lower number for large rows / cells
+#SCAN_BATCH_SIZE='--scan-batch-size -1'
+
+### Write configuration ###
+
+# Edit these to change the range of each cell component. The size is in bytes.
+ROW_RANGE='--min-row-size 128 --max-row-size 128'
+CF_RANGE='--min-cf-size 128 --max-cf-size 128'
+CQ_RANGE='--min-cq-size 128 --max-cq-size 128'
+VALUE_RANGE='--min-value-size 1024 --max-value-size 2048'
+ROW_WIDTH='--min-row-width 1 --max-row-width 10'
+
+# These are the seeds for the random number generates used to generate each cell component.
+ROW_SEED='--row-seed 1'
+CF_SEED='--cf-seed 2'
+CQ_SEED='--cq-seed 3'
+VALUE_SEED='--value-seed 4'
+ROW_WIDTH_SEED='--row-width-seed 5'
+
+# This is the delay in milliseconds between writes. Use <= 0 for no delay.
+WRITE_DELAY='--write-delay 0'

http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/writer.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/writer.sh b/test/system/stress/writer.sh
index db55155..7d9b283 100755
--- a/test/system/stress/writer.sh
+++ b/test/system/stress/writer.sh
@@ -15,30 +15,28 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Edit the credentials to match your system
-USERPASS='-u root -p secret'
-INSTANCE='-z localhost -i inst'
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+   DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+   SOURCE="$(readlink "${SOURCE}")"
+   [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+LOG_DIR=${DIR}/logs
+mkdir -p $LOG_DIR
 
-# Edit these to change the range of each cell component. The size is in bytes.
-ROW_RANGE='--min-row-size 128 --max-row-size 128'
-CF_RANGE='--min-cf-size 128 --max-cf-size 128'
-CQ_RANGE='--min-cq-size 128 --max-cq-size 128'
-VALUE_RANGE='--min-value-size 1024 --max-value-size 2048'
-ROW_WIDTH='--min-row-width 1 --max-row-width 10'
+# Source environment
+. ${DIR}/stress-env.sh
 
-# These are the seeds for the random number generates used to generate each cell component.
-ROW_SEED='--row-seed 1'
-CF_SEED='--cf-seed 2'
-CQ_SEED='--cq-seed 3'
-VALUE_SEED='--value-seed 4'
-ROW_WIDTH_SEED='--row-width-seed 5'
+ts=$(date +%Y%m%d%H%M%S)
+host=$(hostname)
 
-# This is the delay in milliseconds between writes. Use <= 0 for no delay.
-WRITE_DELAY='--write-delay 0'
+# TBD - --clear-table option
 
-# Let's reset the table, for good measure
-../../../bin/accumulo shell $USERPASS -e 'deletetable -f stress_test'
-../../../bin/accumulo shell $USERPASS -e 'createtable stress_test'
-
-../../../bin/accumulo org.apache.accumulo.test.stress.random.Write $INSTANCE $USERPASS $ROW_RANGE $CF_RANGE $CQ_RANGE $VALUE_RANGE \
-  $ROW_SEED $CF_SEED $CQ_SEED $VALUE_SEED $ROW_WIDTH $ROW_WIDTH_SEED $WRITE_DELAY
+${ACCUMULO_HOME}/bin/accumulo org.apache.accumulo.test.stress.random.Write $INSTANCE $USERPASS $ROW_RANGE $CF_RANGE $CQ_RANGE $VALUE_RANGE \
+  $ROW_SEED $CF_SEED $CQ_SEED $VALUE_SEED $ROW_WIDTH $ROW_WIDTH_SEED $WRITE_DELAY \
+    > $LOG_DIR/${ts}_${host}_writer.out \
+    2> $LOG_DIR/${ts}_${host}_writer.err

http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/writers
----------------------------------------------------------------------
diff --git a/test/system/stress/writers b/test/system/stress/writers
new file mode 100644
index 0000000..0a06590
--- /dev/null
+++ b/test/system/stress/writers
@@ -0,0 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+writerhost1
+writerhost2