You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by bh...@apache.org on 2014/06/26 23:29:24 UTC
[1/6] git commit: ACCUMULO-2850 Add enhanced memory stress test
scripts
Repository: accumulo
Updated Branches:
refs/heads/1.5.2-SNAPSHOT 2422d4997 -> e01a72062
refs/heads/1.6.1-SNAPSHOT 934ca6fd4 -> 2a17a7a2b
refs/heads/master 6e89aae58 -> ab6ea922f
ACCUMULO-2850 Add enhanced memory stress test scripts
New scripts for the memory stress test make it easy to start and stop multiple
reader and writer processes, in a manner similar to continuous ingest. See the
updated README.md for information on how the scripts are used.
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/e01a7206
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/e01a7206
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/e01a7206
Branch: refs/heads/1.5.2-SNAPSHOT
Commit: e01a7206220a0ed143815416f28730fbdeb8c818
Parents: 2422d49
Author: Bill Havanki <bh...@cloudera.com>
Authored: Wed Jun 25 13:30:33 2014 -0400
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Thu Jun 26 17:26:40 2014 -0400
----------------------------------------------------------------------
test/system/stress/README.md | 89 +++++++++++++++++++++++----
test/system/stress/reader.sh | 42 ++++++-------
test/system/stress/readers | 17 +++++
test/system/stress/start-readers.sh | 40 ++++++++++++
test/system/stress/start-writers.sh | 40 ++++++++++++
test/system/stress/stop-readers.sh | 36 +++++++++++
test/system/stress/stop-writers.sh | 36 +++++++++++
test/system/stress/stress-env.sh.example | 56 +++++++++++++++++
test/system/stress/writer.sh | 44 +++++++------
test/system/stress/writers | 17 +++++
10 files changed, 362 insertions(+), 55 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/README.md
----------------------------------------------------------------------
diff --git a/test/system/stress/README.md b/test/system/stress/README.md
index 1a0bbae..e400d4b 100644
--- a/test/system/stress/README.md
+++ b/test/system/stress/README.md
@@ -4,10 +4,10 @@ contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
+the License. You may obtain a copy of the License at
+
http://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@@ -15,19 +15,86 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-
Accumulo Stress Test
---------------------
+====================
+
+This is an attempt to observe the behavior Accumulo displays when compacting and
+reading cells. There are two components to this package:
+`org.apache.accumulo.test.stress.random.Write` and
+`org.apache.accumulo.test.stress.random.Scan`.
+
+Since this test is meant to stress the system, there is no pre-defined test
+period and no success criteria that we can programmatically check. During a
+successful test period, the Accumulo cluster should remain stable and
+functional. Possible failure conditions include component failures, such as
+tablet servers running out of memory.
+
+The `org.apache.accumulo.test.stress.random.Write` utility provides facilities
+for writing random sized cells. Users can configure minimum and maximum sized
+portions of a cell. The portions users can configure are the row, column family,
+column qualifier and value. Note that the sizes are uniformly distributed
+between the minimum and maximum values.
+
+The `org.apache.accumulo.test.stress.random.Scan` utility provides users with
+the ability to query tables generated by the Write. It will pick a tablet at
+random and scan the entire range. The amount of times this process is performed
+is user configurable. By default, it happens 1,024 times. Users can also specify
+whether or not the scan should be isolated.
+
+There is no shared state intended by either of these services. This allows
+multiple clients to be run in parallel, either on the same host or distributed
+across hosts.
+
+## Prerequisites
+
+The utilities are run on a normal, initialized Accumulo cluster. By default,
+they only work with a table named "stress_test".
+
+The start and stop scripts rely on pssh. Before running them, you will need
+to install pssh on the machines that will be controlled by them.
+
+## Running
+
+Copy `stress-env.sh.example` to `stress-env.sh` and edit it, supplying at least
+correct values for the USERPASS and INSTANCE variables. The remainder of this
+file describes and provides defaults for many of the configuration options
+for the stress utilities.
+
+### Individual Execution
+
+Before starting a fresh write test, clear out the test table using the Accumulo
+shell.
+
+ > deletetable -f stress_test
+ > createtable stress_test
+
+To run a writer:
+
+ $ ./writer.sh
+
+The writer will begin logging into a "logs" subdirectory, using timestamped
+filenames. You can stop the writer by killing it or using Control-C.
+
+To run a reader:
+
+ $ ./reader.sh
-This is an attempt to observe the behavior Accumulo displays when compacting and reading these cells. There are two components to this package: `org.apache.accumulo.test.stress.random.Write` and `org.apache.accumulo.test.stress.random.Scan`.
+The reader logs like the writer. By default, the reader reads forever, but
+it can be configured for a fixed number of scan iterations.
-Since this test is meant to stress the system, there is no pre-defined test period and no success criteria that we can programmatically check. During a successful test period, the Accumulo cluster should remain stable and functional. Possible failure conditions include component failures, such as tablet servers running out of memory.
+### Group Execution
-The `org.apache.accumulo.test.stress.random.Write` utility provides facilities for writing random sized cells. Users can configure minimum and maximum sized portions of a cell. The portions users can configure are the row, column family, column qualifier and value. Note that the sizes are uniformly distributed between the minimum and maximum values
+To run writers across multiple machines, create a writers file and list each
+host that will run a writer, one per line.
-The `org.apache.accumulo.test.stress.random.Scan` utility provides users with the ability to query tables generated by the Write. It will pick a tablet at random and scan the entire range. The amount of times this process is performed is user configurable. By default, it happens 1,024 times. Users can also specify whether or not the scan should be isolated or not.
+Finally, to start and stop the writers:
-There is no shared state intended by either of these services. This allows multiple clients to be run in parallel, either on the same host or distributedacross hosts.
+ $ ./start-writers.sh
+ $ ./stop-writers.sh
-See the provided `test/system/stress/writer.sh` and `test/system/stress/reader.sh` scripts for example configuration options and how to launch the processes.
+For readers, make a similar readers file, and use `start-readers.sh` and
+`stop-readers.sh`.
+The start scripts copy out stress-env.sh to each of the writers and readers,
+overwriting what is there, so you only need to edit the file on the host from
+which you run those scripts.
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/reader.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/reader.sh b/test/system/stress/reader.sh
index 0b796f0..1138685 100755
--- a/test/system/stress/reader.sh
+++ b/test/system/stress/reader.sh
@@ -15,24 +15,24 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# Edit the credentials to match your system
-USERPASS='-u root -p secret'
-INSTANCE='-z localhost -i inst'
-
-# This is the seed for the range picking logic used by the scanner.
-SCAN_SEED='--scan-seed 1337'
-
-# Controls the number of random tablets the scanner will read sequentially
-#SCAN_ITERATIONS='--num-iterations 1024'
-
-# Alternatively, we can just continously scan
-CONTINUOUS_SCAN='--continuous'
-
-# Controls whether or not the scan will be using an isolated scanner. Add this to the execution
-#SCAN_ISOLATION='--isolate'
-
-# Sets the batch size for the scanner, use a lower number for large rows / cells
-#SCAN_BATCH_SIZE='--scan-batch-size -1'
-
-../../../bin/accumulo org.apache.accumulo.test.stress.random.Scan $INSTANCE $USERPASS $SCAN_SEED $CONTINUOUS_SCAN \
- $SCAN_BATCH_SIZE
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+LOG_DIR=${DIR}/logs
+mkdir -p $LOG_DIR
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+ts=$(date +%Y%m%d%H%M%S)
+host=$(hostname)
+${ACCUMULO_HOME}/bin/accumulo org.apache.accumulo.test.stress.random.Scan $INSTANCE $USERPASS $SCAN_SEED $CONTINUOUS_SCAN $SCAN_BATCH_SIZE \
+ > $LOG_DIR/${ts}_${host}_reader.out \
+ 2> $LOG_DIR/${ts}_${host}_reader.err
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/readers
----------------------------------------------------------------------
diff --git a/test/system/stress/readers b/test/system/stress/readers
new file mode 100644
index 0000000..80a4224
--- /dev/null
+++ b/test/system/stress/readers
@@ -0,0 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+readerhost1
+readerhost2
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/start-readers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/start-readers.sh b/test/system/stress/start-readers.sh
new file mode 100755
index 0000000..709c46e
--- /dev/null
+++ b/test/system/stress/start-readers.sh
@@ -0,0 +1,40 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/readers ]]; then
+ echo readers file is missing
+ exit 1
+fi
+
+# Copy environment out
+pscp -h ${DIR}/readers ${DIR}/stress-env.sh ${DIR}
+
+pssh -h ${DIR}/readers "nohup ${DIR}/reader.sh >${DIR}/reader.out 2>${DIR}/reader.err < /dev/null &"
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/start-writers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/start-writers.sh b/test/system/stress/start-writers.sh
new file mode 100755
index 0000000..b9464cc
--- /dev/null
+++ b/test/system/stress/start-writers.sh
@@ -0,0 +1,40 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/writers ]]; then
+ echo writers file is missing
+ exit 1
+fi
+
+# Copy environment out
+pscp -h ${DIR}/writers ${DIR}/stress-env.sh ${DIR}
+
+pssh -h ${DIR}/writers "nohup ${DIR}/writer.sh >${DIR}/writer.out 2>${DIR}/writer.err < /dev/null &"
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/stop-readers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/stop-readers.sh b/test/system/stress/stop-readers.sh
new file mode 100755
index 0000000..1c59971
--- /dev/null
+++ b/test/system/stress/stop-readers.sh
@@ -0,0 +1,36 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/readers ]]; then
+ echo readers file is missing
+ exit 1
+fi
+pssh -h ${DIR}/readers "pkill -f '[o]rg.apache.accumulo.test.stress.random.Scan'" < /dev/null
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/stop-writers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/stop-writers.sh b/test/system/stress/stop-writers.sh
new file mode 100755
index 0000000..d925cb3
--- /dev/null
+++ b/test/system/stress/stop-writers.sh
@@ -0,0 +1,36 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/writers ]]; then
+ echo writers file is missing
+ exit 1
+fi
+pssh -h ${DIR}/writers "pkill -f '[o]rg.apache.accumulo.test.stress.random.Write'" < /dev/null
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/stress-env.sh.example
----------------------------------------------------------------------
diff --git a/test/system/stress/stress-env.sh.example b/test/system/stress/stress-env.sh.example
new file mode 100644
index 0000000..1360c67
--- /dev/null
+++ b/test/system/stress/stress-env.sh.example
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ACCUMULO_HOME=${ACCUMULO_HOME:-/opt/accumulo}
+
+# Edit the credentials to match your system
+USERPASS='-u root -p secret'
+INSTANCE='-z localhost -i inst'
+
+### Read configuration ###
+
+# This is the seed for the range picking logic used by the scanner.
+SCAN_SEED='--scan-seed 1337'
+
+# Controls the number of random tablets the scanner will read sequentially
+#SCAN_ITERATIONS='--num-iterations 1024'
+
+# Alternatively, we can just continously scan
+CONTINUOUS_SCAN='--continuous'
+
+# Controls whether or not the scan will be using an isolated scanner. Add this to the execution
+#SCAN_ISOLATION='--isolate'
+
+# Sets the batch size for the scanner, use a lower number for large rows / cells
+#SCAN_BATCH_SIZE='--scan-batch-size -1'
+
+### Write configuration ###
+
+# Edit these to change the range of each cell component. The size is in bytes.
+ROW_RANGE='--min-row-size 128 --max-row-size 128'
+CF_RANGE='--min-cf-size 128 --max-cf-size 128'
+CQ_RANGE='--min-cq-size 128 --max-cq-size 128'
+VALUE_RANGE='--min-value-size 1024 --max-value-size 2048'
+ROW_WIDTH='--min-row-width 1 --max-row-width 10'
+
+# These are the seeds for the random number generates used to generate each cell component.
+ROW_SEED='--row-seed 1'
+CF_SEED='--cf-seed 2'
+CQ_SEED='--cq-seed 3'
+VALUE_SEED='--value-seed 4'
+ROW_WIDTH_SEED='--row-width-seed 5'
+
+# This is the delay in milliseconds between writes. Use <= 0 for no delay.
+WRITE_DELAY='--write-delay 0'
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/writer.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/writer.sh b/test/system/stress/writer.sh
index db55155..7d9b283 100755
--- a/test/system/stress/writer.sh
+++ b/test/system/stress/writer.sh
@@ -15,30 +15,28 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# Edit the credentials to match your system
-USERPASS='-u root -p secret'
-INSTANCE='-z localhost -i inst'
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+LOG_DIR=${DIR}/logs
+mkdir -p $LOG_DIR
-# Edit these to change the range of each cell component. The size is in bytes.
-ROW_RANGE='--min-row-size 128 --max-row-size 128'
-CF_RANGE='--min-cf-size 128 --max-cf-size 128'
-CQ_RANGE='--min-cq-size 128 --max-cq-size 128'
-VALUE_RANGE='--min-value-size 1024 --max-value-size 2048'
-ROW_WIDTH='--min-row-width 1 --max-row-width 10'
+# Source environment
+. ${DIR}/stress-env.sh
-# These are the seeds for the random number generates used to generate each cell component.
-ROW_SEED='--row-seed 1'
-CF_SEED='--cf-seed 2'
-CQ_SEED='--cq-seed 3'
-VALUE_SEED='--value-seed 4'
-ROW_WIDTH_SEED='--row-width-seed 5'
+ts=$(date +%Y%m%d%H%M%S)
+host=$(hostname)
-# This is the delay in milliseconds between writes. Use <= 0 for no delay.
-WRITE_DELAY='--write-delay 0'
+# TBD - --clear-table option
-# Let's reset the table, for good measure
-../../../bin/accumulo shell $USERPASS -e 'deletetable -f stress_test'
-../../../bin/accumulo shell $USERPASS -e 'createtable stress_test'
-
-../../../bin/accumulo org.apache.accumulo.test.stress.random.Write $INSTANCE $USERPASS $ROW_RANGE $CF_RANGE $CQ_RANGE $VALUE_RANGE \
- $ROW_SEED $CF_SEED $CQ_SEED $VALUE_SEED $ROW_WIDTH $ROW_WIDTH_SEED $WRITE_DELAY
+${ACCUMULO_HOME}/bin/accumulo org.apache.accumulo.test.stress.random.Write $INSTANCE $USERPASS $ROW_RANGE $CF_RANGE $CQ_RANGE $VALUE_RANGE \
+ $ROW_SEED $CF_SEED $CQ_SEED $VALUE_SEED $ROW_WIDTH $ROW_WIDTH_SEED $WRITE_DELAY \
+ > $LOG_DIR/${ts}_${host}_writer.out \
+ 2> $LOG_DIR/${ts}_${host}_writer.err
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/writers
----------------------------------------------------------------------
diff --git a/test/system/stress/writers b/test/system/stress/writers
new file mode 100644
index 0000000..0a06590
--- /dev/null
+++ b/test/system/stress/writers
@@ -0,0 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+writerhost1
+writerhost2
[3/6] git commit: ACCUMULO-2850 Add enhanced memory stress test
scripts
Posted by bh...@apache.org.
ACCUMULO-2850 Add enhanced memory stress test scripts
New scripts for the memory stress test make it easy to start and stop multiple
reader and writer processes, in a manner similar to continuous ingest. See the
updated README.md for information on how the scripts are used.
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/e01a7206
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/e01a7206
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/e01a7206
Branch: refs/heads/master
Commit: e01a7206220a0ed143815416f28730fbdeb8c818
Parents: 2422d49
Author: Bill Havanki <bh...@cloudera.com>
Authored: Wed Jun 25 13:30:33 2014 -0400
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Thu Jun 26 17:26:40 2014 -0400
----------------------------------------------------------------------
test/system/stress/README.md | 89 +++++++++++++++++++++++----
test/system/stress/reader.sh | 42 ++++++-------
test/system/stress/readers | 17 +++++
test/system/stress/start-readers.sh | 40 ++++++++++++
test/system/stress/start-writers.sh | 40 ++++++++++++
test/system/stress/stop-readers.sh | 36 +++++++++++
test/system/stress/stop-writers.sh | 36 +++++++++++
test/system/stress/stress-env.sh.example | 56 +++++++++++++++++
test/system/stress/writer.sh | 44 +++++++------
test/system/stress/writers | 17 +++++
10 files changed, 362 insertions(+), 55 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/README.md
----------------------------------------------------------------------
diff --git a/test/system/stress/README.md b/test/system/stress/README.md
index 1a0bbae..e400d4b 100644
--- a/test/system/stress/README.md
+++ b/test/system/stress/README.md
@@ -4,10 +4,10 @@ contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
+the License. You may obtain a copy of the License at
+
http://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@@ -15,19 +15,86 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-
Accumulo Stress Test
---------------------
+====================
+
+This is an attempt to observe the behavior Accumulo displays when compacting and
+reading cells. There are two components to this package:
+`org.apache.accumulo.test.stress.random.Write` and
+`org.apache.accumulo.test.stress.random.Scan`.
+
+Since this test is meant to stress the system, there is no pre-defined test
+period and no success criteria that we can programmatically check. During a
+successful test period, the Accumulo cluster should remain stable and
+functional. Possible failure conditions include component failures, such as
+tablet servers running out of memory.
+
+The `org.apache.accumulo.test.stress.random.Write` utility provides facilities
+for writing random sized cells. Users can configure minimum and maximum sized
+portions of a cell. The portions users can configure are the row, column family,
+column qualifier and value. Note that the sizes are uniformly distributed
+between the minimum and maximum values.
+
+The `org.apache.accumulo.test.stress.random.Scan` utility provides users with
+the ability to query tables generated by the Write. It will pick a tablet at
+random and scan the entire range. The amount of times this process is performed
+is user configurable. By default, it happens 1,024 times. Users can also specify
+whether or not the scan should be isolated.
+
+There is no shared state intended by either of these services. This allows
+multiple clients to be run in parallel, either on the same host or distributed
+across hosts.
+
+## Prerequisites
+
+The utilities are run on a normal, initialized Accumulo cluster. By default,
+they only work with a table named "stress_test".
+
+The start and stop scripts rely on pssh. Before running them, you will need
+to install pssh on the machines that will be controlled by them.
+
+## Running
+
+Copy `stress-env.sh.example` to `stress-env.sh` and edit it, supplying at least
+correct values for the USERPASS and INSTANCE variables. The remainder of this
+file describes and provides defaults for many of the configuration options
+for the stress utilities.
+
+### Individual Execution
+
+Before starting a fresh write test, clear out the test table using the Accumulo
+shell.
+
+ > deletetable -f stress_test
+ > createtable stress_test
+
+To run a writer:
+
+ $ ./writer.sh
+
+The writer will begin logging into a "logs" subdirectory, using timestamped
+filenames. You can stop the writer by killing it or using Control-C.
+
+To run a reader:
+
+ $ ./reader.sh
-This is an attempt to observe the behavior Accumulo displays when compacting and reading these cells. There are two components to this package: `org.apache.accumulo.test.stress.random.Write` and `org.apache.accumulo.test.stress.random.Scan`.
+The reader logs like the writer. By default, the reader reads forever, but
+it can be configured for a fixed number of scan iterations.
-Since this test is meant to stress the system, there is no pre-defined test period and no success criteria that we can programmatically check. During a successful test period, the Accumulo cluster should remain stable and functional. Possible failure conditions include component failures, such as tablet servers running out of memory.
+### Group Execution
-The `org.apache.accumulo.test.stress.random.Write` utility provides facilities for writing random sized cells. Users can configure minimum and maximum sized portions of a cell. The portions users can configure are the row, column family, column qualifier and value. Note that the sizes are uniformly distributed between the minimum and maximum values
+To run writers across multiple machines, create a writers file and list each
+host that will run a writer, one per line.
-The `org.apache.accumulo.test.stress.random.Scan` utility provides users with the ability to query tables generated by the Write. It will pick a tablet at random and scan the entire range. The amount of times this process is performed is user configurable. By default, it happens 1,024 times. Users can also specify whether or not the scan should be isolated or not.
+Finally, to start and stop the writers:
-There is no shared state intended by either of these services. This allows multiple clients to be run in parallel, either on the same host or distributedacross hosts.
+ $ ./start-writers.sh
+ $ ./stop-writers.sh
-See the provided `test/system/stress/writer.sh` and `test/system/stress/reader.sh` scripts for example configuration options and how to launch the processes.
+For readers, make a similar readers file, and use `start-readers.sh` and
+`stop-readers.sh`.
+The start scripts copy out stress-env.sh to each of the writers and readers,
+overwriting what is there, so you only need to edit the file on the host from
+which you run those scripts.
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/reader.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/reader.sh b/test/system/stress/reader.sh
index 0b796f0..1138685 100755
--- a/test/system/stress/reader.sh
+++ b/test/system/stress/reader.sh
@@ -15,24 +15,24 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# Edit the credentials to match your system
-USERPASS='-u root -p secret'
-INSTANCE='-z localhost -i inst'
-
-# This is the seed for the range picking logic used by the scanner.
-SCAN_SEED='--scan-seed 1337'
-
-# Controls the number of random tablets the scanner will read sequentially
-#SCAN_ITERATIONS='--num-iterations 1024'
-
-# Alternatively, we can just continously scan
-CONTINUOUS_SCAN='--continuous'
-
-# Controls whether or not the scan will be using an isolated scanner. Add this to the execution
-#SCAN_ISOLATION='--isolate'
-
-# Sets the batch size for the scanner, use a lower number for large rows / cells
-#SCAN_BATCH_SIZE='--scan-batch-size -1'
-
-../../../bin/accumulo org.apache.accumulo.test.stress.random.Scan $INSTANCE $USERPASS $SCAN_SEED $CONTINUOUS_SCAN \
- $SCAN_BATCH_SIZE
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+LOG_DIR=${DIR}/logs
+mkdir -p $LOG_DIR
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+ts=$(date +%Y%m%d%H%M%S)
+host=$(hostname)
+${ACCUMULO_HOME}/bin/accumulo org.apache.accumulo.test.stress.random.Scan $INSTANCE $USERPASS $SCAN_SEED $CONTINUOUS_SCAN $SCAN_BATCH_SIZE \
+ > $LOG_DIR/${ts}_${host}_reader.out \
+ 2> $LOG_DIR/${ts}_${host}_reader.err
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/readers
----------------------------------------------------------------------
diff --git a/test/system/stress/readers b/test/system/stress/readers
new file mode 100644
index 0000000..80a4224
--- /dev/null
+++ b/test/system/stress/readers
@@ -0,0 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+readerhost1
+readerhost2
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/start-readers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/start-readers.sh b/test/system/stress/start-readers.sh
new file mode 100755
index 0000000..709c46e
--- /dev/null
+++ b/test/system/stress/start-readers.sh
@@ -0,0 +1,40 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/readers ]]; then
+ echo readers file is missing
+ exit 1
+fi
+
+# Copy environment out
+pscp -h ${DIR}/readers ${DIR}/stress-env.sh ${DIR}
+
+pssh -h ${DIR}/readers "nohup ${DIR}/reader.sh >${DIR}/reader.out 2>${DIR}/reader.err < /dev/null &"
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/start-writers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/start-writers.sh b/test/system/stress/start-writers.sh
new file mode 100755
index 0000000..b9464cc
--- /dev/null
+++ b/test/system/stress/start-writers.sh
@@ -0,0 +1,40 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/writers ]]; then
+ echo writers file is missing
+ exit 1
+fi
+
+# Copy environment out
+pscp -h ${DIR}/writers ${DIR}/stress-env.sh ${DIR}
+
+pssh -h ${DIR}/writers "nohup ${DIR}/writer.sh >${DIR}/writer.out 2>${DIR}/writer.err < /dev/null &"
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/stop-readers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/stop-readers.sh b/test/system/stress/stop-readers.sh
new file mode 100755
index 0000000..1c59971
--- /dev/null
+++ b/test/system/stress/stop-readers.sh
@@ -0,0 +1,36 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/readers ]]; then
+ echo readers file is missing
+ exit 1
+fi
+pssh -h ${DIR}/readers "pkill -f '[o]rg.apache.accumulo.test.stress.random.Scan'" < /dev/null
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/stop-writers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/stop-writers.sh b/test/system/stress/stop-writers.sh
new file mode 100755
index 0000000..d925cb3
--- /dev/null
+++ b/test/system/stress/stop-writers.sh
@@ -0,0 +1,36 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/writers ]]; then
+ echo writers file is missing
+ exit 1
+fi
+pssh -h ${DIR}/writers "pkill -f '[o]rg.apache.accumulo.test.stress.random.Write'" < /dev/null
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/stress-env.sh.example
----------------------------------------------------------------------
diff --git a/test/system/stress/stress-env.sh.example b/test/system/stress/stress-env.sh.example
new file mode 100644
index 0000000..1360c67
--- /dev/null
+++ b/test/system/stress/stress-env.sh.example
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ACCUMULO_HOME=${ACCUMULO_HOME:-/opt/accumulo}
+
+# Edit the credentials to match your system
+USERPASS='-u root -p secret'
+INSTANCE='-z localhost -i inst'
+
+### Read configuration ###
+
+# This is the seed for the range picking logic used by the scanner.
+SCAN_SEED='--scan-seed 1337'
+
+# Controls the number of random tablets the scanner will read sequentially
+#SCAN_ITERATIONS='--num-iterations 1024'
+
+# Alternatively, we can just continously scan
+CONTINUOUS_SCAN='--continuous'
+
+# Controls whether or not the scan will be using an isolated scanner. Add this to the execution
+#SCAN_ISOLATION='--isolate'
+
+# Sets the batch size for the scanner, use a lower number for large rows / cells
+#SCAN_BATCH_SIZE='--scan-batch-size -1'
+
+### Write configuration ###
+
+# Edit these to change the range of each cell component. The size is in bytes.
+ROW_RANGE='--min-row-size 128 --max-row-size 128'
+CF_RANGE='--min-cf-size 128 --max-cf-size 128'
+CQ_RANGE='--min-cq-size 128 --max-cq-size 128'
+VALUE_RANGE='--min-value-size 1024 --max-value-size 2048'
+ROW_WIDTH='--min-row-width 1 --max-row-width 10'
+
+# These are the seeds for the random number generates used to generate each cell component.
+ROW_SEED='--row-seed 1'
+CF_SEED='--cf-seed 2'
+CQ_SEED='--cq-seed 3'
+VALUE_SEED='--value-seed 4'
+ROW_WIDTH_SEED='--row-width-seed 5'
+
+# This is the delay in milliseconds between writes. Use <= 0 for no delay.
+WRITE_DELAY='--write-delay 0'
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/writer.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/writer.sh b/test/system/stress/writer.sh
index db55155..7d9b283 100755
--- a/test/system/stress/writer.sh
+++ b/test/system/stress/writer.sh
@@ -15,30 +15,28 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# Edit the credentials to match your system
-USERPASS='-u root -p secret'
-INSTANCE='-z localhost -i inst'
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+LOG_DIR=${DIR}/logs
+mkdir -p $LOG_DIR
-# Edit these to change the range of each cell component. The size is in bytes.
-ROW_RANGE='--min-row-size 128 --max-row-size 128'
-CF_RANGE='--min-cf-size 128 --max-cf-size 128'
-CQ_RANGE='--min-cq-size 128 --max-cq-size 128'
-VALUE_RANGE='--min-value-size 1024 --max-value-size 2048'
-ROW_WIDTH='--min-row-width 1 --max-row-width 10'
+# Source environment
+. ${DIR}/stress-env.sh
-# These are the seeds for the random number generates used to generate each cell component.
-ROW_SEED='--row-seed 1'
-CF_SEED='--cf-seed 2'
-CQ_SEED='--cq-seed 3'
-VALUE_SEED='--value-seed 4'
-ROW_WIDTH_SEED='--row-width-seed 5'
+ts=$(date +%Y%m%d%H%M%S)
+host=$(hostname)
-# This is the delay in milliseconds between writes. Use <= 0 for no delay.
-WRITE_DELAY='--write-delay 0'
+# TBD - --clear-table option
-# Let's reset the table, for good measure
-../../../bin/accumulo shell $USERPASS -e 'deletetable -f stress_test'
-../../../bin/accumulo shell $USERPASS -e 'createtable stress_test'
-
-../../../bin/accumulo org.apache.accumulo.test.stress.random.Write $INSTANCE $USERPASS $ROW_RANGE $CF_RANGE $CQ_RANGE $VALUE_RANGE \
- $ROW_SEED $CF_SEED $CQ_SEED $VALUE_SEED $ROW_WIDTH $ROW_WIDTH_SEED $WRITE_DELAY
+${ACCUMULO_HOME}/bin/accumulo org.apache.accumulo.test.stress.random.Write $INSTANCE $USERPASS $ROW_RANGE $CF_RANGE $CQ_RANGE $VALUE_RANGE \
+ $ROW_SEED $CF_SEED $CQ_SEED $VALUE_SEED $ROW_WIDTH $ROW_WIDTH_SEED $WRITE_DELAY \
+ > $LOG_DIR/${ts}_${host}_writer.out \
+ 2> $LOG_DIR/${ts}_${host}_writer.err
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/writers
----------------------------------------------------------------------
diff --git a/test/system/stress/writers b/test/system/stress/writers
new file mode 100644
index 0000000..0a06590
--- /dev/null
+++ b/test/system/stress/writers
@@ -0,0 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+writerhost1
+writerhost2
[5/6] git commit: Merge branch '1.5.2-SNAPSHOT' into 1.6.1-SNAPSHOT
Posted by bh...@apache.org.
Merge branch '1.5.2-SNAPSHOT' into 1.6.1-SNAPSHOT
Conflicts:
test/system/stress/README.md
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/2a17a7a2
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/2a17a7a2
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/2a17a7a2
Branch: refs/heads/1.6.1-SNAPSHOT
Commit: 2a17a7a2b4eebdfb0a89f51b97adaab52904bc95
Parents: 934ca6f e01a720
Author: Bill Havanki <bh...@cloudera.com>
Authored: Thu Jun 26 17:29:00 2014 -0400
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Thu Jun 26 17:29:00 2014 -0400
----------------------------------------------------------------------
test/system/stress/README.md | 95 +++++++++++++++++++++++----
test/system/stress/reader.sh | 42 ++++++------
test/system/stress/readers | 17 +++++
test/system/stress/start-readers.sh | 40 +++++++++++
test/system/stress/start-writers.sh | 40 +++++++++++
test/system/stress/stop-readers.sh | 36 ++++++++++
test/system/stress/stop-writers.sh | 36 ++++++++++
test/system/stress/stress-env.sh.example | 56 ++++++++++++++++
test/system/stress/writer.sh | 44 ++++++-------
test/system/stress/writers | 17 +++++
10 files changed, 367 insertions(+), 56 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/2a17a7a2/test/system/stress/README.md
----------------------------------------------------------------------
diff --cc test/system/stress/README.md
index 9546b21,e400d4b..9eb9fdc
--- a/test/system/stress/README.md
+++ b/test/system/stress/README.md
@@@ -15,20 -15,86 +15,91 @@@ See the License for the specific langua
limitations under the License.
-->
-
Accumulo Stress Test
- --------------------
+ ====================
+
+ This is an attempt to observe the behavior Accumulo displays when compacting and
+ reading cells. There are two components to this package:
+ `org.apache.accumulo.test.stress.random.Write` and
+ `org.apache.accumulo.test.stress.random.Scan`.
+
+ Since this test is meant to stress the system, there is no pre-defined test
+ period and no success criteria that we can programmatically check. During a
+ successful test period, the Accumulo cluster should remain stable and
+ functional. Possible failure conditions include component failures, such as
+ tablet servers running out of memory.
+
+ The `org.apache.accumulo.test.stress.random.Write` utility provides facilities
+ for writing random sized cells. Users can configure minimum and maximum sized
+ portions of a cell. The portions users can configure are the row, column family,
+ column qualifier and value. Note that the sizes are uniformly distributed
+ between the minimum and maximum values.
+
+ The `org.apache.accumulo.test.stress.random.Scan` utility provides users with
+ the ability to query tables generated by the Write. It will pick a tablet at
+ random and scan the entire range. The amount of times this process is performed
+ is user configurable. By default, it happens 1,024 times. Users can also specify
+ whether or not the scan should be isolated.
+
+ There is no shared state intended by either of these services. This allows
+ multiple clients to be run in parallel, either on the same host or distributed
+ across hosts.
+
+ ## Prerequisites
+
+ The utilities are run on a normal, initialized Accumulo cluster. By default,
+ they only work with a table named "stress_test".
+
+ The start and stop scripts rely on pssh. Before running them, you will need
+ to install pssh on the machines that will be controlled by them.
+
+ ## Running
+
+ Copy `stress-env.sh.example` to `stress-env.sh` and edit it, supplying at least
+ correct values for the USERPASS and INSTANCE variables. The remainder of this
+ file describes and provides defaults for many of the configuration options
+ for the stress utilities.
+
+ ### Individual Execution
+
+ Before starting a fresh write test, clear out the test table using the Accumulo
+ shell.
+
+ > deletetable -f stress_test
+ > createtable stress_test
+
+ To run a writer:
+
+ $ ./writer.sh
+
+ The writer will begin logging into a "logs" subdirectory, using timestamped
+ filenames. You can stop the writer by killing it or using Control-C.
+
+ To run a reader:
+
+ $ ./reader.sh
+
+ The reader logs like the writer. By default, the reader reads forever, but
+ it can be configured for a fixed number of scan iterations.
- This is an attempt to observe the behavior Accumulo displays when compacting and reading these cells. There are two components to this package: `org.apache.accumulo.test.stress.random.Write` and `org.apache.accumulo.test.stress.random.Scan`.
+ ### Group Execution
- Since this test is meant to stress the system, there is no pre-defined test period and no success criteria that we can programmatically check. During a successful test period, the Accumulo cluster should remain stable and functional. Possible failure conditions include component failures, such as tablet servers running out of memory.
+ To run writers across multiple machines, create a writers file and list each
+ host that will run a writer, one per line.
- The `org.apache.accumulo.test.stress.random.Write` utility provides facilities for writing random sized cells. Users can configure minimum and maximum sized portions of a cell. The portions users can configure are the row, column family, column qualifier and value. Note that the sizes are uniformly distributed between the minimum and maximum values
+ Finally, to start and stop the writers:
- The `org.apache.accumulo.test.stress.random.Scan` utility provides users with the ability to query tables generated by the Write. It will pick a tablet at random and scan the entire range. The amount of times this process is performed is user configurable. By default, it happens 1,024 times. Users can also specify whether or not the scan should be isolated or not.
+ $ ./start-writers.sh
+ $ ./stop-writers.sh
- There is no shared state intended by either of these services. This allows multiple clients to be run in parallel, either on the same host or distributedacross hosts.
+ For readers, make a similar readers file, and use `start-readers.sh` and
+ `stop-readers.sh`.
- See the provided `test/system/stress/writer.sh` and `test/system/stress/reader.sh` scripts for example configuration options and how to launch the processes.
+ The start scripts copy out stress-env.sh to each of the writers and readers,
+ overwriting what is there, so you only need to edit the file on the host from
+ which you run those scripts.
+
- Note that in versions of Accumulo greater than 1.6.0, users should verify that the cell sizes they supply meet the criteria set forth by the key constraints on their tables. By default, tables are configured to reject keys that are are greater than 1MB.
++Note that in versions of Accumulo greater than 1.6.0, users should verify that
++the cell sizes they supply meet the criteria set forth by the key constraints
++on their tables. By default, tables are configured to reject keys that are
++greater than 1MB.
[6/6] git commit: Merge branch '1.6.1-SNAPSHOT'
Posted by bh...@apache.org.
Merge branch '1.6.1-SNAPSHOT'
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/ab6ea922
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/ab6ea922
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/ab6ea922
Branch: refs/heads/master
Commit: ab6ea922fdcf0a99d04728825d943ecd1eee41ae
Parents: 6e89aae 2a17a7a
Author: Bill Havanki <bh...@cloudera.com>
Authored: Thu Jun 26 17:29:11 2014 -0400
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Thu Jun 26 17:29:11 2014 -0400
----------------------------------------------------------------------
test/system/stress/README.md | 95 +++++++++++++++++++++++----
test/system/stress/reader.sh | 42 ++++++------
test/system/stress/readers | 17 +++++
test/system/stress/start-readers.sh | 40 +++++++++++
test/system/stress/start-writers.sh | 40 +++++++++++
test/system/stress/stop-readers.sh | 36 ++++++++++
test/system/stress/stop-writers.sh | 36 ++++++++++
test/system/stress/stress-env.sh.example | 56 ++++++++++++++++
test/system/stress/writer.sh | 44 ++++++-------
test/system/stress/writers | 17 +++++
10 files changed, 367 insertions(+), 56 deletions(-)
----------------------------------------------------------------------
[4/6] git commit: Merge branch '1.5.2-SNAPSHOT' into 1.6.1-SNAPSHOT
Posted by bh...@apache.org.
Merge branch '1.5.2-SNAPSHOT' into 1.6.1-SNAPSHOT
Conflicts:
test/system/stress/README.md
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/2a17a7a2
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/2a17a7a2
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/2a17a7a2
Branch: refs/heads/master
Commit: 2a17a7a2b4eebdfb0a89f51b97adaab52904bc95
Parents: 934ca6f e01a720
Author: Bill Havanki <bh...@cloudera.com>
Authored: Thu Jun 26 17:29:00 2014 -0400
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Thu Jun 26 17:29:00 2014 -0400
----------------------------------------------------------------------
test/system/stress/README.md | 95 +++++++++++++++++++++++----
test/system/stress/reader.sh | 42 ++++++------
test/system/stress/readers | 17 +++++
test/system/stress/start-readers.sh | 40 +++++++++++
test/system/stress/start-writers.sh | 40 +++++++++++
test/system/stress/stop-readers.sh | 36 ++++++++++
test/system/stress/stop-writers.sh | 36 ++++++++++
test/system/stress/stress-env.sh.example | 56 ++++++++++++++++
test/system/stress/writer.sh | 44 ++++++-------
test/system/stress/writers | 17 +++++
10 files changed, 367 insertions(+), 56 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/2a17a7a2/test/system/stress/README.md
----------------------------------------------------------------------
diff --cc test/system/stress/README.md
index 9546b21,e400d4b..9eb9fdc
--- a/test/system/stress/README.md
+++ b/test/system/stress/README.md
@@@ -15,20 -15,86 +15,91 @@@ See the License for the specific langua
limitations under the License.
-->
-
Accumulo Stress Test
- --------------------
+ ====================
+
+ This is an attempt to observe the behavior Accumulo displays when compacting and
+ reading cells. There are two components to this package:
+ `org.apache.accumulo.test.stress.random.Write` and
+ `org.apache.accumulo.test.stress.random.Scan`.
+
+ Since this test is meant to stress the system, there is no pre-defined test
+ period and no success criteria that we can programmatically check. During a
+ successful test period, the Accumulo cluster should remain stable and
+ functional. Possible failure conditions include component failures, such as
+ tablet servers running out of memory.
+
+ The `org.apache.accumulo.test.stress.random.Write` utility provides facilities
+ for writing random sized cells. Users can configure minimum and maximum sized
+ portions of a cell. The portions users can configure are the row, column family,
+ column qualifier and value. Note that the sizes are uniformly distributed
+ between the minimum and maximum values.
+
+ The `org.apache.accumulo.test.stress.random.Scan` utility provides users with
+ the ability to query tables generated by the Write. It will pick a tablet at
+ random and scan the entire range. The amount of times this process is performed
+ is user configurable. By default, it happens 1,024 times. Users can also specify
+ whether or not the scan should be isolated.
+
+ There is no shared state intended by either of these services. This allows
+ multiple clients to be run in parallel, either on the same host or distributed
+ across hosts.
+
+ ## Prerequisites
+
+ The utilities are run on a normal, initialized Accumulo cluster. By default,
+ they only work with a table named "stress_test".
+
+ The start and stop scripts rely on pssh. Before running them, you will need
+ to install pssh on the machines that will be controlled by them.
+
+ ## Running
+
+ Copy `stress-env.sh.example` to `stress-env.sh` and edit it, supplying at least
+ correct values for the USERPASS and INSTANCE variables. The remainder of this
+ file describes and provides defaults for many of the configuration options
+ for the stress utilities.
+
+ ### Individual Execution
+
+ Before starting a fresh write test, clear out the test table using the Accumulo
+ shell.
+
+ > deletetable -f stress_test
+ > createtable stress_test
+
+ To run a writer:
+
+ $ ./writer.sh
+
+ The writer will begin logging into a "logs" subdirectory, using timestamped
+ filenames. You can stop the writer by killing it or using Control-C.
+
+ To run a reader:
+
+ $ ./reader.sh
+
+ The reader logs like the writer. By default, the reader reads forever, but
+ it can be configured for a fixed number of scan iterations.
- This is an attempt to observe the behavior Accumulo displays when compacting and reading these cells. There are two components to this package: `org.apache.accumulo.test.stress.random.Write` and `org.apache.accumulo.test.stress.random.Scan`.
+ ### Group Execution
- Since this test is meant to stress the system, there is no pre-defined test period and no success criteria that we can programmatically check. During a successful test period, the Accumulo cluster should remain stable and functional. Possible failure conditions include component failures, such as tablet servers running out of memory.
+ To run writers across multiple machines, create a writers file and list each
+ host that will run a writer, one per line.
- The `org.apache.accumulo.test.stress.random.Write` utility provides facilities for writing random sized cells. Users can configure minimum and maximum sized portions of a cell. The portions users can configure are the row, column family, column qualifier and value. Note that the sizes are uniformly distributed between the minimum and maximum values
+ Finally, to start and stop the writers:
- The `org.apache.accumulo.test.stress.random.Scan` utility provides users with the ability to query tables generated by the Write. It will pick a tablet at random and scan the entire range. The amount of times this process is performed is user configurable. By default, it happens 1,024 times. Users can also specify whether or not the scan should be isolated or not.
+ $ ./start-writers.sh
+ $ ./stop-writers.sh
- There is no shared state intended by either of these services. This allows multiple clients to be run in parallel, either on the same host or distributedacross hosts.
+ For readers, make a similar readers file, and use `start-readers.sh` and
+ `stop-readers.sh`.
- See the provided `test/system/stress/writer.sh` and `test/system/stress/reader.sh` scripts for example configuration options and how to launch the processes.
+ The start scripts copy out stress-env.sh to each of the writers and readers,
+ overwriting what is there, so you only need to edit the file on the host from
+ which you run those scripts.
+
- Note that in versions of Accumulo greater than 1.6.0, users should verify that the cell sizes they supply meet the criteria set forth by the key constraints on their tables. By default, tables are configured to reject keys that are are greater than 1MB.
++Note that in versions of Accumulo greater than 1.6.0, users should verify that
++the cell sizes they supply meet the criteria set forth by the key constraints
++on their tables. By default, tables are configured to reject keys that are
++greater than 1MB.
[2/6] git commit: ACCUMULO-2850 Add enhanced memory stress test
scripts
Posted by bh...@apache.org.
ACCUMULO-2850 Add enhanced memory stress test scripts
New scripts for the memory stress test make it easy to start and stop multiple
reader and writer processes, in a manner similar to continuous ingest. See the
updated README.md for information on how the scripts are used.
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/e01a7206
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/e01a7206
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/e01a7206
Branch: refs/heads/1.6.1-SNAPSHOT
Commit: e01a7206220a0ed143815416f28730fbdeb8c818
Parents: 2422d49
Author: Bill Havanki <bh...@cloudera.com>
Authored: Wed Jun 25 13:30:33 2014 -0400
Committer: Bill Havanki <bh...@cloudera.com>
Committed: Thu Jun 26 17:26:40 2014 -0400
----------------------------------------------------------------------
test/system/stress/README.md | 89 +++++++++++++++++++++++----
test/system/stress/reader.sh | 42 ++++++-------
test/system/stress/readers | 17 +++++
test/system/stress/start-readers.sh | 40 ++++++++++++
test/system/stress/start-writers.sh | 40 ++++++++++++
test/system/stress/stop-readers.sh | 36 +++++++++++
test/system/stress/stop-writers.sh | 36 +++++++++++
test/system/stress/stress-env.sh.example | 56 +++++++++++++++++
test/system/stress/writer.sh | 44 +++++++------
test/system/stress/writers | 17 +++++
10 files changed, 362 insertions(+), 55 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/README.md
----------------------------------------------------------------------
diff --git a/test/system/stress/README.md b/test/system/stress/README.md
index 1a0bbae..e400d4b 100644
--- a/test/system/stress/README.md
+++ b/test/system/stress/README.md
@@ -4,10 +4,10 @@ contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
+the License. You may obtain a copy of the License at
+
http://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@@ -15,19 +15,86 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-
Accumulo Stress Test
---------------------
+====================
+
+This is an attempt to observe the behavior Accumulo displays when compacting and
+reading cells. There are two components to this package:
+`org.apache.accumulo.test.stress.random.Write` and
+`org.apache.accumulo.test.stress.random.Scan`.
+
+Since this test is meant to stress the system, there is no pre-defined test
+period and no success criteria that we can programmatically check. During a
+successful test period, the Accumulo cluster should remain stable and
+functional. Possible failure conditions include component failures, such as
+tablet servers running out of memory.
+
+The `org.apache.accumulo.test.stress.random.Write` utility provides facilities
+for writing random sized cells. Users can configure minimum and maximum sized
+portions of a cell. The portions users can configure are the row, column family,
+column qualifier and value. Note that the sizes are uniformly distributed
+between the minimum and maximum values.
+
+The `org.apache.accumulo.test.stress.random.Scan` utility provides users with
+the ability to query tables generated by the Write. It will pick a tablet at
+random and scan the entire range. The amount of times this process is performed
+is user configurable. By default, it happens 1,024 times. Users can also specify
+whether or not the scan should be isolated.
+
+There is no shared state intended by either of these services. This allows
+multiple clients to be run in parallel, either on the same host or distributed
+across hosts.
+
+## Prerequisites
+
+The utilities are run on a normal, initialized Accumulo cluster. By default,
+they only work with a table named "stress_test".
+
+The start and stop scripts rely on pssh. Before running them, you will need
+to install pssh on the machines that will be controlled by them.
+
+## Running
+
+Copy `stress-env.sh.example` to `stress-env.sh` and edit it, supplying at least
+correct values for the USERPASS and INSTANCE variables. The remainder of this
+file describes and provides defaults for many of the configuration options
+for the stress utilities.
+
+### Individual Execution
+
+Before starting a fresh write test, clear out the test table using the Accumulo
+shell.
+
+ > deletetable -f stress_test
+ > createtable stress_test
+
+To run a writer:
+
+ $ ./writer.sh
+
+The writer will begin logging into a "logs" subdirectory, using timestamped
+filenames. You can stop the writer by killing it or using Control-C.
+
+To run a reader:
+
+ $ ./reader.sh
-This is an attempt to observe the behavior Accumulo displays when compacting and reading these cells. There are two components to this package: `org.apache.accumulo.test.stress.random.Write` and `org.apache.accumulo.test.stress.random.Scan`.
+The reader logs like the writer. By default, the reader reads forever, but
+it can be configured for a fixed number of scan iterations.
-Since this test is meant to stress the system, there is no pre-defined test period and no success criteria that we can programmatically check. During a successful test period, the Accumulo cluster should remain stable and functional. Possible failure conditions include component failures, such as tablet servers running out of memory.
+### Group Execution
-The `org.apache.accumulo.test.stress.random.Write` utility provides facilities for writing random sized cells. Users can configure minimum and maximum sized portions of a cell. The portions users can configure are the row, column family, column qualifier and value. Note that the sizes are uniformly distributed between the minimum and maximum values
+To run writers across multiple machines, create a writers file and list each
+host that will run a writer, one per line.
-The `org.apache.accumulo.test.stress.random.Scan` utility provides users with the ability to query tables generated by the Write. It will pick a tablet at random and scan the entire range. The amount of times this process is performed is user configurable. By default, it happens 1,024 times. Users can also specify whether or not the scan should be isolated or not.
+Finally, to start and stop the writers:
-There is no shared state intended by either of these services. This allows multiple clients to be run in parallel, either on the same host or distributedacross hosts.
+ $ ./start-writers.sh
+ $ ./stop-writers.sh
-See the provided `test/system/stress/writer.sh` and `test/system/stress/reader.sh` scripts for example configuration options and how to launch the processes.
+For readers, make a similar readers file, and use `start-readers.sh` and
+`stop-readers.sh`.
+The start scripts copy out stress-env.sh to each of the writers and readers,
+overwriting what is there, so you only need to edit the file on the host from
+which you run those scripts.
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/reader.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/reader.sh b/test/system/stress/reader.sh
index 0b796f0..1138685 100755
--- a/test/system/stress/reader.sh
+++ b/test/system/stress/reader.sh
@@ -15,24 +15,24 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# Edit the credentials to match your system
-USERPASS='-u root -p secret'
-INSTANCE='-z localhost -i inst'
-
-# This is the seed for the range picking logic used by the scanner.
-SCAN_SEED='--scan-seed 1337'
-
-# Controls the number of random tablets the scanner will read sequentially
-#SCAN_ITERATIONS='--num-iterations 1024'
-
-# Alternatively, we can just continously scan
-CONTINUOUS_SCAN='--continuous'
-
-# Controls whether or not the scan will be using an isolated scanner. Add this to the execution
-#SCAN_ISOLATION='--isolate'
-
-# Sets the batch size for the scanner, use a lower number for large rows / cells
-#SCAN_BATCH_SIZE='--scan-batch-size -1'
-
-../../../bin/accumulo org.apache.accumulo.test.stress.random.Scan $INSTANCE $USERPASS $SCAN_SEED $CONTINUOUS_SCAN \
- $SCAN_BATCH_SIZE
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+LOG_DIR=${DIR}/logs
+mkdir -p $LOG_DIR
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+ts=$(date +%Y%m%d%H%M%S)
+host=$(hostname)
+${ACCUMULO_HOME}/bin/accumulo org.apache.accumulo.test.stress.random.Scan $INSTANCE $USERPASS $SCAN_SEED $CONTINUOUS_SCAN $SCAN_BATCH_SIZE \
+ > $LOG_DIR/${ts}_${host}_reader.out \
+ 2> $LOG_DIR/${ts}_${host}_reader.err
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/readers
----------------------------------------------------------------------
diff --git a/test/system/stress/readers b/test/system/stress/readers
new file mode 100644
index 0000000..80a4224
--- /dev/null
+++ b/test/system/stress/readers
@@ -0,0 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+readerhost1
+readerhost2
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/start-readers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/start-readers.sh b/test/system/stress/start-readers.sh
new file mode 100755
index 0000000..709c46e
--- /dev/null
+++ b/test/system/stress/start-readers.sh
@@ -0,0 +1,40 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/readers ]]; then
+ echo readers file is missing
+ exit 1
+fi
+
+# Copy environment out
+pscp -h ${DIR}/readers ${DIR}/stress-env.sh ${DIR}
+
+pssh -h ${DIR}/readers "nohup ${DIR}/reader.sh >${DIR}/reader.out 2>${DIR}/reader.err < /dev/null &"
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/start-writers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/start-writers.sh b/test/system/stress/start-writers.sh
new file mode 100755
index 0000000..b9464cc
--- /dev/null
+++ b/test/system/stress/start-writers.sh
@@ -0,0 +1,40 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/writers ]]; then
+ echo writers file is missing
+ exit 1
+fi
+
+# Copy environment out
+pscp -h ${DIR}/writers ${DIR}/stress-env.sh ${DIR}
+
+pssh -h ${DIR}/writers "nohup ${DIR}/writer.sh >${DIR}/writer.out 2>${DIR}/writer.err < /dev/null &"
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/stop-readers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/stop-readers.sh b/test/system/stress/stop-readers.sh
new file mode 100755
index 0000000..1c59971
--- /dev/null
+++ b/test/system/stress/stop-readers.sh
@@ -0,0 +1,36 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/readers ]]; then
+ echo readers file is missing
+ exit 1
+fi
+pssh -h ${DIR}/readers "pkill -f '[o]rg.apache.accumulo.test.stress.random.Scan'" < /dev/null
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/stop-writers.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/stop-writers.sh b/test/system/stress/stop-writers.sh
new file mode 100755
index 0000000..d925cb3
--- /dev/null
+++ b/test/system/stress/stop-writers.sh
@@ -0,0 +1,36 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+
+# Source environment
+. ${DIR}/stress-env.sh
+
+if [[ ! -f ${DIR}/writers ]]; then
+ echo writers file is missing
+ exit 1
+fi
+pssh -h ${DIR}/writers "pkill -f '[o]rg.apache.accumulo.test.stress.random.Write'" < /dev/null
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/stress-env.sh.example
----------------------------------------------------------------------
diff --git a/test/system/stress/stress-env.sh.example b/test/system/stress/stress-env.sh.example
new file mode 100644
index 0000000..1360c67
--- /dev/null
+++ b/test/system/stress/stress-env.sh.example
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ACCUMULO_HOME=${ACCUMULO_HOME:-/opt/accumulo}
+
+# Edit the credentials to match your system
+USERPASS='-u root -p secret'
+INSTANCE='-z localhost -i inst'
+
+### Read configuration ###
+
+# This is the seed for the range picking logic used by the scanner.
+SCAN_SEED='--scan-seed 1337'
+
+# Controls the number of random tablets the scanner will read sequentially
+#SCAN_ITERATIONS='--num-iterations 1024'
+
+# Alternatively, we can just continously scan
+CONTINUOUS_SCAN='--continuous'
+
+# Controls whether or not the scan will be using an isolated scanner. Add this to the execution
+#SCAN_ISOLATION='--isolate'
+
+# Sets the batch size for the scanner, use a lower number for large rows / cells
+#SCAN_BATCH_SIZE='--scan-batch-size -1'
+
+### Write configuration ###
+
+# Edit these to change the range of each cell component. The size is in bytes.
+ROW_RANGE='--min-row-size 128 --max-row-size 128'
+CF_RANGE='--min-cf-size 128 --max-cf-size 128'
+CQ_RANGE='--min-cq-size 128 --max-cq-size 128'
+VALUE_RANGE='--min-value-size 1024 --max-value-size 2048'
+ROW_WIDTH='--min-row-width 1 --max-row-width 10'
+
+# These are the seeds for the random number generates used to generate each cell component.
+ROW_SEED='--row-seed 1'
+CF_SEED='--cf-seed 2'
+CQ_SEED='--cq-seed 3'
+VALUE_SEED='--value-seed 4'
+ROW_WIDTH_SEED='--row-width-seed 5'
+
+# This is the delay in milliseconds between writes. Use <= 0 for no delay.
+WRITE_DELAY='--write-delay 0'
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/writer.sh
----------------------------------------------------------------------
diff --git a/test/system/stress/writer.sh b/test/system/stress/writer.sh
index db55155..7d9b283 100755
--- a/test/system/stress/writer.sh
+++ b/test/system/stress/writer.sh
@@ -15,30 +15,28 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# Edit the credentials to match your system
-USERPASS='-u root -p secret'
-INSTANCE='-z localhost -i inst'
+# Start: Resolve Script Directory
+# Ref: http://stackoverflow.com/questions/59895/
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "${SOURCE}" ]; do # resolve $SOURCE until the file is no longer a symlink
+ DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+ SOURCE="$(readlink "${SOURCE}")"
+ [[ "${SOURCE}" != /* ]] && SOURCE="${DIR}/${SOURCE}" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+done
+DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
+# Stop: Resolve Script Directory
+LOG_DIR=${DIR}/logs
+mkdir -p $LOG_DIR
-# Edit these to change the range of each cell component. The size is in bytes.
-ROW_RANGE='--min-row-size 128 --max-row-size 128'
-CF_RANGE='--min-cf-size 128 --max-cf-size 128'
-CQ_RANGE='--min-cq-size 128 --max-cq-size 128'
-VALUE_RANGE='--min-value-size 1024 --max-value-size 2048'
-ROW_WIDTH='--min-row-width 1 --max-row-width 10'
+# Source environment
+. ${DIR}/stress-env.sh
-# These are the seeds for the random number generates used to generate each cell component.
-ROW_SEED='--row-seed 1'
-CF_SEED='--cf-seed 2'
-CQ_SEED='--cq-seed 3'
-VALUE_SEED='--value-seed 4'
-ROW_WIDTH_SEED='--row-width-seed 5'
+ts=$(date +%Y%m%d%H%M%S)
+host=$(hostname)
-# This is the delay in milliseconds between writes. Use <= 0 for no delay.
-WRITE_DELAY='--write-delay 0'
+# TBD - --clear-table option
-# Let's reset the table, for good measure
-../../../bin/accumulo shell $USERPASS -e 'deletetable -f stress_test'
-../../../bin/accumulo shell $USERPASS -e 'createtable stress_test'
-
-../../../bin/accumulo org.apache.accumulo.test.stress.random.Write $INSTANCE $USERPASS $ROW_RANGE $CF_RANGE $CQ_RANGE $VALUE_RANGE \
- $ROW_SEED $CF_SEED $CQ_SEED $VALUE_SEED $ROW_WIDTH $ROW_WIDTH_SEED $WRITE_DELAY
+${ACCUMULO_HOME}/bin/accumulo org.apache.accumulo.test.stress.random.Write $INSTANCE $USERPASS $ROW_RANGE $CF_RANGE $CQ_RANGE $VALUE_RANGE \
+ $ROW_SEED $CF_SEED $CQ_SEED $VALUE_SEED $ROW_WIDTH $ROW_WIDTH_SEED $WRITE_DELAY \
+ > $LOG_DIR/${ts}_${host}_writer.out \
+ 2> $LOG_DIR/${ts}_${host}_writer.err
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e01a7206/test/system/stress/writers
----------------------------------------------------------------------
diff --git a/test/system/stress/writers b/test/system/stress/writers
new file mode 100644
index 0000000..0a06590
--- /dev/null
+++ b/test/system/stress/writers
@@ -0,0 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+writerhost1
+writerhost2