You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@usergrid.apache.org by to...@apache.org on 2015/08/31 22:24:18 UTC
[2/3] usergrid git commit: USERGRID-871: add ability to add query
parameters (used for connections=none)
USERGRID-871: add ability to add query parameters (used for connections=none)
and also an org/app setup test
Project: http://git-wip-us.apache.org/repos/asf/usergrid/repo
Commit: http://git-wip-us.apache.org/repos/asf/usergrid/commit/245fb7ea
Tree: http://git-wip-us.apache.org/repos/asf/usergrid/tree/245fb7ea
Diff: http://git-wip-us.apache.org/repos/asf/usergrid/diff/245fb7ea
Branch: refs/heads/two-dot-o-dev
Commit: 245fb7eaad1f89b4182abefdb14655a739c7b73b
Parents: 579b692
Author: Mike Dunker <mi...@calbears.net>
Authored: Mon Aug 31 13:20:13 2015 -0700
Committer: Mike Dunker <mi...@calbears.net>
Committed: Mon Aug 31 13:20:13 2015 -0700
----------------------------------------------------------------------
stack/loadtests/README.md | 10 +--
stack/loadtests/runOrgAppSetup.sh | 88 ++++++++++++++++++++
stack/loadtests/runRandomEntityByNameTest.sh | 4 +-
stack/loadtests/runRandomEntityByUuidTest.sh | 6 +-
.../usergrid/enums/ConfigProperties.scala | 8 +-
.../apache/usergrid/enums/ScenarioType.scala | 3 +-
.../scenarios/EntityCollectionScenarios.scala | 6 ++
.../org/apache/usergrid/settings/Settings.scala | 22 +++++
.../simulations/ConfigurableSimulation.scala | 1 +
stack/loadtests/testConfig.sh | 1 +
stack/pom.xml | 1 +
11 files changed, 138 insertions(+), 12 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/usergrid/blob/245fb7ea/stack/loadtests/README.md
----------------------------------------------------------------------
diff --git a/stack/loadtests/README.md b/stack/loadtests/README.md
index 0e70702..74f93bf 100644
--- a/stack/loadtests/README.md
+++ b/stack/loadtests/README.md
@@ -1,5 +1,5 @@
#Gatling Load Tests
-The Usergrid loadtests directory (/stack/loadtests) contains a framework for performance testing [Apache Usergrid](http://usergrid.apache.org/). These tests currently use version 2.1.7 of [Gatling](http://gatling.io), an open-source load-testing tool.
+The Usergrid loadtests directory (/stack/loadtests) contains a framework for performance testing [Apache Usergrid](http://usergrid.apache.org/). These tests currently use version 2.1.7 of [Gatling](http://gatling.io), an open-source load-testing tool. This version of Gatling, as well as the tests, use Java 7.
The test code is written in [Scala](http://www.scala-lang.org/), which is Gatling's test language.
@@ -55,7 +55,7 @@ Retrieves random entities via UUID from a CSV file.
Updates entities in order via name (prefix + entity number).
##Gatling configuration items
-Understanding how configuration items work together can best be accomplished by reading the Usergrid Gatling test scripts. Some configuration items are dependent on others, and some are ignored by specific tests. Configuration items and their defaults can be found in the ConfigProperties.scala enumeration in the enums directory. The spelling of each item below is used in maven calls via -D{configName}={value} (for example, -Dorg=gatlingtest).
+Understanding how configuration items work together can best be accomplished by reading the Usergrid Gatling test scripts. Some configuration items are dependent on others, and some are ignored by specific tests. Configuration items and their defaults can be found in the ConfigProperties.scala enumeration in the enums directory. The spelling of each item below is used in Maven calls via -D{configName}={value} (for example, -Dorg=gatlingtest).
Defaults listed are those that are specified by the Usergrid Gatling code, not necessarily defaults in the test scripts. Defaults are **bold**.
@@ -170,11 +170,11 @@ Feel free to skip this section -- it contains information to help you understand
* /stack/loadtests/src/main/scala/org/apache/usergrid/simulations - Contains [Gatling simulations](http://gatling.io/docs/2.1.7/general/simulation_structure.html) for running the tests
* AuditSimulation.scala - Tests for a) finding all entities in an organization and writing the names and UUIDs to a CSV file, and b) validating the existence of the entities in the CSV file; audit tests can be used to test that a copy/migration of an organization is complete
- * ConfigurableSimulation.scala - contains many different types of tests that can be configured via Gatling test shell script or maven Gatling call
- * CustomInjectionSimulation.scala - tests that allow full configuration of [Gatling user injection] via Gatling test shell script or maven Gatling call
+ * ConfigurableSimulation.scala - contains many different types of tests that can be configured via Gatling test shell script or Maven Gatling call
+ * CustomInjectionSimulation.scala - tests that allow full configuration of [Gatling user injection] via Gatling test shell script or Maven Gatling call
##Running tests using Maven
-Gatling uses [Apache Maven](https://maven.apache.org/) to handle dependencies and run the tests. The test scripts run gatling via maven, and have good examples of the maven calls.
+Gatling uses [Apache Maven](https://maven.apache.org/) to handle dependencies and run the tests. The test scripts run Gatling via Maven, and have good examples of the Maven calls.
###Example Maven (run from loadtests directory)
http://git-wip-us.apache.org/repos/asf/usergrid/blob/245fb7ea/stack/loadtests/runOrgAppSetup.sh
----------------------------------------------------------------------
diff --git a/stack/loadtests/runOrgAppSetup.sh b/stack/loadtests/runOrgAppSetup.sh
new file mode 100755
index 0000000..19936b7
--- /dev/null
+++ b/stack/loadtests/runOrgAppSetup.sh
@@ -0,0 +1,88 @@
+#!/bin/bash
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+DIR="${BASH_SOURCE%/*}"
+if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
+. "$DIR/testConfig.sh"
+
+# from testConfig.sh
+#URL=
+#ADMIN_USER=
+#ADMIN_PASSWORD=
+#ENTITY_WORKER_NUM= #may be overridden on command line
+#ENTITY_WORKER_COUNT= #may be overridden on command line
+#AUTH_TYPE=
+#TOKEN_TYPE=
+#NUM_ENTITIES= #may be overridden on command line
+#ENTITY_TYPE=
+#ENTITY_PREFIX=
+#ENTITY_SEED= #may be overridden on command line
+#RETRY_COUNT=
+#ENTITY_PROGRESS_COUNT=
+#CONSTANT_USERS_PER_SEC=
+#CONSTANT_USERS_DURATION=
+
+die() { echo "$@" 1>&2 ; exit 1; }
+
+[ "$#" -ge 2 ] || die "At least 2 arguments required, $# provided. Example is $0 ORG APP [COLLECTION [SANDBOX_COLLECTION (true/false)]]"
+
+ORG="$1"
+APP="$2"
+COLLECTION="gatlingitems"
+[ "$#" -ge 3 ] && COLLECTION="$3"
+SANDBOX_COLLECTION=true
+[ "$#" -ge 4 ] && SANDBOX_COLLECTION="$4"
+
+shift $#
+
+SCENARIO_TYPE=doNothing
+SKIP_SETUP=false
+CREATE_ORG=true
+CREATE_APP=true
+RAMP_USERS=1
+RAMP_TIME=1
+
+#Compile everything
+mvn compile
+
+#Execute the test
+mvn gatling:execute \
+-DbaseUrl=${URL} \
+-DadminUser=${ADMIN_USER} \
+-DadminPassword=${ADMIN_PASSWORD} \
+-Dorg=${ORG} \
+-Dapp=${APP} \
+-DauthType=${AUTH_TYPE} \
+-DtokenType=${TOKEN_TYPE} \
+-DcreateOrg=${CREATE_ORG} \
+-DcreateApp=${CREATE_APP} \
+-DsandboxCollection=${SANDBOX_COLLECTION} \
+-DnumEntities=${NUM_ENTITIES} \
+-DskipSetup=${SKIP_SETUP} \
+-Dcollection=${COLLECTION} \
+-DentityType=${ENTITY_TYPE} \
+-DentityPrefix=${ENTITY_PREFIX} \
+-DentitySeed=${ENTITY_SEED} \
+-DretryCount=${RETRY_COUNT} \
+-DentityProgressCount=${ENTITY_PROGRESS_COUNT} \
+-DconstantUsersPerSec=${CONSTANT_USERS_PER_SEC} \
+-DconstantUsersDuration=${CONSTANT_USERS_DURATION} \
+-DscenarioType=${SCENARIO_TYPE} \
+-DloadEntities=${LOAD_ENTITIES} \
+-DrampUsers=${RAMP_USERS} \
+-DrampTime=${RAMP_TIME} \
+-DprintFailedRequests=${PRINT_FAILED_REQUESTS} \
+-Dgatling.simulationClass=org.apache.usergrid.simulations.ConfigurableSimulation
+
http://git-wip-us.apache.org/repos/asf/usergrid/blob/245fb7ea/stack/loadtests/runRandomEntityByNameTest.sh
----------------------------------------------------------------------
diff --git a/stack/loadtests/runRandomEntityByNameTest.sh b/stack/loadtests/runRandomEntityByNameTest.sh
index 35be011..f52747b 100755
--- a/stack/loadtests/runRandomEntityByNameTest.sh
+++ b/stack/loadtests/runRandomEntityByNameTest.sh
@@ -43,12 +43,13 @@ if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
die() { echo "$@" 1>&2 ; exit 1; }
-[ "$#" -ge 2 ] || die "At least 2 arguments required, $# provided. Example is $0 RAMP_USERS RAMP_TIME(seconds) [NUM_ENTITIES [ENTITY_SEED]]"
+[ "$#" -ge 2 ] || die "At least 2 arguments required, $# provided. Example is $0 RAMP_USERS RAMP_TIME(seconds) [NUM_ENTITIES [ENTITY_SEED [QUERY_PARAMS]]]"
RAMP_USERS="$1"
RAMP_TIME="$2"
[ "$#" -ge 3 ] && NUM_ENTITIES="$3"
[ "$#" -ge 4 ] && ENTITY_SEED="$4"
+[ "$#" -ge 5 ] && QUERY_PARAMS="$5"
shift $#
@@ -85,5 +86,6 @@ mvn gatling:execute \
-DrampUsers=${RAMP_USERS} \
-DrampTime=${RAMP_TIME} \
-DprintFailedRequests=${PRINT_FAILED_REQUESTS} \
+-DqueryParams=${QUERY_PARAMS} \
-Dgatling.simulationClass=org.apache.usergrid.simulations.ConfigurableSimulation
http://git-wip-us.apache.org/repos/asf/usergrid/blob/245fb7ea/stack/loadtests/runRandomEntityByUuidTest.sh
----------------------------------------------------------------------
diff --git a/stack/loadtests/runRandomEntityByUuidTest.sh b/stack/loadtests/runRandomEntityByUuidTest.sh
index b224615..01f4fe5 100755
--- a/stack/loadtests/runRandomEntityByUuidTest.sh
+++ b/stack/loadtests/runRandomEntityByUuidTest.sh
@@ -41,13 +41,14 @@ if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
die() { echo "$@" 1>&2 ; exit 1; }
-[ "$#" -ge 3 ] || die "At least 3 arguments required, $# provided. Example is $0 RAMP_USERS RAMP_TIME(seconds) UUID_FILENAME"
+[ "$#" -ge 3 ] || die "At least 3 arguments required, $# provided. Example is $0 RAMP_USERS RAMP_TIME(seconds) UUID_FILENAME [QUERY_PARAMS]"
RAMP_USERS="$1"
RAMP_TIME="$2"
UUID_FILENAME="$3"
+[ "$#" -ge 4 ] && QUERY_PARAMS="$4"
-shift 3
+shift $#
SCENARIO_TYPE=uuidRandomInfinite
@@ -81,5 +82,6 @@ mvn gatling:execute \
-DrampTime=${RAMP_TIME} \
-DuuidFilename=${UUID_FILENAME} \
-DprintFailedRequests=${PRINT_FAILED_REQUESTS} \
+-DqueryParams=${QUERY_PARAMS} \
-Dgatling.simulationClass=org.apache.usergrid.simulations.ConfigurableSimulation
http://git-wip-us.apache.org/repos/asf/usergrid/blob/245fb7ea/stack/loadtests/src/main/scala/org/apache/usergrid/enums/ConfigProperties.scala
----------------------------------------------------------------------
diff --git a/stack/loadtests/src/main/scala/org/apache/usergrid/enums/ConfigProperties.scala b/stack/loadtests/src/main/scala/org/apache/usergrid/enums/ConfigProperties.scala
index 0a5f5af..6647549 100644
--- a/stack/loadtests/src/main/scala/org/apache/usergrid/enums/ConfigProperties.scala
+++ b/stack/loadtests/src/main/scala/org/apache/usergrid/enums/ConfigProperties.scala
@@ -79,6 +79,7 @@ object ConfigProperties {
val MultiPropertyCount = "multiPropertyCount"
val MultiPropertySizeInK = "multiPropertySizeInK"
val EntityNumberProperty = "entityNumberProperty"
+ val QueryParams = "queryParams"
val Values = Seq(Org,App,AdminUser,AdminPassword,BaseUrl,AuthType,TokenType,SkipSetup,CreateOrg,CreateApp,LoadEntities,
ScenarioType,RampUsers,ConstantUsersPerSec,ConstantUsersDuration,UserSeed,AppUser,AppUserPassword,NumEntities,
@@ -87,7 +88,7 @@ object ConfigProperties {
OrgCreationName,OrgCreationEmail,OrgCreationPassword,UpdateProperty,UpdateValue,EntityWorkerCount,EntityWorkerNum,
UuidFilename,AuditUuidFilename,FailedUuidFilename,SandboxCollection,PurgeUsers,RetryCount,LaterThanTimestamp,
EntityProgressCount,InjectionList,PrintFailedRequests,GetViaQuery,MultiPropertyPrefix,MultiPropertyCount,
- MultiPropertySizeInK,EntityNumberProperty)
+ MultiPropertySizeInK,EntityNumberProperty,QueryParams)
def isValid(str: String): Boolean = {
Values.contains(str)
@@ -139,8 +140,8 @@ object ConfigProperties {
case UpdateValue => new Date().toString
case EntityWorkerCount => 0
case EntityWorkerNum => 0
- case UuidFilename => "/tmp/dummyUuid.csv"
- case AuditUuidFilename => "/tmp/dummyAuditUuid.csv"
+ case UuidFilename => "dummyUuid.csv"
+ case AuditUuidFilename => "dummyAuditUuid.csv"
case FailedUuidFilename => "/tmp/dummyFailedUuid.csv"
case SandboxCollection => false
case PurgeUsers => 100
@@ -154,6 +155,7 @@ object ConfigProperties {
case MultiPropertyCount => 1
case MultiPropertySizeInK => 1
case EntityNumberProperty => ""
+ case QueryParams => ""
}
} else {
null
http://git-wip-us.apache.org/repos/asf/usergrid/blob/245fb7ea/stack/loadtests/src/main/scala/org/apache/usergrid/enums/ScenarioType.scala
----------------------------------------------------------------------
diff --git a/stack/loadtests/src/main/scala/org/apache/usergrid/enums/ScenarioType.scala b/stack/loadtests/src/main/scala/org/apache/usergrid/enums/ScenarioType.scala
index 6f9a20e..c86dc33 100644
--- a/stack/loadtests/src/main/scala/org/apache/usergrid/enums/ScenarioType.scala
+++ b/stack/loadtests/src/main/scala/org/apache/usergrid/enums/ScenarioType.scala
@@ -26,9 +26,10 @@ object ScenarioType {
val GetByNameSequential = "getByNameSequential"
val AuditGetCollectionEntities = "auditGetCollectionEntities"
val AuditVerifyCollectionEntities = "auditVerifyCollectionEntities"
+ val DoNothing = "doNothing"
val Values = Seq(GetAllByCursor,NameRandomInfinite,LoadEntities,DeleteEntities,UpdateEntities,UuidRandomInfinite,
- GetByNameSequential,AuditGetCollectionEntities,AuditVerifyCollectionEntities)
+ GetByNameSequential,AuditGetCollectionEntities,AuditVerifyCollectionEntities,DoNothing)
def isValid(str: String): Boolean = {
Values.contains(str)
http://git-wip-us.apache.org/repos/asf/usergrid/blob/245fb7ea/stack/loadtests/src/main/scala/org/apache/usergrid/scenarios/EntityCollectionScenarios.scala
----------------------------------------------------------------------
diff --git a/stack/loadtests/src/main/scala/org/apache/usergrid/scenarios/EntityCollectionScenarios.scala b/stack/loadtests/src/main/scala/org/apache/usergrid/scenarios/EntityCollectionScenarios.scala
index 354420b..38a70ff 100644
--- a/stack/loadtests/src/main/scala/org/apache/usergrid/scenarios/EntityCollectionScenarios.scala
+++ b/stack/loadtests/src/main/scala/org/apache/usergrid/scenarios/EntityCollectionScenarios.scala
@@ -147,6 +147,7 @@ object EntityCollectionScenarios {
val getRandomEntityByUuidAnonymous = exec(
http("GET entity by UUID (anonymous)")
.get("/" + Settings.collection + "/${uuid}")
+ .queryParamMap(Settings.queryParamMap)
.headers(Headers.authAnonymous)
.check(status.is(200))
)
@@ -154,6 +155,7 @@ object EntityCollectionScenarios {
val getRandomEntityByUuidWithToken = exec(
http("GET entity by UUID (token)")
.get("/" + Settings.collection + "/${uuid}")
+ .queryParamMap(Settings.queryParamMap)
.headers(Headers.authToken)
.check(status.is(200))
)
@@ -287,6 +289,7 @@ object EntityCollectionScenarios {
doIf("${validEntity}", "yes") {
exec(http("GET entity by name sequential (anonymous)")
.get("/" + Settings.collection + "/${entityName}")
+ .queryParamMap(Settings.queryParamMap)
.headers(Headers.authAnonymous)
.check(status.is(200), extractCreateUuid(SessionVarUuid)))
.exec(session => {
@@ -300,6 +303,7 @@ object EntityCollectionScenarios {
doIf("${validEntity}", "yes") {
exec(http("GET entity by name sequential (anonymous)")
.get("/" + Settings.collection + "/${entityName}")
+ .queryParamMap(Settings.queryParamMap)
.headers(Headers.authToken)
.check(status.is(200), extractCreateUuid(SessionVarUuid)))
.exec(session => {
@@ -329,4 +333,6 @@ object EntityCollectionScenarios {
}
}
+ val doNothing = scenario("Do Nothing").exec(http("Get Page").get("http://google.com"))
+
}
http://git-wip-us.apache.org/repos/asf/usergrid/blob/245fb7ea/stack/loadtests/src/main/scala/org/apache/usergrid/settings/Settings.scala
----------------------------------------------------------------------
diff --git a/stack/loadtests/src/main/scala/org/apache/usergrid/settings/Settings.scala b/stack/loadtests/src/main/scala/org/apache/usergrid/settings/Settings.scala
index c78a347..0a800cc 100755
--- a/stack/loadtests/src/main/scala/org/apache/usergrid/settings/Settings.scala
+++ b/stack/loadtests/src/main/scala/org/apache/usergrid/settings/Settings.scala
@@ -17,6 +17,8 @@
package org.apache.usergrid.settings
import java.io.{PrintWriter, FileOutputStream}
+import java.net.URLDecoder
+import java.util
import java.util.concurrent.TimeUnit
import java.util.concurrent.atomic.AtomicInteger
import javax.xml.bind.DatatypeConverter
@@ -176,6 +178,9 @@ object Settings {
val injectionList = initStrSetting(ConfigProperties.InjectionList)
val printFailedRequests:Boolean = initBoolSetting(ConfigProperties.PrintFailedRequests)
val getViaQuery:Boolean = initBoolSetting(ConfigProperties.GetViaQuery)
+ private val queryParamConfig = initStrSetting(ConfigProperties.QueryParams)
+ val queryParamMap: Map[String,String] = mapFromQueryParamConfigString(queryParamConfig)
+
val multiPropertyPrefix = initStrSetting(ConfigProperties.MultiPropertyPrefix)
val multiPropertyCount:Int = initIntSetting(ConfigProperties.MultiPropertyCount)
val multiPropertySizeInK:Int = initIntSetting(ConfigProperties.MultiPropertySizeInK)
@@ -478,6 +483,23 @@ object Settings {
println()
}
+ def mapFromQueryParamConfigString(queryParamConfigString: String): Map[String,String] = {
+ val params = mutable.Map[String,String]()
+ val paramStrings:Array[String] = queryParamConfigString split "&"
+ for (i <- paramStrings.indices) {
+ val param = paramStrings(i)
+ val pair = param split "="
+ val key = URLDecoder.decode(pair(0), "UTF-8")
+ val value = pair.length match {
+ case l if l > 1 => URLDecoder.decode(pair(1), "UTF-8")
+ case _ => ""
+ }
+ params(key) = value
+ println(s"QueryParam $key = $value")
+ }
+ params.toMap
+ }
+
printSettingsSummary(false)
}
http://git-wip-us.apache.org/repos/asf/usergrid/blob/245fb7ea/stack/loadtests/src/main/scala/org/apache/usergrid/simulations/ConfigurableSimulation.scala
----------------------------------------------------------------------
diff --git a/stack/loadtests/src/main/scala/org/apache/usergrid/simulations/ConfigurableSimulation.scala b/stack/loadtests/src/main/scala/org/apache/usergrid/simulations/ConfigurableSimulation.scala
index 96b87e5..9d4b05d 100755
--- a/stack/loadtests/src/main/scala/org/apache/usergrid/simulations/ConfigurableSimulation.scala
+++ b/stack/loadtests/src/main/scala/org/apache/usergrid/simulations/ConfigurableSimulation.scala
@@ -47,6 +47,7 @@ class ConfigurableSimulation extends Simulation {
case ScenarioType.NameRandomInfinite => EntityCollectionScenarios.getRandomEntitiesByName
case ScenarioType.UuidRandomInfinite => EntityCollectionScenarios.getRandomEntitiesByUuid
case ScenarioType.GetByNameSequential => EntityCollectionScenarios.getEntitiesByNameSequential
+ case ScenarioType.DoNothing => EntityCollectionScenarios.doNothing
case _ => null
}
}
http://git-wip-us.apache.org/repos/asf/usergrid/blob/245fb7ea/stack/loadtests/testConfig.sh
----------------------------------------------------------------------
diff --git a/stack/loadtests/testConfig.sh b/stack/loadtests/testConfig.sh
index 792df27..72b17c3 100755
--- a/stack/loadtests/testConfig.sh
+++ b/stack/loadtests/testConfig.sh
@@ -61,3 +61,4 @@ INJECTION_LIST="rampUsers(100,300);nothingFor(300)"
PRINT_FAILED_REQUESTS=true
GET_VIA_QUERY=false
+QUERY_PARAMS=
http://git-wip-us.apache.org/repos/asf/usergrid/blob/245fb7ea/stack/pom.xml
----------------------------------------------------------------------
diff --git a/stack/pom.xml b/stack/pom.xml
index 16783b2..40a56fa 100644
--- a/stack/pom.xml
+++ b/stack/pom.xml
@@ -1553,6 +1553,7 @@
<exclude>**/tempExport*</exclude>
<exclude>loadtests/loadtest_setup.sh</exclude>
<exclude>loadtests/gatling/user-files/request-bodies/**</exclude>
+ <exclude>loadtests/.java-version</exclude>
<!-- gatling loadtest data files -->
<exclude>loadtests/src/main/scala/org/apache/usergrid/datagenerators/datafiles/**.txt</exclude>