[incubator-openwhisk] branch master updated: Add gatling-throughput test for cold invocations. (#3699)

[ma...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

markusthoemmes pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-openwhisk.git


The following commit(s) were added to refs/heads/master by this push:
     new f3e1417  Add gatling-throughput test for cold invocations. (#3699)
f3e1417 is described below

commit f3e1417ee5f805ecabccb39997e099d605b5a097
Author: Christian Bickel <gi...@cbickel.de>
AuthorDate: Mon May 28 09:00:02 2018 +0200

    Add gatling-throughput test for cold invocations. (#3699)
    
    Add another gatling load test. It executes as much cold invocations as possible.
---
 .travis.yml                                        |  2 +
 tests/performance/README.md                        | 33 ++++++++
 .../src/gatling/resources/data/users.csv           |  3 +
 .../scala/ColdBlockingInvokeSimulation.scala       | 90 ++++++++++++++++++++++
 4 files changed, 128 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index ab7a0f6..a34a2a3 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -69,5 +69,7 @@ jobs:
         - OPENWHISK_HOST="172.17.0.1" CONNECTIONS="100" REQUESTS_PER_SEC="1" ./gradlew gatlingRun-ApiV1Simulation
         - OPENWHISK_HOST="172.17.0.1" MEAN_RESPONSE_TIME="1000" API_KEY="$(cat ansible/files/auth.guest)" EXCLUDED_KINDS="python:default,java:default,swift:default" ./gradlew gatlingRun-LatencySimulation
         - OPENWHISK_HOST="172.17.0.1" API_KEY="$(cat ansible/files/auth.guest)" CONNECTIONS="100" REQUESTS_PER_SEC="1" ./gradlew gatlingRun-BlockingInvokeOneActionSimulation
+        # The following configuration does not make much sense. But we do not have enough users. But it's good to verify, that the test is still working.
+        - OPENWHISK_HOST="172.17.0.1" USERS="1" REQUESTS_PER_SEC="1" ./gradlew gatlingRun-ColdBlockingInvokeSimulation
       env:
         - DESCRIPTION="Execute wrk-performance test suite."
diff --git a/tests/performance/README.md b/tests/performance/README.md
index 71302c1..9ea838d 100644
--- a/tests/performance/README.md
+++ b/tests/performance/README.md
@@ -65,6 +65,8 @@ On using CI/CD pipelines (e.g. Jenkins) you will be able to set a threshold on a
 This Simulation calls the `api/v1`.
 You can specify the endpoint, the amount of connections against the backend and the duration of this burst.
 
+The test is doing as many requests as possible for the given amount of time (`SECONDS`). Afterwards it compares if the test reached the intended throughput (`REQUESTS_PER_SEC`, `MIN_REQUESTS_PER_SEC`).
+
 Available environment variables:
 
 ```
@@ -122,6 +124,8 @@ The warmup-phase will not be part of the assertions.
 
 To run the test, you can specify the amount of concurrent requests. Keep in mind, that the actions are invoked blocking and the system is limited to `AMOUNT_OF_INVOKERS * SLOTS_PER_INVOKER * NON_BLACKBOX_INVOKER_RATIO` concurrent actions/requests.
 
+The test is doing as many requests as possible for the given amount of time (`SECONDS`). Afterwards it compares if the test reached the intended throughput (`REQUESTS_PER_SEC`, `MIN_REQUESTS_PER_SEC`).
+
 Available environment variables:
 ```
 OPENWHISK_HOST          (required)
@@ -136,3 +140,32 @@ You can run the simulation with
 ```
 OPENWHISK_HOST="openwhisk.mydomain.com" CONNECTIONS="10" REQUESTS_PER_SEC="50" API_KEY="UUID:KEY" ./gradlew gatlingRun-BlockingInvokeOneActionSimulation
 ```
+
+##### ColdBlockingInvokeSimulation
+
+This simulation makes as much cold invocations as possible. Therefor you have to specify, how many users should be used.
+This amount of users is executing actions in parallel. I recommend using the same amount of users like your amount of node-js action slots in your invokers.
+
+The users, that are used are loaded from the file `gatling_tests/src/gatling/resources/data/users.csv`. If you want to increase the number of parallel users, you have to specify at least this amount of valid users in that file.
+
+Each user creates n actions (default is 5). Afterwards all users are executing their actions in parallel. But each user is rotating it's action. That's how the cold starts are enforced.
+
+The aim of the test is, to test the throughput of the system, if all containers are always cold.
+
+The action that is invoked, writes one log line and returns a little json.
+
+The test is doing as many requests as possible for the given amount of time (`SECONDS`). Afterwards it compares if the test reached the intended throughput (`REQUESTS_PER_SEC`, `MIN_REQUESTS_PER_SEC`).
+
+Available environment variables:
+```
+OPENWHISK_HOST          (required)
+USERS                   (required)
+SECONDS                 (default: 10)
+REQUESTS_PER_SEC        (required)
+MIN_REQUESTS_PER_SEC    (default: REQUESTS_PER_SEC)
+```
+
+You can run the simulation with
+```
+OPENWHISK_HOST="openwhisk.mydomain.com" USERS="10" REQUESTS_PER_SEC="50" ./gradlew gatlingRun-ColdBlockingInvokeSimulation
+```
diff --git a/tests/performance/gatling_tests/src/gatling/resources/data/users.csv b/tests/performance/gatling_tests/src/gatling/resources/data/users.csv
new file mode 100644
index 0000000..3d1d092
--- /dev/null
+++ b/tests/performance/gatling_tests/src/gatling/resources/data/users.csv
@@ -0,0 +1,3 @@
+uuid,key
+23bc46b1-71f6-4ed5-8c54-816aa4f8c502,123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP
+add_more_users,here
diff --git a/tests/performance/gatling_tests/src/gatling/scala/ColdBlockingInvokeSimulation.scala b/tests/performance/gatling_tests/src/gatling/scala/ColdBlockingInvokeSimulation.scala
new file mode 100644
index 0000000..1e06a8b
--- /dev/null
+++ b/tests/performance/gatling_tests/src/gatling/scala/ColdBlockingInvokeSimulation.scala
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.nio.charset.StandardCharsets
+
+import extension.whisk.OpenWhiskProtocolBuilder
+import extension.whisk.Predef._
+import io.gatling.core.Predef._
+import io.gatling.core.session.Expression
+import io.gatling.core.structure.ScenarioBuilder
+import io.gatling.core.util.Resource
+import org.apache.commons.io.FileUtils
+
+import scala.concurrent.duration._
+
+class ColdBlockingInvokeSimulation extends Simulation {
+  // Specify parameters for the run
+  val host = sys.env("OPENWHISK_HOST")
+
+  val users: Int = sys.env("USERS").toInt
+  val seconds: FiniteDuration = sys.env.getOrElse("SECONDS", "10").toInt.seconds
+  val actionsPerUser: Int = sys.env.getOrElse("ACTIONS_PER_USER", "5").toInt
+
+  // Specify thresholds
+  val requestsPerSec: Int = sys.env("REQUESTS_PER_SEC").toInt
+  val minimalRequestsPerSec: Int = sys.env.getOrElse("MIN_REQUESTS_PER_SEC", requestsPerSec.toString).toInt
+
+  // Generate the OpenWhiskProtocol
+  val openWhiskProtocol: OpenWhiskProtocolBuilder = openWhisk.apiHost(host)
+
+  val feeder = csv("users.csv").queue
+
+  // Define scenario
+  val test: ScenarioBuilder = scenario("Invoke one action blocking")
+    .feed(feeder)
+    .doIf(true) {
+      // This assignment assures to use the same user within this block. Otherwise create, invoke and delete would
+      // use other users.
+      val uuid: Expression[String] = "${uuid}"
+      val key: Expression[String] = "${key}"
+
+      val actionsPerUser = 5
+      val actionName: String = "action-${i}"
+
+      // Each user uses the given amount of actions
+      repeat(actionsPerUser, "i") {
+        exec(
+          openWhisk("Create action")
+            .authenticate(uuid, key)
+            .action(actionName)
+            .create(FileUtils
+              .readFileToString(Resource.body("nodeJSAction.js").get.file, StandardCharsets.UTF_8)))
+      }.rendezVous(users)
+        // Execute all actions for the given amount of time.
+        .during(seconds) {
+          // Cycle through the actions of this user, to not invoke the same action directly one after each other.
+          // Otherwise there is the possiblity, that it is warm.
+          repeat(actionsPerUser, "i") {
+            exec(openWhisk("Invoke action").authenticate(uuid, key).action(actionName).invoke())
+          }
+        }
+        .rendezVous(users)
+        .repeat(actionsPerUser, "i") {
+          exec(openWhisk("Delete action").authenticate(uuid, key).action(actionName).delete())
+        }
+    }
+
+  setUp(test.inject(atOnceUsers(users)))
+    .protocols(openWhiskProtocol)
+    // One failure will make the build yellow
+    .assertions(details("Invoke action").requestsPerSec.gt(minimalRequestsPerSec))
+    .assertions(details("Invoke action").requestsPerSec.gt(requestsPerSec))
+    // Mark the build yellow, if there are failed requests. And red if both conditions fail.
+    .assertions(details("Invoke action").failedRequests.count.is(0))
+    .assertions(details("Invoke action").failedRequests.percent.lte(0.1))
+}

-- 
To stop receiving notification emails like this one, please contact
markusthoemmes@apache.org.