You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@openwhisk.apache.org by ma...@apache.org on 2018/04/13 08:16:41 UTC
[incubator-openwhisk] branch master updated: Add gatling as
performance test suite. (#3526)
This is an automated email from the ASF dual-hosted git repository.
markusthoemmes pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-openwhisk.git
The following commit(s) were added to refs/heads/master by this push:
new 56891fb Add gatling as performance test suite. (#3526)
56891fb is described below
commit 56891fbeb1ca306b4685396f13f07cb4b60a3022
Author: Christian Bickel <gi...@cbickel.de>
AuthorDate: Fri Apr 13 10:16:38 2018 +0200
Add gatling as performance test suite. (#3526)
Adds an initial gatling based performance test, including a framework to talk to the OpenWhisk API in a declarative way.
This first implementation only asserts /api/v1 performance, which can be helpful to measure general network/router performance.
Co-authored-by: Markus Thömmes <ma...@me.com>
---
.travis.yml | 1 +
performance/README.md | 44 +++++++++++---
performance/gatling_tests/build.gradle | 19 ++++++
.../src/gatling/resources/conf/logback.xml | 14 +++++
.../src/gatling/scala/ApiV1Simulation.scala | 51 ++++++++++++++++
.../extension/whisk/OpenWhiskActionBuilder.scala | 38 ++++++++++++
.../scala/extension/whisk/OpenWhiskDsl.scala | 26 +++++++++
.../extension/whisk/OpenWhiskProtocolBuilder.scala | 67 ++++++++++++++++++++++
.../src/gatling/scala/extension/whisk/Predef.scala | 20 +++++++
performance/preparation/deploy.sh | 1 +
settings.gradle | 1 +
11 files changed, 275 insertions(+), 7 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 52d6109..7fc1587 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -64,5 +64,6 @@ jobs:
- ./performance/preparation/deploy.sh
- TERM=dumb ./performance/wrk_tests/latency.sh "https://172.17.0.1:10001" "$(cat ansible/files/auth.guest)" 2m
- TERM=dumb ./performance/wrk_tests/throughput.sh "https://172.17.0.1:10001" "$(cat ansible/files/auth.guest)" 4 2 2m
+ - OPENWHISK_HOST="172.17.0.1" CONNECTIONS="100" REQUESTS_PER_SEC="1" ./gradlew gatlingRun-ApiV1Simulation
env:
- DESCRIPTION="Execute wrk-performance test suite."
diff --git a/performance/README.md b/performance/README.md
index c1d668f..f089021 100644
--- a/performance/README.md
+++ b/performance/README.md
@@ -2,7 +2,7 @@
A few simple but efficient test suites for determining the maximum throughput and end-user latency of the Apache OpenWhisk system.
## Workflow
-- A standard OpenWhisk system is deployed. (_Note that the edge NGINX router and API Gateway are currently left out. As a consequence, the tests talk directly to the controller._)
+- A standard OpenWhisk system is deployed. (_Note that the API Gateway is currently left out for the tests._)
- All limits are set to 999999, which in our current use case means "No throttling at all".
- The deployment is using the docker setup proposed by the OpenWhisk development team: `overlay` driver and HTTP API enabled via a UNIX port.
@@ -13,8 +13,10 @@ The [machine provided by Travis](https://docs.travis-ci.com/user/ci-environment/
## Suites
-### Latency Test
-Determines the end-to-end latency a user experience when doing a blocking invocation. The action used is a no-op so the numbers returned are the plain overhead of the OpenWhisk system.
+### wrk
+
+#### Latency Test
+Determines the end-to-end latency a user experience when doing a blocking invocation. The action used is a no-op so the numbers returned are the plain overhead of the OpenWhisk system. The requests are directly against the controller.
- 1 HTTP request at a time (concurrency: 1)
- You can specify how long this test will run. Default are 30s.
@@ -22,12 +24,40 @@ Determines the end-to-end latency a user experience when doing a blocking invoca
**Note:** The throughput number has a 100% correlation with the latency in this case. This test does not serve to determine the maximum throughput of the system.
-### Throughput Test
-Determines the maximum throughput a user can get out of the system while using a single action. The action used is a no-op, so the numbers are plain OpenWhisk overhead. Note that the throughput does not directly correlate to end-to-end latency here, as the system does more processing in the background as it shows to the user in a blocking invocation.
+#### Throughput Test
+Determines the maximum throughput a user can get out of the system while using a single action. The action used is a no-op, so the numbers are plain OpenWhisk overhead. Note that the throughput does not directly correlate to end-to-end latency here, as the system does more processing in the background as it shows to the user in a blocking invocation. The requests are directly against the controller.
- 4 HTTP requests at a time (concurrency: 4) (using CPU cores * 2 to exploit some buffering)
- 10.000 samples with a single user
- no-op action
-## Running tests against your own system is simple too!
-All you have to do is use the corresponding script located in /*_tests folder, remembering that the parameters are defined inline.
+#### Running tests against your own system is simple too!
+All you have to do is use the corresponding script located in `/*_tests` folder, remembering that the parameters are defined inline.
+
+### gatling
+
+#### Simulations
+
+You can specify two thresholds for the simulations.
+The reason is, that Gatling is able to handle each assertion as a JUnit test.
+On using CI/CD pipelines (e.g. Jenkins) you will be able to set a threshold on an amount of failed testcases to mark the build as stable, unstable and failed.
+
+##### ApiV1Simulation
+
+This Simulation calls the `api/v1`.
+You can specify the endpoint, the amount of connections against the backend and the duration of this burst.
+
+Available environment variables:
+
+```
+OPENWHISK_HOST (required)
+CONNECTIONS (required)
+SECONDS (default: 10)
+REQUESTS_PER_SEC (required)
+MIN_REQUESTS_PER_SEC (default: REQUESTS_PER_SEC)
+```
+
+You can run the simulation with (in OPENWHISK_HOME)
+```
+OPENWHISK_HOST="openwhisk.mydomain.com" CONNECTIONS="10" REQUESTS_PER_SEC="50" ./gradlew gatlingRun-ApiV1Simulation
+```
diff --git a/performance/gatling_tests/build.gradle b/performance/gatling_tests/build.gradle
new file mode 100644
index 0000000..7e8f1ad
--- /dev/null
+++ b/performance/gatling_tests/build.gradle
@@ -0,0 +1,19 @@
+plugins {
+ id "com.github.lkishalmi.gatling" version "0.7.1"
+}
+
+apply plugin: 'eclipse'
+apply plugin: 'scala'
+
+repositories {
+ mavenCentral()
+}
+
+dependencies {
+ compile "org.scala-lang:scala-library:${gradle.scala.version}"
+ compile "io.gatling.highcharts:gatling-charts-highcharts:2.2.5"
+}
+
+tasks.withType(ScalaCompile) {
+ scalaCompileOptions.additionalParameters = gradle.scala.compileFlags
+}
diff --git a/performance/gatling_tests/src/gatling/resources/conf/logback.xml b/performance/gatling_tests/src/gatling/resources/conf/logback.xml
new file mode 100644
index 0000000..62bd2b2
--- /dev/null
+++ b/performance/gatling_tests/src/gatling/resources/conf/logback.xml
@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<configuration>
+ <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
+ <encoder>
+ <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
+ <immediateFlush>false</immediateFlush>
+ </encoder>
+ </appender>
+ <!-- This will log the details of the failed requests -->
+ <logger name="io.gatling.http.ahc" level="DEBUG" />
+ <root level="WARN">
+ <appender-ref ref="CONSOLE"/>
+ </root>
+</configuration>
diff --git a/performance/gatling_tests/src/gatling/scala/ApiV1Simulation.scala b/performance/gatling_tests/src/gatling/scala/ApiV1Simulation.scala
new file mode 100644
index 0000000..91c5302
--- /dev/null
+++ b/performance/gatling_tests/src/gatling/scala/ApiV1Simulation.scala
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import extension.whisk.Predef._
+import io.gatling.core.Predef._
+
+import scala.concurrent.duration._
+
+class ApiV1Simulation extends Simulation {
+
+ // Specify parameters for the run
+ val host = sys.env("OPENWHISK_HOST")
+ val connections = sys.env("CONNECTIONS").toInt
+ val seconds = sys.env.getOrElse("SECONDS", "10").toInt.seconds
+
+ // Specify thresholds
+ val requestsPerSec = sys.env("REQUESTS_PER_SEC").toInt
+ val minimalRequestsPerSec = sys.env.getOrElse("MIN_REQUESTS_PER_SEC", requestsPerSec.toString).toInt
+
+ // Generate the OpenWhiskProtocol
+ val openWhiskProtocol = openWhisk.apiHost(host)
+
+ // Define scenario
+ val test = scenario("api/v1 endpoint")
+ .during(seconds) {
+ exec(openWhisk("Call api/v1 endpoint").info())
+ }
+
+ setUp(test.inject(atOnceUsers(connections)))
+ .protocols(openWhiskProtocol)
+ // One failure will make the build yellow
+ .assertions(details("Call api/v1 endpoint").requestsPerSec.gt(minimalRequestsPerSec))
+ .assertions(details("Call api/v1 endpoint").requestsPerSec.gt(requestsPerSec))
+ // Mark the build yellow, if there are failed requests. And red if both conditions fail.
+ .assertions(details("Call api/v1 endpoint").failedRequests.count.is(0))
+ .assertions(details("Call api/v1 endpoint").failedRequests.percent.lte(0.1))
+}
diff --git a/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskActionBuilder.scala b/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskActionBuilder.scala
new file mode 100644
index 0000000..ef6d2ef
--- /dev/null
+++ b/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskActionBuilder.scala
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package extension.whisk
+
+import io.gatling.core.Predef._
+import io.gatling.core.action.Action
+import io.gatling.core.action.builder.ActionBuilder
+import io.gatling.core.session.Expression
+import io.gatling.core.structure.ScenarioContext
+import io.gatling.http.request.builder.{Http, HttpRequestBuilder}
+
+case class OpenWhiskActionBuilderBase(requestName: Expression[String]) {
+
+ implicit private val http = new Http(requestName)
+
+ def info() = OpenWhiskActionBuilder(http.get("/api/v1"))
+}
+
+case class OpenWhiskActionBuilder(http: HttpRequestBuilder) extends ActionBuilder {
+ override def build(ctx: ScenarioContext, next: Action): Action = {
+ http.build(ctx, next)
+ }
+}
diff --git a/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskDsl.scala b/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskDsl.scala
new file mode 100644
index 0000000..ea51847
--- /dev/null
+++ b/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskDsl.scala
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package extension.whisk
+
+import io.gatling.core.session.Expression
+
+trait OpenWhiskDsl {
+ def openWhisk = OpenWhiskProtocolBuilderBase
+
+ def openWhisk(requestName: Expression[String]) = new OpenWhiskActionBuilderBase(requestName)
+}
diff --git a/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskProtocolBuilder.scala b/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskProtocolBuilder.scala
new file mode 100644
index 0000000..fe499cf
--- /dev/null
+++ b/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskProtocolBuilder.scala
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package extension.whisk
+
+import com.softwaremill.quicklens._
+import io.gatling.core.Predef._
+import io.gatling.core.config.GatlingConfiguration
+import io.gatling.http.Predef._
+import io.gatling.http.protocol.HttpProtocol
+
+import scala.language.implicitConversions
+
+/**
+ * This is the OpenWhiskProtocol.
+ *
+ * @param apiHost url or address to connect to
+ * @param protocol protocol to use. e.g. https
+ * @param port port to use. e.g. 443
+ */
+case class OpenWhiskProtocol(apiHost: String, protocol: String = "https", port: Int = 443)
+
+case object OpenWhiskProtocolBuilderBase {
+ def apiHost(url: String): OpenWhiskProtocolBuilder = OpenWhiskProtocolBuilder(OpenWhiskProtocol(url))
+}
+
+object OpenWhiskProtocolBuilder {
+
+ /** convert the OpenWhiskProtocolBuilder to an HttpProtocol. */
+ implicit def toHttpProtocol(builder: OpenWhiskProtocolBuilder)(
+ implicit configuration: GatlingConfiguration): HttpProtocol = builder.build
+}
+
+case class OpenWhiskProtocolBuilder(private val protocol: OpenWhiskProtocol) {
+
+ /** set the api host */
+ def apiHost(url: String): OpenWhiskProtocolBuilder = this.modify(_.protocol.apiHost).setTo(url)
+
+ /** set the protocol */
+ def protocol(protocol: String): OpenWhiskProtocolBuilder = this.modify(_.protocol.protocol).setTo(protocol)
+
+ /** set the port */
+ def port(port: Int): OpenWhiskProtocolBuilder = this.modify(_.protocol.port).setTo(port)
+
+ /** build the http protocol with the parameters provided by the openwhisk-protocol. */
+ def build(implicit configuration: GatlingConfiguration) = {
+ http
+ .baseURL(s"${protocol.protocol}://${protocol.apiHost}:${protocol.port}")
+ .contentTypeHeader("application/json")
+ .userAgentHeader("gatlingLoadTest")
+ .build
+ }
+}
diff --git a/performance/gatling_tests/src/gatling/scala/extension/whisk/Predef.scala b/performance/gatling_tests/src/gatling/scala/extension/whisk/Predef.scala
new file mode 100644
index 0000000..5733f7d
--- /dev/null
+++ b/performance/gatling_tests/src/gatling/scala/extension/whisk/Predef.scala
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package extension.whisk
+
+object Predef extends OpenWhiskDsl
diff --git a/performance/preparation/deploy.sh b/performance/preparation/deploy.sh
index 0c9bfbe..8a11fd6 100755
--- a/performance/preparation/deploy.sh
+++ b/performance/preparation/deploy.sh
@@ -21,3 +21,4 @@ $ANSIBLE_CMD wipe.yml
$ANSIBLE_CMD kafka.yml
$ANSIBLE_CMD controller.yml
$ANSIBLE_CMD invoker.yml
+$ANSIBLE_CMD edge.yml
diff --git a/settings.gradle b/settings.gradle
index 4c76198..d3f56e2 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -17,6 +17,7 @@ include 'sdk:docker'
include 'tests'
include 'tests:dat:blackbox:badaction'
include 'tests:dat:blackbox:badproxy'
+include 'performance:gatling_tests'
rootProject.name = 'openwhisk'
--
To stop receiving notification emails like this one, please contact
markusthoemmes@apache.org.