You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by Christian Bickel <ap...@cbickel.de> on 2018/03/19 10:36:27 UTC
Performance tests of OpenWhisk
Hi,
performance is one of the most important goals of OpenWhisk. But in the last few weeks, we had some performance regressions, which have not been catched fast enough.
Therefor I'm currently working on some Gatling-Tests, to stress the system and have some thresholds for the tests, to become red if there is a performance regression.
There are also some tests in the following performance repository: https://github.com/apache/incubator-openwhisk-performance (https://github.com/apache/incubator-openwhisk-performance).
But as performance is so important for us, I'd suggest, to add the content of the performance repository into the core repository. In addition, I'll put the new Gatling-load-tests into the new performance directory as well.
As far as I know, the performance repository is it's own repository, to have it's own travis. But I don't think, that it makes much sense, to test the performance in travis, as we only have one small machine there.
If you have some concerns on merging these two repositories just tell the reasons. Otherwise I will go forward.
Greetings
Christian
Re: Performance tests of OpenWhisk
Posted by Rodric Rabbah <ro...@gmail.com>.
Agreed that we can’t yet everything but perhaps these may be sensible.
1. Latency regression because of a cold start related bug (might need at least two invokers and controllers)
2. For a single invoker (non black box), some expected l/sustained throughout range (rps)
3. If we disable the black box cluster, might it be possible to catch some lb imbalance?
-r
> On Mar 19, 2018, at 7:52 AM, Christian Bickel <ap...@cbickel.de> wrote:
>
> As I said, we only have a very small travis machine.
>
> But I think it's at least worth a try, if we get consistent performance numbers in travis, to set a threshold and fail if it's below this threshold.
>
> Greetings
> Christian
>
> March 19, 2018 12:03 PM, "Rodric Rabbah" <ro...@gmail.com> wrote:
>
>> Great - are you considering also a Travis CI matrix?
>>
>> -r
Re: Performance tests of OpenWhisk
Posted by Christian Bickel <ap...@cbickel.de>.
As I said, we only have a very small travis machine.
But I think it's at least worth a try, if we get consistent performance numbers in travis, to set a threshold and fail if it's below this threshold.
Greetings
Christian
March 19, 2018 12:03 PM, "Rodric Rabbah" <ro...@gmail.com> wrote:
> Great - are you considering also a Travis CI matrix?
>
> -r
Re: Performance tests of OpenWhisk
Posted by Rodric Rabbah <ro...@gmail.com>.
Great - are you considering also a Travis CI matrix?
-r
RE: Performance tests of OpenWhisk
Posted by 김동경 <do...@navercorp.com>.
+1 on this!!
-----Original Message-----
From: Christian Bickel <ap...@cbickel.de>
To: <de...@openwhisk.apache.org>
Cc:
Sent: 2018-03-19 (월) 19:36:27
Subject: Performance tests of OpenWhisk
Hi,
performance is one of the most important goals of OpenWhisk. But in the last few weeks, we had some performance regressions, which have not been catched fast enough.
Therefor I'm currently working on some Gatling-Tests, to stress the system and have some thresholds for the tests, to become red if there is a performance regression.
There are also some tests in the following performance repository: https://github.com/apache/incubator-openwhisk-performance (https://github.com/apache/incubator-openwhisk-performance).
But as performance is so important for us, I'd suggest, to add the content of the performance repository into the core repository. In addition, I'll put the new Gatling-load-tests into the new performance directory as well.
As far as I know, the performance repository is it's own repository, to have it's own travis. But I don't think, that it makes much sense, to test the performance in travis, as we only have one small machine there.
If you have some concerns on merging these two repositories just tell the reasons. Otherwise I will go forward.
Greetings
Christian