You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by Michael Marth <mm...@adobe.com.INVALID> on 2018/04/06 13:38:34 UTC
How to best run non-local tests in ASF (was: Performance tests for
OpenWhisk)
Hi mentors (and others),
Had an offline discussion this week in which the question came up how an ASF project should best go about running performance/throughput/scalability tests – i.e. tests that cannot be run locally and require a repeatable environment.
Some options:
* interested companies run the tests on their own infra and publish results. Pretty lame, especially because typically only that company’s engineers can access the env and investigate further.
* interested companies donate cash to sponsor compute resources, committers can run and investigate the tests. Ideal from tech perspective, but I have no idea how that cash would make its way from the ASF to a particular project.
* maybe a middle-ground: interested party that happens to have a public cloud offering gives credentials to committers
I am mainly interested to learn if there are other ASF projects (e.g. in the Big Data/Hadoop ecosystem) that do something similar. Or if there is an ASF-recommended way to do this. Or else, where I could ask this question?
Thanks!
Michael
From: Michael Marth <mm...@adobe.com>
Date: Wednesday 3 May 2017 20:57
To: "dev@openwhisk.apache.org" <de...@openwhisk.apache.org>
Subject: Re: Performance tests for OpenWhisk
Markus,
Quick update: sent the below to users@infra. So far no reaction. The archive is here [1] but Bertrand tells me only ASF member have access - for whatever reason.
Michael
[1] https://lists.apache.org/thread.html/70999f9233dac9b416ef9dedc97c0ef196a938c05d6a407b94ba3479@%3Cusers.infra.apache.org%3E
On Fri, Apr 28, 2017 at 2:23 PM, Michael Marth <mm...@adobe.com>> wrote:
Dear Infra team,
I am enquiring on behalf of the OpenWhisk project (currently in Incubator)
[1].
We would like to periodically run performance tests on a distributed
environment (OpenWhisk typically runs on more than 1 machine). So we are
basically looking for an ability to spin up/tear down a number of (virtual)
machines and exclusively use them for a certain amount of time (so that the
VMs are not shared and the performance test results are comparable over
time).
The order of magnitude would be ~5-10 VMs for 1 hour 3 times a week.
I would like to find out if there is an ASF-supported mechanism to do that.
For example, can Infra provide such infrastructure? Or is there a cloud
provider (like Azure) that might sponsor such efforts with VMs? Or maybe
there is an established way for commercial companies that are interested in
an ASF project to sponsor (fund) such tests?
If none of the above exists, then it would also be helpful for us to get to
know how other projects run such sort of tests.
Thanks a lot!
Michael
[1]
https://lists.apache.org/thread.html/b66ab5b438f2db5cdc8c5f5eabece201b4ad090058fa3a9a3bd09d12@%3Cdev.openwhisk.apache.org%3E
From: Markus Thömmes <ma...@me.com>>
Reply-To: "dev@openwhisk.apache.org<ma...@openwhisk.apache.org>" <de...@openwhisk.apache.org>>
Date: Wednesday 26 April 2017 12:59
To: "dev@openwhisk.apache.org<ma...@openwhisk.apache.org>" <de...@openwhisk.apache.org>>
Subject: Re: Performance tests for OpenWhisk
Hi Michael,
yeah that sounds pretty much spot on. I'd like to have at least 2 VMs with 4+ cores and 8GB memory. One VM would host the management stack while one would be dedicated to an Invoker only. That way we could assert single-invoker performance the easiest.
Thanks for helping!
Cheers,
Markus
Am 26. April 2017 um 11:36 schrieb Michael Marth <mm...@adobe.com>>:
Markus,
Does what I describe reflect what you are looking for?
If yes, I am happy to ask on infra.
Let me know
Michael
On 26/04/17 07:52, "Bertrand Delacretaz" <bd...@apache.org>> wrote:
Hi Michael,
On Tue, Apr 25, 2017 at 6:52 PM, Michael Marth <mm...@adobe.com>> wrote:
...Maybe our mentors can chime in. Has this been discussed in the ASF board or so?...
Best would be to ask the ASF infrastructure team via
users@infra.apache.org<ma...@infra.apache.org> - briefly describe what you need to see what's
possible.
-Bertrand
Re: How to best run non-local tests in ASF (was: Performance tests for
OpenWhisk)
Posted by Matt Rutkowski <mr...@us.ibm.com>.
Hi Michael,
The model that I am familiar with is from Apache SystemML where Apache
Infra. could not provide hardware Compute resources (actual GPU) that
could effectively run the code for testing in a reasonable amount of time.
Therefore, IBM worked out a deal with Apache where IBM donated suitable
Compute resources and provided access to Apache Infra. to manage them.
I reached out this morning to a colleague in my group at IBM, Luciano
Resende, who worked on negotiating and setting up these testing pipelines
b/w IBM and Apache (SystemML), and also described similar arrangements for
Apache Spark and Bahir projects.
Here is what he described to me:
here are the possibilities for having a heavy Ci infrastructure
for an Apache Project (assuming Apache CI infrastructure does not provide
you enough resources)
Self hosted CI infrastructure: This is the scenario that we use in Apache
Spark, Apache SystemML and some portions of Apache Bahir. A company
provision machines (in the case of Spark it's AMPLAB and for the others
it's IBM) and than we configure and manage these machines with the project
communities, providing public access to build outputs and management
access per request for committers/pmc members.
Apache managed donated machines: In this scenario, which was a little more
popular a few years ago, you can procure a set of machines and donate
these to Apache trough a target donation which in summary means it should
be used by your project and not shared with the overall projects. In this
case, the management of the infrastructure is done by Apache, and these
nodes would be added to their jenkins infrastructure and your jobs
assigned to run on these machines.
He indicated that if we want to hear more that he can join this thread to
answer any further or more detailed questions.
Kind regards,
Matt
From: Michael Marth <mm...@adobe.com.INVALID>
To: "dev@openwhisk.apache.org" <de...@openwhisk.apache.org>
Date: 04/06/2018 08:39 AM
Subject: How to best run non-local tests in ASF (was: Performance
tests for OpenWhisk)
Hi mentors (and others),
Had an offline discussion this week in which the question came up how an
ASF project should best go about running
performance/throughput/scalability tests – i.e. tests that cannot be run
locally and require a repeatable environment.
Some options:
* interested companies run the tests on their own infra and publish
results. Pretty lame, especially because typically only that company’s
engineers can access the env and investigate further.
* interested companies donate cash to sponsor compute resources,
committers can run and investigate the tests. Ideal from tech perspective,
but I have no idea how that cash would make its way from the ASF to a
particular project.
* maybe a middle-ground: interested party that happens to have a public
cloud offering gives credentials to committers
I am mainly interested to learn if there are other ASF projects (e.g. in
the Big Data/Hadoop ecosystem) that do something similar. Or if there is
an ASF-recommended way to do this. Or else, where I could ask this
question?
Thanks!
Michael
From: Michael Marth <mm...@adobe.com>
Date: Wednesday 3 May 2017 20:57
To: "dev@openwhisk.apache.org" <de...@openwhisk.apache.org>
Subject: Re: Performance tests for OpenWhisk
Markus,
Quick update: sent the below to users@infra. So far no reaction. The
archive is here [1] but Bertrand tells me only ASF member have access -
for whatever reason.
Michael
[1]
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_70999f9233dac9b416ef9dedc97c0ef196a938c05d6a407b94ba3479-40-253Cusers.infra.apache.org-253E&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6zQLM7Gc0Sv1iwayKOKa4_SFxRIxS478q2gZlAJj4Zw&m=1jQPrKo-XPFNcyWvKgVdJSRBLNVI_DSgmZJd-kr8xnk&s=A4hNPlAaL65ukbkNROftPjO0CZ1GDbggEVvTaWQOpQc&e=
On Fri, Apr 28, 2017 at 2:23 PM, Michael Marth <mmarth@adobe.com<
mailto:mmarth@adobe.com>> wrote:
Dear Infra team,
I am enquiring on behalf of the OpenWhisk project (currently in Incubator)
[1].
We would like to periodically run performance tests on a distributed
environment (OpenWhisk typically runs on more than 1 machine). So we are
basically looking for an ability to spin up/tear down a number of
(virtual)
machines and exclusively use them for a certain amount of time (so that
the
VMs are not shared and the performance test results are comparable over
time).
The order of magnitude would be ~5-10 VMs for 1 hour 3 times a week.
I would like to find out if there is an ASF-supported mechanism to do
that.
For example, can Infra provide such infrastructure? Or is there a cloud
provider (like Azure) that might sponsor such efforts with VMs? Or maybe
there is an established way for commercial companies that are interested
in
an ASF project to sponsor (fund) such tests?
If none of the above exists, then it would also be helpful for us to get
to
know how other projects run such sort of tests.
Thanks a lot!
Michael
[1]
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_b66ab5b438f2db5cdc8c5f5eabece201b4ad090058fa3a9a3bd09d12-40-253Cdev.openwhisk.apache.org-253E&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6zQLM7Gc0Sv1iwayKOKa4_SFxRIxS478q2gZlAJj4Zw&m=1jQPrKo-XPFNcyWvKgVdJSRBLNVI_DSgmZJd-kr8xnk&s=TJS_T4VQna4dEwll7V3IJWgLXKsSi-YbFRUznqgBogk&e=
From: Markus Thömmes <ma...@me.com>>
Reply-To: "dev@openwhisk.apache.org<ma...@openwhisk.apache.org>"
<de...@openwhisk.apache.org>>
Date: Wednesday 26 April 2017 12:59
To: "dev@openwhisk.apache.org<ma...@openwhisk.apache.org>"
<de...@openwhisk.apache.org>>
Subject: Re: Performance tests for OpenWhisk
Hi Michael,
yeah that sounds pretty much spot on. I'd like to have at least 2 VMs with
4+ cores and 8GB memory. One VM would host the management stack while one
would be dedicated to an Invoker only. That way we could assert
single-invoker performance the easiest.
Thanks for helping!
Cheers,
Markus
Am 26. April 2017 um 11:36 schrieb Michael Marth <mmarth@adobe.com<
mailto:mmarth@adobe.com>>:
Markus,
Does what I describe reflect what you are looking for?
If yes, I am happy to ask on infra.
Let me know
Michael
On 26/04/17 07:52, "Bertrand Delacretaz" <bdelacretaz@apache.org<
mailto:bdelacretaz@apache.org>> wrote:
Hi Michael,
On Tue, Apr 25, 2017 at 6:52 PM, Michael Marth <mmarth@adobe.com<
mailto:mmarth@adobe.com>> wrote:
...Maybe our mentors can chime in. Has this been discussed in the ASF
board or so?...
Best would be to ask the ASF infrastructure team via
users@infra.apache.org<ma...@infra.apache.org> - briefly describe
what you need to see what's
possible.
-Bertrand
Re: How to best run non-local tests in ASF (was: Performance tests for
OpenWhisk)
Posted by Vincent S Hou <sh...@us.ibm.com>.
Great thanks to Adobe Inc for the donation to OpenWhisk.
I just touched based with Carlos about how we setup and Jenkins environment for OpenWhisk. I will pick up the Jenkins work for OpenWhisk.
For anyone, who is interested in the Jenkins Pipeline for OpenWhisk. Welcome to help.
Thank you.
Best wishes.
Vincent Hou (侯胜博)
Advisory Software Engineer, OpenWhisk Contributor, Open Technology, IBM Cloud
Notes ID: Vincent S Hou/Raleigh/IBM, E-mail: shou@us.ibm.com,
Phone: +1(919)254-7182
Address: 4205 S Miami Blvd (Cornwallis Drive), Durham, NC 27703, United States
-----Rodric Rabbah <ro...@gmail.com> wrote: -----
To: dev@openwhisk.apache.org
From: Rodric Rabbah <ro...@gmail.com>
Date: 11/07/2018 02:31PM
Subject: Re: How to best run non-local tests in ASF (was: Performance tests for OpenWhisk)
Thank you Adobe Inc and the Adobe team for the donation and making it possible to have infra for the project! And thanks Carlos for also pushing this forward.
-r
> On Nov 7, 2018, at 2:25 PM, Carlos Santana <cs...@gmail.com> wrote:
>
> We got the 3 VMs Woot !!
> it only took 6 months
>
> The INFRA ticket [1] that I opened back in June I was able to close today.
> I ran a quick test from a Jenkins job to check the 3 VMs, I added the
> information in our Wiki page [2]
>
> On behave of the Apache OpenWhisk (Incubating) community I want to thank
> Adobe Inc. for the donation of $1,000 (one thousand) US dollars to the ASF
> to allow us get these VMs dedicated to the project and be able to run
> Playground CICD tests.
>
> [1] https://issues.apache.org/jira/browse/INFRA-16668
> [2]
> https://cwiki.apache.org/confluence/display/OPENWHISK/Jenkins+VMs+for+Playgrounds
Re: How to best run non-local tests in ASF (was: Performance tests
for OpenWhisk)
Posted by Rodric Rabbah <ro...@gmail.com>.
Thank you Adobe Inc and the Adobe team for the donation and making it possible to have infra for the project! And thanks Carlos for also pushing this forward.
-r
> On Nov 7, 2018, at 2:25 PM, Carlos Santana <cs...@gmail.com> wrote:
>
> We got the 3 VMs Woot !!
> it only took 6 months
>
> The INFRA ticket [1] that I opened back in June I was able to close today.
> I ran a quick test from a Jenkins job to check the 3 VMs, I added the
> information in our Wiki page [2]
>
> On behave of the Apache OpenWhisk (Incubating) community I want to thank
> Adobe Inc. for the donation of $1,000 (one thousand) US dollars to the ASF
> to allow us get these VMs dedicated to the project and be able to run
> Playground CICD tests.
>
> [1] https://issues.apache.org/jira/browse/INFRA-16668
> [2]
> https://cwiki.apache.org/confluence/display/OPENWHISK/Jenkins+VMs+for+Playgrounds
Re: How to best run non-local tests in ASF (was: Performance tests
for OpenWhisk)
Posted by Carlos Santana <cs...@gmail.com>.
We got the 3 VMs Woot !!
it only took 6 months
The INFRA ticket [1] that I opened back in June I was able to close today.
I ran a quick test from a Jenkins job to check the 3 VMs, I added the
information in our Wiki page [2]
On behave of the Apache OpenWhisk (Incubating) community I want to thank
Adobe Inc. for the donation of $1,000 (one thousand) US dollars to the ASF
to allow us get these VMs dedicated to the project and be able to run
Playground CICD tests.
[1] https://issues.apache.org/jira/browse/INFRA-16668
[2]
https://cwiki.apache.org/confluence/display/OPENWHISK/Jenkins+VMs+for+Playgrounds
On Tue, Jun 19, 2018 at 8:32 PM Carlos Santana <cs...@gmail.com> wrote:
> Thanks Michael for securing some funding to get us started with this.
>
> With the recent work from Chettam in refactoring our gradle testing tasks,
> and using Travis stages to build in parallel and over come the 50 minute
> limit we used to hit,
> By adding an additional stage we should be able to run all the tests that
> are present in the core repo. But this would not get us to be able to test
> the deployment scripts and run the tests using a distributed environment,
> in Travis we only have 1 VM.
>
> So we are requesting as a start for INFRA to setup 3 VMs in Apache
> Jenkins, this should allow us to setup an OpenWhisk where we can have 1
> node for control plane, and 2 nodes for invokers.
>
> This way we should be able to test:
> - distributed deployment
> - running all tests against a distributed environment
> - HA/Balancing testing (i.e. shooting 1 invoker and the other invoker
> should handle the load)
>
>
> I opened the INFRA ticket today with the specs [1]
> [1] https://issues.apache.org/jira/browse/INFRA-16668
>
> -- Carlos
>
>
> On Mon, Jun 18, 2018 at 10:48 AM Michael Marth <mm...@adobe.com.invalid>
> wrote:
>
>> A quick update on this:
>>
>> Carlos (and others) has been working on the mechanics of getting
>> distributed tests to run on hardware that is managed by Infra. For this
>> purpose we should have a couple of Infra-managed VMs that are dedicated to
>> OpenWhisk. Bertrand has gotten me in touch with the ASF fundraising people
>> to figure out the mechanics of how to transfer a donation directed at this
>> purpose.
>> Will keep you posted as there's progress.
>>
>>
>>
>> On 09.04.18, 11:50, "Bertrand Delacretaz" <bd...@apache.org>
>> wrote:
>>
>> On Sun, Apr 8, 2018 at 3:36 AM, Carlos Santana <cs...@gmail.com>
>> wrote:
>> > ...Michael could you direct the inquiries on the infra list?...
>>
>> I think it makes sense to keep this thread here for now and dig deeper
>> on the (type of) solution that Luciano has been explaining.
>>
>> Note that http://apache.org/foundation/thanks.html now includes a
>> "targeted sponsors" section where companies can be recognized as
>> sponsors by donating cloud services, among other options. That might
>> be an additional incentive for someone to donate computing resources
>> for this purpose.
>>
>> I'm happy to make connections with the ASF's VP of Fundraising if
>> needed to facilitate things.
>>
>> -Bertrand
>>
>>
>>
Re: How to best run non-local tests in ASF (was: Performance tests
for OpenWhisk)
Posted by Carlos Santana <cs...@gmail.com>.
Thanks Michael for securing some funding to get us started with this.
With the recent work from Chettam in refactoring our gradle testing tasks,
and using Travis stages to build in parallel and over come the 50 minute
limit we used to hit,
By adding an additional stage we should be able to run all the tests that
are present in the core repo. But this would not get us to be able to test
the deployment scripts and run the tests using a distributed environment,
in Travis we only have 1 VM.
So we are requesting as a start for INFRA to setup 3 VMs in Apache Jenkins,
this should allow us to setup an OpenWhisk where we can have 1 node for
control plane, and 2 nodes for invokers.
This way we should be able to test:
- distributed deployment
- running all tests against a distributed environment
- HA/Balancing testing (i.e. shooting 1 invoker and the other invoker
should handle the load)
I opened the INFRA ticket today with the specs [1]
[1] https://issues.apache.org/jira/browse/INFRA-16668
-- Carlos
On Mon, Jun 18, 2018 at 10:48 AM Michael Marth <mm...@adobe.com.invalid>
wrote:
> A quick update on this:
>
> Carlos (and others) has been working on the mechanics of getting
> distributed tests to run on hardware that is managed by Infra. For this
> purpose we should have a couple of Infra-managed VMs that are dedicated to
> OpenWhisk. Bertrand has gotten me in touch with the ASF fundraising people
> to figure out the mechanics of how to transfer a donation directed at this
> purpose.
> Will keep you posted as there's progress.
>
>
>
> On 09.04.18, 11:50, "Bertrand Delacretaz" <bd...@apache.org> wrote:
>
> On Sun, Apr 8, 2018 at 3:36 AM, Carlos Santana <cs...@gmail.com>
> wrote:
> > ...Michael could you direct the inquiries on the infra list?...
>
> I think it makes sense to keep this thread here for now and dig deeper
> on the (type of) solution that Luciano has been explaining.
>
> Note that http://apache.org/foundation/thanks.html now includes a
> "targeted sponsors" section where companies can be recognized as
> sponsors by donating cloud services, among other options. That might
> be an additional incentive for someone to donate computing resources
> for this purpose.
>
> I'm happy to make connections with the ASF's VP of Fundraising if
> needed to facilitate things.
>
> -Bertrand
>
>
>
Re: How to best run non-local tests in ASF (was: Performance tests
for OpenWhisk)
Posted by Michael Marth <mm...@adobe.com.INVALID>.
A quick update on this:
Carlos (and others) has been working on the mechanics of getting distributed tests to run on hardware that is managed by Infra. For this purpose we should have a couple of Infra-managed VMs that are dedicated to OpenWhisk. Bertrand has gotten me in touch with the ASF fundraising people to figure out the mechanics of how to transfer a donation directed at this purpose.
Will keep you posted as there's progress.
On 09.04.18, 11:50, "Bertrand Delacretaz" <bd...@apache.org> wrote:
On Sun, Apr 8, 2018 at 3:36 AM, Carlos Santana <cs...@gmail.com> wrote:
> ...Michael could you direct the inquiries on the infra list?...
I think it makes sense to keep this thread here for now and dig deeper
on the (type of) solution that Luciano has been explaining.
Note that http://apache.org/foundation/thanks.html now includes a
"targeted sponsors" section where companies can be recognized as
sponsors by donating cloud services, among other options. That might
be an additional incentive for someone to donate computing resources
for this purpose.
I'm happy to make connections with the ASF's VP of Fundraising if
needed to facilitate things.
-Bertrand
Re: How to best run non-local tests in ASF (was: Performance tests
for OpenWhisk)
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Sun, Apr 8, 2018 at 3:36 AM, Carlos Santana <cs...@gmail.com> wrote:
> ...Michael could you direct the inquiries on the infra list?...
I think it makes sense to keep this thread here for now and dig deeper
on the (type of) solution that Luciano has been explaining.
Note that http://apache.org/foundation/thanks.html now includes a
"targeted sponsors" section where companies can be recognized as
sponsors by donating cloud services, among other options. That might
be an additional incentive for someone to donate computing resources
for this purpose.
I'm happy to make connections with the ASF's VP of Fundraising if
needed to facilitate things.
-Bertrand
Re: How to best run non-local tests in ASF (was: Performance tests
for OpenWhisk)
Posted by Carlos Santana <cs...@gmail.com>.
Sorry Sebastian about that, I thought you were the one I had the discussion
in the infra chat.
I think it’s a good suggestion to follow up in the infra mailing list.
Michael could you direct the inquiries on the infra list?
— Carlos
On Sat, Apr 7, 2018 at 6:47 AM sebb <se...@gmail.com> wrote:
> On 6 April 2018 at 15:49, Carlos Santana <cs...@gmail.com> wrote:
> > Sebastian Bazley from ASF INFRA would be good resource to contact to
> drive
> > this.
>
> I'm not sure why you used my name.
> Although I do some infra-related work, I am not actually part of Infra.
> (nor do I wish to be).
>
> Nor am I familiar with setting up VMs.
>
> Sorry, but this is not something I could take on, nor do I wish to.
>
> Please contact Infra via its user mailing list.
>
> > In the past when I brought this with infra I think I understood from him
> > that companies can donate funds and explicitly state that are for the
> > OpenWhisk project then the INFRA team will allocate VMs to OpenWhisk to
> run
> > load/performance tests, same way that Hadoop project has decimated VMs in
> > Jenkins today.
> >
> > — Carlos
> >
> > On Fri, Apr 6, 2018 at 9:38 AM Michael Marth <mm...@adobe.com.invalid>
> > wrote:
> >>
> >> Hi mentors (and others),
> >>
> >> Had an offline discussion this week in which the question came up how an
> >> ASF project should best go about running
> performance/throughput/scalability
> >> tests – i.e. tests that cannot be run locally and require a repeatable
> >> environment.
> >>
> >> Some options:
> >> * interested companies run the tests on their own infra and publish
> >> results. Pretty lame, especially because typically only that company’s
> >> engineers can access the env and investigate further.
> >> * interested companies donate cash to sponsor compute resources,
> >> committers can run and investigate the tests. Ideal from tech
> perspective,
> >> but I have no idea how that cash would make its way from the ASF to a
> >> particular project.
> >> * maybe a middle-ground: interested party that happens to have a public
> >> cloud offering gives credentials to committers
> >>
> >> I am mainly interested to learn if there are other ASF projects (e.g. in
> >> the Big Data/Hadoop ecosystem) that do something similar. Or if there
> is an
> >> ASF-recommended way to do this. Or else, where I could ask this
> question?
> >>
> >> Thanks!
> >> Michael
> >>
> >> From: Michael Marth <mm...@adobe.com>
> >> Date: Wednesday 3 May 2017 20:57
> >> To: "dev@openwhisk.apache.org" <de...@openwhisk.apache.org>
> >> Subject: Re: Performance tests for OpenWhisk
> >>
> >> Markus,
> >>
> >> Quick update: sent the below to users@infra. So far no reaction. The
> >> archive is here [1] but Bertrand tells me only ASF member have access
> - for
> >> whatever reason.
> >>
> >> Michael
> >>
> >> [1]
> >>
> https://lists.apache.org/thread.html/70999f9233dac9b416ef9dedc97c0ef196a938c05d6a407b94ba3479@%3Cusers.infra.apache.org%3E
> >>
> >>
> >> On Fri, Apr 28, 2017 at 2:23 PM, Michael Marth
> >> <mm...@adobe.com>> wrote:
> >> Dear Infra team,
> >>
> >> I am enquiring on behalf of the OpenWhisk project (currently in
> Incubator)
> >> [1].
> >>
> >> We would like to periodically run performance tests on a distributed
> >> environment (OpenWhisk typically runs on more than 1 machine). So we are
> >> basically looking for an ability to spin up/tear down a number of
> >> (virtual)
> >> machines and exclusively use them for a certain amount of time (so that
> >> the
> >> VMs are not shared and the performance test results are comparable over
> >> time).
> >> The order of magnitude would be ~5-10 VMs for 1 hour 3 times a week.
> >>
> >> I would like to find out if there is an ASF-supported mechanism to do
> >> that.
> >> For example, can Infra provide such infrastructure? Or is there a cloud
> >> provider (like Azure) that might sponsor such efforts with VMs? Or maybe
> >> there is an established way for commercial companies that are interested
> >> in
> >> an ASF project to sponsor (fund) such tests?
> >>
> >> If none of the above exists, then it would also be helpful for us to get
> >> to
> >> know how other projects run such sort of tests.
> >>
> >> Thanks a lot!
> >> Michael
> >>
> >>
> >> [1]
> >>
> >>
> https://lists.apache.org/thread.html/b66ab5b438f2db5cdc8c5f5eabece201b4ad090058fa3a9a3bd09d12@%3Cdev.openwhisk.apache.org%3E
> >>
> >>
> >>
> >>
> >> From: Markus Thömmes <markusthoemmes@me.com<mailto:
> markusthoemmes@me.com>>
> >> Reply-To: "dev@openwhisk.apache.org<ma...@openwhisk.apache.org>"
> >> <de...@openwhisk.apache.org>>
> >> Date: Wednesday 26 April 2017 12:59
> >> To: "dev@openwhisk.apache.org<ma...@openwhisk.apache.org>"
> >> <de...@openwhisk.apache.org>>
> >> Subject: Re: Performance tests for OpenWhisk
> >>
> >> Hi Michael,
> >>
> >> yeah that sounds pretty much spot on. I'd like to have at least 2 VMs
> with
> >> 4+ cores and 8GB memory. One VM would host the management stack while
> one
> >> would be dedicated to an Invoker only. That way we could assert
> >> single-invoker performance the easiest.
> >>
> >> Thanks for helping!
> >>
> >> Cheers,
> >> Markus
> >>
> >> Am 26. April 2017 um 11:36 schrieb Michael Marth
> >> <mm...@adobe.com>>:
> >> Markus,
> >>
> >> Does what I describe reflect what you are looking for?
> >> If yes, I am happy to ask on infra.
> >>
> >> Let me know
> >> Michael
> >>
> >>
> >>
> >> On 26/04/17 07:52, "Bertrand Delacretaz"
> >> <bd...@apache.org>> wrote:
> >>
> >>
> >> Hi Michael,
> >>
> >> On Tue, Apr 25, 2017 at 6:52 PM, Michael Marth
> >> <mm...@adobe.com>> wrote:
> >> ...Maybe our mentors can chime in. Has this been discussed in the ASF
> >> board or so?...
> >>
> >> Best would be to ask the ASF infrastructure team via
> >> users@infra.apache.org<ma...@infra.apache.org> - briefly
> describe
> >> what you need to see what's
> >> possible.
> >>
> >> -Bertrand
>
Re: How to best run non-local tests in ASF (was: Performance tests
for OpenWhisk)
Posted by sebb <se...@gmail.com>.
On 6 April 2018 at 15:49, Carlos Santana <cs...@gmail.com> wrote:
> Sebastian Bazley from ASF INFRA would be good resource to contact to drive
> this.
I'm not sure why you used my name.
Although I do some infra-related work, I am not actually part of Infra.
(nor do I wish to be).
Nor am I familiar with setting up VMs.
Sorry, but this is not something I could take on, nor do I wish to.
Please contact Infra via its user mailing list.
> In the past when I brought this with infra I think I understood from him
> that companies can donate funds and explicitly state that are for the
> OpenWhisk project then the INFRA team will allocate VMs to OpenWhisk to run
> load/performance tests, same way that Hadoop project has decimated VMs in
> Jenkins today.
>
> — Carlos
>
> On Fri, Apr 6, 2018 at 9:38 AM Michael Marth <mm...@adobe.com.invalid>
> wrote:
>>
>> Hi mentors (and others),
>>
>> Had an offline discussion this week in which the question came up how an
>> ASF project should best go about running performance/throughput/scalability
>> tests – i.e. tests that cannot be run locally and require a repeatable
>> environment.
>>
>> Some options:
>> * interested companies run the tests on their own infra and publish
>> results. Pretty lame, especially because typically only that company’s
>> engineers can access the env and investigate further.
>> * interested companies donate cash to sponsor compute resources,
>> committers can run and investigate the tests. Ideal from tech perspective,
>> but I have no idea how that cash would make its way from the ASF to a
>> particular project.
>> * maybe a middle-ground: interested party that happens to have a public
>> cloud offering gives credentials to committers
>>
>> I am mainly interested to learn if there are other ASF projects (e.g. in
>> the Big Data/Hadoop ecosystem) that do something similar. Or if there is an
>> ASF-recommended way to do this. Or else, where I could ask this question?
>>
>> Thanks!
>> Michael
>>
>> From: Michael Marth <mm...@adobe.com>
>> Date: Wednesday 3 May 2017 20:57
>> To: "dev@openwhisk.apache.org" <de...@openwhisk.apache.org>
>> Subject: Re: Performance tests for OpenWhisk
>>
>> Markus,
>>
>> Quick update: sent the below to users@infra. So far no reaction. The
>> archive is here [1] but Bertrand tells me only ASF member have access - for
>> whatever reason.
>>
>> Michael
>>
>> [1]
>> https://lists.apache.org/thread.html/70999f9233dac9b416ef9dedc97c0ef196a938c05d6a407b94ba3479@%3Cusers.infra.apache.org%3E
>>
>>
>> On Fri, Apr 28, 2017 at 2:23 PM, Michael Marth
>> <mm...@adobe.com>> wrote:
>> Dear Infra team,
>>
>> I am enquiring on behalf of the OpenWhisk project (currently in Incubator)
>> [1].
>>
>> We would like to periodically run performance tests on a distributed
>> environment (OpenWhisk typically runs on more than 1 machine). So we are
>> basically looking for an ability to spin up/tear down a number of
>> (virtual)
>> machines and exclusively use them for a certain amount of time (so that
>> the
>> VMs are not shared and the performance test results are comparable over
>> time).
>> The order of magnitude would be ~5-10 VMs for 1 hour 3 times a week.
>>
>> I would like to find out if there is an ASF-supported mechanism to do
>> that.
>> For example, can Infra provide such infrastructure? Or is there a cloud
>> provider (like Azure) that might sponsor such efforts with VMs? Or maybe
>> there is an established way for commercial companies that are interested
>> in
>> an ASF project to sponsor (fund) such tests?
>>
>> If none of the above exists, then it would also be helpful for us to get
>> to
>> know how other projects run such sort of tests.
>>
>> Thanks a lot!
>> Michael
>>
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b66ab5b438f2db5cdc8c5f5eabece201b4ad090058fa3a9a3bd09d12@%3Cdev.openwhisk.apache.org%3E
>>
>>
>>
>>
>> From: Markus Thömmes <ma...@me.com>>
>> Reply-To: "dev@openwhisk.apache.org<ma...@openwhisk.apache.org>"
>> <de...@openwhisk.apache.org>>
>> Date: Wednesday 26 April 2017 12:59
>> To: "dev@openwhisk.apache.org<ma...@openwhisk.apache.org>"
>> <de...@openwhisk.apache.org>>
>> Subject: Re: Performance tests for OpenWhisk
>>
>> Hi Michael,
>>
>> yeah that sounds pretty much spot on. I'd like to have at least 2 VMs with
>> 4+ cores and 8GB memory. One VM would host the management stack while one
>> would be dedicated to an Invoker only. That way we could assert
>> single-invoker performance the easiest.
>>
>> Thanks for helping!
>>
>> Cheers,
>> Markus
>>
>> Am 26. April 2017 um 11:36 schrieb Michael Marth
>> <mm...@adobe.com>>:
>> Markus,
>>
>> Does what I describe reflect what you are looking for?
>> If yes, I am happy to ask on infra.
>>
>> Let me know
>> Michael
>>
>>
>>
>> On 26/04/17 07:52, "Bertrand Delacretaz"
>> <bd...@apache.org>> wrote:
>>
>>
>> Hi Michael,
>>
>> On Tue, Apr 25, 2017 at 6:52 PM, Michael Marth
>> <mm...@adobe.com>> wrote:
>> ...Maybe our mentors can chime in. Has this been discussed in the ASF
>> board or so?...
>>
>> Best would be to ask the ASF infrastructure team via
>> users@infra.apache.org<ma...@infra.apache.org> - briefly describe
>> what you need to see what's
>> possible.
>>
>> -Bertrand
Re: How to best run non-local tests in ASF (was: Performance tests
for OpenWhisk)
Posted by Carlos Santana <cs...@gmail.com>.
Sebastian Bazley from ASF INFRA would be good resource to contact to drive
this.
In the past when I brought this with infra I think I understood from him
that companies can donate funds and explicitly state that are for the
OpenWhisk project then the INFRA team will allocate VMs to OpenWhisk to run
load/performance tests, same way that Hadoop project has decimated VMs in
Jenkins today.
— Carlos
On Fri, Apr 6, 2018 at 9:38 AM Michael Marth <mm...@adobe.com.invalid>
wrote:
> Hi mentors (and others),
>
> Had an offline discussion this week in which the question came up how an
> ASF project should best go about running performance/throughput/scalability
> tests – i.e. tests that cannot be run locally and require a repeatable
> environment.
>
> Some options:
> * interested companies run the tests on their own infra and publish
> results. Pretty lame, especially because typically only that company’s
> engineers can access the env and investigate further.
> * interested companies donate cash to sponsor compute resources,
> committers can run and investigate the tests. Ideal from tech perspective,
> but I have no idea how that cash would make its way from the ASF to a
> particular project.
> * maybe a middle-ground: interested party that happens to have a public
> cloud offering gives credentials to committers
>
> I am mainly interested to learn if there are other ASF projects (e.g. in
> the Big Data/Hadoop ecosystem) that do something similar. Or if there is an
> ASF-recommended way to do this. Or else, where I could ask this question?
>
> Thanks!
> Michael
>
> From: Michael Marth <mm...@adobe.com>
> Date: Wednesday 3 May 2017 20:57
> To: "dev@openwhisk.apache.org" <de...@openwhisk.apache.org>
> Subject: Re: Performance tests for OpenWhisk
>
> Markus,
>
> Quick update: sent the below to users@infra. So far no reaction. The
> archive is here [1] but Bertrand tells me only ASF member have access -
> for whatever reason.
>
> Michael
>
> [1]
> https://lists.apache.org/thread.html/70999f9233dac9b416ef9dedc97c0ef196a938c05d6a407b94ba3479@%3Cusers.infra.apache.org%3E
>
>
> On Fri, Apr 28, 2017 at 2:23 PM, Michael Marth <mmarth@adobe.com<mailto:
> mmarth@adobe.com>> wrote:
> Dear Infra team,
>
> I am enquiring on behalf of the OpenWhisk project (currently in Incubator)
> [1].
>
> We would like to periodically run performance tests on a distributed
> environment (OpenWhisk typically runs on more than 1 machine). So we are
> basically looking for an ability to spin up/tear down a number of (virtual)
> machines and exclusively use them for a certain amount of time (so that the
> VMs are not shared and the performance test results are comparable over
> time).
> The order of magnitude would be ~5-10 VMs for 1 hour 3 times a week.
>
> I would like to find out if there is an ASF-supported mechanism to do that.
> For example, can Infra provide such infrastructure? Or is there a cloud
> provider (like Azure) that might sponsor such efforts with VMs? Or maybe
> there is an established way for commercial companies that are interested in
> an ASF project to sponsor (fund) such tests?
>
> If none of the above exists, then it would also be helpful for us to get to
> know how other projects run such sort of tests.
>
> Thanks a lot!
> Michael
>
>
> [1]
>
> https://lists.apache.org/thread.html/b66ab5b438f2db5cdc8c5f5eabece201b4ad090058fa3a9a3bd09d12@%3Cdev.openwhisk.apache.org%3E
>
>
>
>
> From: Markus Thömmes <ma...@me.com>>
> Reply-To: "dev@openwhisk.apache.org<ma...@openwhisk.apache.org>" <
> dev@openwhisk.apache.org<ma...@openwhisk.apache.org>>
> Date: Wednesday 26 April 2017 12:59
> To: "dev@openwhisk.apache.org<ma...@openwhisk.apache.org>" <
> dev@openwhisk.apache.org<ma...@openwhisk.apache.org>>
> Subject: Re: Performance tests for OpenWhisk
>
> Hi Michael,
>
> yeah that sounds pretty much spot on. I'd like to have at least 2 VMs with
> 4+ cores and 8GB memory. One VM would host the management stack while one
> would be dedicated to an Invoker only. That way we could assert
> single-invoker performance the easiest.
>
> Thanks for helping!
>
> Cheers,
> Markus
>
> Am 26. April 2017 um 11:36 schrieb Michael Marth <mmarth@adobe.com<mailto:
> mmarth@adobe.com>>:
> Markus,
>
> Does what I describe reflect what you are looking for?
> If yes, I am happy to ask on infra.
>
> Let me know
> Michael
>
>
>
> On 26/04/17 07:52, "Bertrand Delacretaz" <bdelacretaz@apache.org<mailto:
> bdelacretaz@apache.org>> wrote:
>
>
> Hi Michael,
>
> On Tue, Apr 25, 2017 at 6:52 PM, Michael Marth <mmarth@adobe.com<mailto:
> mmarth@adobe.com>> wrote:
> ...Maybe our mentors can chime in. Has this been discussed in the ASF
> board or so?...
>
> Best would be to ask the ASF infrastructure team via
> users@infra.apache.org<ma...@infra.apache.org> - briefly describe
> what you need to see what's
> possible.
>
> -Bertrand
>