You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Marcus Sorensen <sh...@gmail.com> on 2013/09/11 08:02:59 UTC
community testing
CloudStack Dev,
I was emailed about some of the testing questions I brought up
over the last few threads, and a few things were pointed out to me
that I think we should try to remedy. Primarily, that the testing
environment is owned by Citrix, the QA team is primarily Citrix-run,
and the testing done is focused on the use models that Citrix
develops.
I've been assured that the test infrastructure is for everyone,
and I'm not at all trying to say that there's a problem with Citrix
focusing their work on their own interests, but I'm not sure that
anyone outside of Citrix really knows how to add their own stuff to
this testing infrastructure (perhaps for lack of trying, I don't
know).
I haven't really put together enough thought to know how to tackle
this, but my gut tells me that we need some sort of community-owned
testing roll-up, where everyone can do their own testing in whatever
infrastructure and submit hourly, daily, weekly results. If my test
fits into the Citrix test infrastructure and I can figure out how to
get it there, great. If not, I can roll my own and integrate it via
some API. For example the SolidFire guys may wan to run automated
regression testing. That probably won't be doable in the Citrix
infrastructure, but they may want to script a daily
git-pull/build/deploy zone/create volume and it seems logical that
we'd want to support it.
Thoughts? Anyone have experience with such things? Can we have a
master/slave scenario with Jenkins? Perhaps the Citrix environment
already supports something like this via Jenkins API?
Re: community testing
Posted by Prasanna Santhanam <ts...@apache.org>.
If a test is meddling with the infrastructure then we have two
options:
1. isolate the test and run it separately when other tests aren't
running
2. prepare a deployment with those nfs storage preconfigured.
It can be difficult to always get the infra right for a test so tests
often will skip themselves if they run in unsuitable infrastructure.
So yes - its never going to be 100% automatable.
On Wed, Sep 11, 2013 at 12:43:32AM -0600, Marcus Sorensen wrote:
> But if the test requires some sort of preconfiguration, what then (e.g.
> test NFS primary storage would need a local or remote NFS configured)? do I
> need to roll my own, or can I touch the existing test infra and do the
> preconfigure?
> On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
>
> > Yes - Once your test goes into the repo, it should get picked in the
> > subsequent
> > run.
> >
> > Jenkins installations from various companies can be combined into a single
> > landing page. Jenkins itself doesn't support master/slave but it does
> > through
> > the gearman plugin. It's something I have tried using with VMs but not with
> > real infra - but it is entirely possible.
> >
> > On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
> > > I think there are jenkins slaves that run the nicera plugins on/at
> > Schuberg
> > > Philis housed infrastructure. The Citrix jenkins nodes also runs as
> > slaves
> > > that connect back to the apache owned/controlled jenkins. No reason why
> > > testing infra need be so consolidated, it just so happens no one is
> > putting
> > > their hardware where their mouth is.
> > >
> > > I also assume if your marvin tests get accepted upstream, they'll be
> > > included in the nightly runs/reports. Prasanna correct me if I'm wrong.
> > >
> > >
> > > On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <shadowsor@gmail.com
> > >wrote:
> > >
> > > > CloudStack Dev,
> > > > I was emailed about some of the testing questions I brought up
> > > > over the last few threads, and a few things were pointed out to me
> > > > that I think we should try to remedy. Primarily, that the testing
> > > > environment is owned by Citrix, the QA team is primarily Citrix-run,
> > > > and the testing done is focused on the use models that Citrix
> > > > develops.
> > > > I've been assured that the test infrastructure is for everyone,
> > > > and I'm not at all trying to say that there's a problem with Citrix
> > > > focusing their work on their own interests, but I'm not sure that
> > > > anyone outside of Citrix really knows how to add their own stuff to
> > > > this testing infrastructure (perhaps for lack of trying, I don't
> > > > know).
> > > > I haven't really put together enough thought to know how to tackle
> > > > this, but my gut tells me that we need some sort of community-owned
> > > > testing roll-up, where everyone can do their own testing in whatever
> > > > infrastructure and submit hourly, daily, weekly results. If my test
> > > > fits into the Citrix test infrastructure and I can figure out how to
> > > > get it there, great. If not, I can roll my own and integrate it via
> > > > some API. For example the SolidFire guys may wan to run automated
> > > > regression testing. That probably won't be doable in the Citrix
> > > > infrastructure, but they may want to script a daily
> > > > git-pull/build/deploy zone/create volume and it seems logical that
> > > > we'd want to support it.
> > > > Thoughts? Anyone have experience with such things? Can we have a
> > > > master/slave scenario with Jenkins? Perhaps the Citrix environment
> > > > already supports something like this via Jenkins API?
> > > >
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
> >
> >
--
Prasanna.,
------------------------
Powered by BigRock.com
RE: community testing
Posted by "Musayev, Ilya" <im...@webmd.net>.
I'm certainly interested.
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Tuesday, September 24, 2013 5:12 AM
> To: dev
> Subject: Re: community testing
>
> due to lack of response, I might add.
>
> On Mon, Sep 23, 2013 at 11:01 PM, Sebastien Goasguen
> <ru...@gmail.com> wrote:
> >
> > On Sep 23, 2013, at 4:22 PM, Animesh Chaturvedi
> <an...@citrix.com> wrote:
> >
> >>
> >>
> >>> -----Original Message-----
> >>> From: sebgoa [mailto:runseb@gmail.com]
> >>> Sent: Wednesday, September 11, 2013 11:58 PM
> >>> To: Prasanna Santhanam
> >>> Cc: Mike Tutkowski; Marcus Sorensen; dev@cloudstack.apache.org;
> >>> Ahmad Emneina
> >>> Subject: Re: community testing
> >>>
> >>>
> >>> While everything happens on the mailing list, maybe we can setup a
> >>> meeting to discuss this. Google hangout ?
> >>> Anyone interested in testing could join.
> >>>
> >>> Mike is west coast I believe, Prasanna is in india. Scheduling might
> >>> be tough but we should try ?
> >>>
> >>> -sebastien
> >>>
> >>>> [Animesh>] Sebastien did this meeting happen? If so can you share
> >>>> the discussion notes
> >>
> >
> > no this was shutdown in another thread.
> >
Re: community testing
Posted by Daan Hoogland <da...@gmail.com>.
due to lack of response, I might add.
On Mon, Sep 23, 2013 at 11:01 PM, Sebastien Goasguen <ru...@gmail.com> wrote:
>
> On Sep 23, 2013, at 4:22 PM, Animesh Chaturvedi <an...@citrix.com> wrote:
>
>>
>>
>>> -----Original Message-----
>>> From: sebgoa [mailto:runseb@gmail.com]
>>> Sent: Wednesday, September 11, 2013 11:58 PM
>>> To: Prasanna Santhanam
>>> Cc: Mike Tutkowski; Marcus Sorensen; dev@cloudstack.apache.org; Ahmad
>>> Emneina
>>> Subject: Re: community testing
>>>
>>>
>>> While everything happens on the mailing list, maybe we can setup a
>>> meeting to discuss this. Google hangout ?
>>> Anyone interested in testing could join.
>>>
>>> Mike is west coast I believe, Prasanna is in india. Scheduling might be
>>> tough but we should try ?
>>>
>>> -sebastien
>>>
>>>> [Animesh>] Sebastien did this meeting happen? If so can you share the discussion notes
>>
>
> no this was shutdown in another thread.
>
Re: community testing
Posted by Sebastien Goasguen <ru...@gmail.com>.
On Sep 23, 2013, at 4:22 PM, Animesh Chaturvedi <an...@citrix.com> wrote:
>
>
>> -----Original Message-----
>> From: sebgoa [mailto:runseb@gmail.com]
>> Sent: Wednesday, September 11, 2013 11:58 PM
>> To: Prasanna Santhanam
>> Cc: Mike Tutkowski; Marcus Sorensen; dev@cloudstack.apache.org; Ahmad
>> Emneina
>> Subject: Re: community testing
>>
>>
>> While everything happens on the mailing list, maybe we can setup a
>> meeting to discuss this. Google hangout ?
>> Anyone interested in testing could join.
>>
>> Mike is west coast I believe, Prasanna is in india. Scheduling might be
>> tough but we should try ?
>>
>> -sebastien
>>
>>> [Animesh>] Sebastien did this meeting happen? If so can you share the discussion notes
>
no this was shutdown in another thread.
RE: community testing
Posted by Animesh Chaturvedi <an...@citrix.com>.
> -----Original Message-----
> From: sebgoa [mailto:runseb@gmail.com]
> Sent: Wednesday, September 11, 2013 11:58 PM
> To: Prasanna Santhanam
> Cc: Mike Tutkowski; Marcus Sorensen; dev@cloudstack.apache.org; Ahmad
> Emneina
> Subject: Re: community testing
>
>
> While everything happens on the mailing list, maybe we can setup a
> meeting to discuss this. Google hangout ?
> Anyone interested in testing could join.
>
> Mike is west coast I believe, Prasanna is in india. Scheduling might be
> tough but we should try ?
>
> -sebastien
>
> >[Animesh>] Sebastien did this meeting happen? If so can you share the discussion notes
Re: community testing
Posted by sebgoa <ru...@gmail.com>.
On Sep 12, 2013, at 7:24 AM, Prasanna Santhanam <ts...@apache.org> wrote:
> Thanks Marcus - I wanted to work through some concrete steps for the infra now
> that we have some idea what we want to do. We should have basic product setup
> and install and a single deployvm test for all possible configurations:
>
> As a start can we work to setup devcloud and devcloud-kvm environments
> for jenkins? Every checkin should trigger a devcloud instance to
> spinup, start a server and run a simple deployvm test. This is done
> per-commit for the simulator but would be good to have it for devcloud
> as well.
>
> Can you help with devcloud-kvm? I can help with any jenkins configurations and
> the tests themselves. If someone would like to bring up a node for devcloud as
> well that would be great. It should help everyone understand how to add
> infrastructure to jenkins for tests.
While everything happens on the mailing list, maybe we can setup a meeting to discuss this. Google hangout ?
Anyone interested in testing could join.
Mike is west coast I believe, Prasanna is in india. Scheduling might be tough but we should try ?
-sebastien
>
> On Wed, Sep 11, 2013 at 11:17:42PM -0600, Mike Tutkowski wrote:
>> Yeah, I would love to get a SolidFire test suite up and running and plugged
>> into the main builds.
>>
>>
>> On Wed, Sep 11, 2013 at 11:15 PM, Marcus Sorensen <sh...@gmail.com>wrote:
>>
>>> I think the test infra as described is great, but I think we're
>>> hurting a little more for basics. For example, we don't need a full
>>> infrastructure with hardware to ensure that the support matrix works.
>>> I could bring up a VM with CentOS and one with Ubuntu, and test NFS,
>>> CLVM, and RBD on each. CLVM just needs a volume group, NFS can be
>>> exported locally, and RBD can run on localhost node (ceph has how-tos
>>> for this to get your feet wet, that also buys us S3 compatible object
>>> storage for testing secondary). Two VMs with maybe 2 cores, 4GB ram
>>> each, and I think we could knock out a big swath of the basic "does it
>>> work on the supported platforms" that we're missing with a very simple
>>> automated testing. We can easily donate that much.
>>>
>>> I agree that third parties would need to plug in their own testing
>>> (solid fire, as you mention). And certainly testing full blown
>>> deployment from the ground up like it sounds like we are doing is
>>> great and necessary, I just want to plug a few holes and add some
>>> basic sanity checking that we seem to keep getting tripped up on.
>>>
>>> On Wed, Sep 11, 2013 at 10:23 PM, Prasanna Santhanam <ts...@apache.org>
>>> wrote:
>>>> As Sebastien said, it's easy to get you the credentials for jenkins.
>>>> Anyone with commit rights can request for an account. In fact one is
>>>> created soon as you commit. I just need to adjust the credentials.
>>>> (We'll move to git based job configurations but later)
>>>>
>>>> Citrix is unable to test various configurations for lack of necessary
>>>> resources. for eg: It would be hard to test something that requires
>>>> hardware resources like Nicira/Midokura/Solidfire. The current testbed
>>>> is also limited in that it only deploys standard zone models. I have
>>>> only one storage node to spare on which NFS is configured.
>>>>
>>>> CloudStack can be deployed and configured in so many ways that I don't
>>>> think a single testbed cycling through all models is going to be
>>>> effective in testing every possible configuration in time. This is why
>>>> I'd like everyone of us to chip-in and use each others resources to
>>>> make the infrastructure better.
>>>>
>>>> The RBD store at least will require sometime for us to bring up. It
>>>> would be best if we could roll a few hosts from different datacenters
>>>> up into jenkins. Object storage backed CS with something like Riak is
>>>> another untested configuration. It is definitely tested within Citrix
>>>> Labs but those testbeds are internal and cannot be exposed to the
>>>> community. We've got corporate IT which wouldn't like that very much :)
>>>>
>>>> Ultimately, I'd want testbeds span across companies contributing to
>>>> cloudstack. I wouldn't want any single company X to hold the resources
>>>> and control allocation for testing even though that is not the case at
>>>> all.
>>>>
>>>> We still need to figure out how securely these deployments can be
>>>> brought into jenkins and who holds keys to the infrastructure. I'm no
>>>> secure conscious sysadmin so I'm hoping for inputs from operators
>>>> deploying cloudstack.
>>>>
>>>> On Wed, Sep 11, 2013 at 11:12:34AM -0600, Marcus Sorensen wrote:
>>>>> Again, I'm not knocking Citrix. If anything, the issue is that they
>>> tend to
>>>>> be so generous and community oriented that it surprises me when I find
>>> out
>>>>> that certain donation is limited to their interests. Its perfectly
>>>>> reasonable, e.g. my own donations are mostly limited to KVM.
>>>>> On Sep 11, 2013 10:52 AM, "Marcus Sorensen" <sh...@gmail.com>
>>> wrote:
>>>>>
>>>>>> I do understand that. The email I received just triggered warning
>>> bells
>>>>>> because it gave me the impression that the QA team as it stands isn't
>>>>>> testing anything that Citrix doesn't care about, regardless of what
>>> the
>>>>>> community has put on the support matrix. This includes even basic
>>> configs
>>>>>> that the community claims to support like KVM on Ubuntu as the 4.1
>>> release
>>>>>> shows, and other things that we may already have infra for but just
>>> haven't
>>>>>> implemented.
>>>>>>
>>>>>> That led me to wondering how much control the community really has
>>> over
>>>>>> testing. Its good to know that we can roll our own nodes up into
>>> Jenkins,
>>>>>> and/or modify tests if the infrastructure is already there. We just
>>> need to
>>>>>> raise awareness as a community that there are still holes in
>>> resources and
>>>>>> a need for donations to provide the minimum testing required for our
>>>>>> support matrix. I think David's email about release requirements is a
>>> good
>>>>>> step.
>>>>>>
>>>>>> If possible I'd like to modify the existing KVM testing to support
>>> testing
>>>>>> NFS, CLVM, and RBD. This can all be done with a single host (that
>>>>>> presumably already exists), we just need to set up the storage on the
>>> host
>>>>>> and add create pool commands and volume create/delete tests. I'll
>>> have to
>>>>>> figure out how to go about getting admin rights on the KVM test hosts
>>> to
>>>>>> configure the storage types or work with someone. If we can't do that
>>> due
>>>>>> to company logistics, I can easily stand up a VM or two to cover all
>>> of the
>>>>>> KVM mgmt/host hypervisor and storage configs if I can figure out how
>>> to
>>>>>> integrate.
>>>>>> On Sep 11, 2013 2:10 AM, "Sebastien Goasguen" <ru...@gmail.com>
>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> On Sep 11, 2013, at 3:02 AM, Prasanna Santhanam <ts...@apache.org>
>>> wrote:
>>>>>>>
>>>>>>>> CloudStack API actions are agnostic of underlying infrastructure
>>> and
>>>>>>>> most cases can fall into such a category as you describe. But
>>> imagine
>>>>>>>> this - I want to test snapshots ..
>>>>>>>>
>>>>>>>> so i take a snapshot and verify if it backedup correctly against a
>>>>>>>> ceph object store, nfs store or iscsi store. that sort of test is
>>>>>>>> going to involve more than just api actions.
>>>>>>>>
>>>>>>>> or say - i want to test multiple shared networks a VM gets deployed
>>>>>>>> into. Do I assume the deployment has multiple shared networks? Can
>>> i
>>>>>>>> add my own network into the deployment?
>>>>>>>>
>>>>>>>> or even - I want to exhaust all the public network IPs and check if
>>>>>>>> the next deployed VM picks up an IP in the new public range I've
>>>>>>>> added. This sort of test assumes that all the necessary networking
>>> is
>>>>>>>> in place and also hurts VM deployments of all tests that run at the
>>>>>>>> same time.
>>>>>>>>
>>>>>>>> It's a difficult balance to strike but we have to begin somewhere.
>>>>>>>> Start with the basic minimum that every infra can run. infra
>>> specifc
>>>>>>>> tests skip if thing are unsuitable, but will run for someone who
>>> wants
>>>>>>>> to test that feature
>>>>>>>
>>>>>>> A small point here to make is that jenkins.cloudstack.org is open to
>>>>>>> anyone.
>>>>>>> Prasanna has created an account for me and I am (slowly) working on
>>>>>>> adding tests for clients including aws.
>>>>>>>
>>>>>>> Anyone could use this jenkins instance, bring in slaves from "home"
>>> and
>>>>>>> setup tests?
>>>>>>>
>>>>>>> Back to the solidfire example, I think Mike could easily contribute
>>> one
>>>>>>> node that has a solidfire storage, then contribute Marvin tests that
>>> would
>>>>>>> run on jenkins.c.o and target his slave specifically. Same for KVM on
>>>>>>> Ubuntu...
>>>>>>>
>>>>>>> -sebastien
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Sep 10, 2013 at 11:53:15PM -0700, Ahmad Emneina wrote:
>>>>>>>>> That's a good question, I'm not sure how preconditions work with
>>>>>>>>> Marvin cases, but I know the tests are run generically. Say I run
>>>>>>>>> copyvolumeToPrimary (not sure this test exists, hypothetical at
>>> the
>>>>>>>>> moment), it gets run against a slew of infrastructure
>>> configurations
>>>>>>>>> using local storage as well as shared (NSF, iscsi, ceph...) back
>>>>>>>>> ends. So just dropping my test into a storage suite should give it
>>>>>>>>> some guarantee its hitting a few different storage back-ends.
>>> That's
>>>>>>>>> how i understand it works today, I'll defer to Prasanna or
>>> Sudha...
>>>>>>>>> Or anyone else that runs tests aggressively to fill in the gaps
>>> and
>>>>>>>>> make corrections.
>>>>>>>>>
>>>>>>>>> Ahmad
>>>>>>>>>
>>>>>>>>> On Sep 10, 2013, at 11:43 PM, Marcus Sorensen <
>>> shadowsor@gmail.com>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> But if the test requires some sort of preconfiguration, what then
>>>>>>> (e.g. test NFS primary storage would need a local or remote NFS
>>>>>>> configured)? do I need to roll my own, or can I touch the existing
>>> test
>>>>>>> infra and do the preconfigure?
>>>>>>>>>>
>>>>>>>>>> On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <ts...@apache.org>
>>>>>>> wrote:
>>>>>>>>>>> Yes - Once your test goes into the repo, it should get picked
>>> in the
>>>>>>> subsequent
>>>>>>>>>>> run.
>>>>>>>>>>>
>>>>>>>>>>> Jenkins installations from various companies can be combined
>>> into a
>>>>>>> single
>>>>>>>>>>> landing page. Jenkins itself doesn't support master/slave but it
>>>>>>> does through
>>>>>>>>>>> the gearman plugin. It's something I have tried using with VMs
>>> but
>>>>>>> not with
>>>>>>>>>>> real infra - but it is entirely possible.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
>>>>>>>>>>>> I think there are jenkins slaves that run the nicera plugins
>>> on/at
>>>>>>> Schuberg
>>>>>>>>>>>> Philis housed infrastructure. The Citrix jenkins nodes also
>>> runs as
>>>>>>> slaves
>>>>>>>>>>>> that connect back to the apache owned/controlled jenkins. No
>>> reason
>>>>>>> why
>>>>>>>>>>>> testing infra need be so consolidated, it just so happens no
>>> one is
>>>>>>> putting
>>>>>>>>>>>> their hardware where their mouth is.
>>>>>>>>>>>>
>>>>>>>>>>>> I also assume if your marvin tests get accepted upstream,
>>> they'll be
>>>>>>>>>>>> included in the nightly runs/reports. Prasanna correct me if
>>> I'm
>>>>>>> wrong.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <
>>>>>>> shadowsor@gmail.com>wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> CloudStack Dev,
>>>>>>>>>>>>> I was emailed about some of the testing questions I
>>> brought up
>>>>>>>>>>>>> over the last few threads, and a few things were pointed out
>>> to me
>>>>>>>>>>>>> that I think we should try to remedy. Primarily, that the
>>> testing
>>>>>>>>>>>>> environment is owned by Citrix, the QA team is primarily
>>>>>>> Citrix-run,
>>>>>>>>>>>>> and the testing done is focused on the use models that Citrix
>>>>>>>>>>>>> develops.
>>>>>>>>>>>>> I've been assured that the test infrastructure is for
>>> everyone,
>>>>>>>>>>>>> and I'm not at all trying to say that there's a problem with
>>> Citrix
>>>>>>>>>>>>> focusing their work on their own interests, but I'm not sure
>>> that
>>>>>>>>>>>>> anyone outside of Citrix really knows how to add their own
>>> stuff to
>>>>>>>>>>>>> this testing infrastructure (perhaps for lack of trying, I
>>> don't
>>>>>>>>>>>>> know).
>>>>>>>>>>>>> I haven't really put together enough thought to know how to
>>>>>>> tackle
>>>>>>>>>>>>> this, but my gut tells me that we need some sort of
>>> community-owned
>>>>>>>>>>>>> testing roll-up, where everyone can do their own testing in
>>>>>>> whatever
>>>>>>>>>>>>> infrastructure and submit hourly, daily, weekly results. If
>>> my test
>>>>>>>>>>>>> fits into the Citrix test infrastructure and I can figure out
>>> how
>>>>>>> to
>>>>>>>>>>>>> get it there, great. If not, I can roll my own and integrate
>>> it via
>>>>>>>>>>>>> some API. For example the SolidFire guys may wan to run
>>> automated
>>>>>>>>>>>>> regression testing. That probably won't be doable in the
>>> Citrix
>>>>>>>>>>>>> infrastructure, but they may want to script a daily
>>>>>>>>>>>>> git-pull/build/deploy zone/create volume and it seems logical
>>> that
>>>>>>>>>>>>> we'd want to support it.
>>>>>>>>>>>>> Thoughts? Anyone have experience with such things? Can we
>>> have a
>>>>>>>>>>>>> master/slave scenario with Jenkins? Perhaps the Citrix
>>> environment
>>>>>>>>>>>>> already supports something like this via Jenkins API?
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Prasanna.,
>>>>>>>>>>>
>>>>>>>>>>> ------------------------
>>>>>>>>>>> Powered by BigRock.com
>>>>>>>>
>>>>>>>> --
>>>>>>>> Prasanna.,
>>>>>>>>
>>>>>>>> ------------------------
>>>>>>>> Powered by BigRock.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>> Prasanna.,
>>>>
>>>> ------------------------
>>>> Powered by BigRock.com
>>>>
>>>
>>
>>
>>
>> --
>> *Mike Tutkowski*
>> *Senior CloudStack Developer, SolidFire Inc.*
>> e: mike.tutkowski@solidfire.com
>> o: 303.746.7302
>> Advancing the way the world uses the
>> cloud<http://solidfire.com/solution/overview/?video=play>
>> *?*
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
Re: community testing
Posted by Prasanna Santhanam <ts...@apache.org>.
Thanks Marcus - I wanted to work through some concrete steps for the infra now
that we have some idea what we want to do. We should have basic product setup
and install and a single deployvm test for all possible configurations:
As a start can we work to setup devcloud and devcloud-kvm environments
for jenkins? Every checkin should trigger a devcloud instance to
spinup, start a server and run a simple deployvm test. This is done
per-commit for the simulator but would be good to have it for devcloud
as well.
Can you help with devcloud-kvm? I can help with any jenkins configurations and
the tests themselves. If someone would like to bring up a node for devcloud as
well that would be great. It should help everyone understand how to add
infrastructure to jenkins for tests.
On Wed, Sep 11, 2013 at 11:17:42PM -0600, Mike Tutkowski wrote:
> Yeah, I would love to get a SolidFire test suite up and running and plugged
> into the main builds.
>
>
> On Wed, Sep 11, 2013 at 11:15 PM, Marcus Sorensen <sh...@gmail.com>wrote:
>
> > I think the test infra as described is great, but I think we're
> > hurting a little more for basics. For example, we don't need a full
> > infrastructure with hardware to ensure that the support matrix works.
> > I could bring up a VM with CentOS and one with Ubuntu, and test NFS,
> > CLVM, and RBD on each. CLVM just needs a volume group, NFS can be
> > exported locally, and RBD can run on localhost node (ceph has how-tos
> > for this to get your feet wet, that also buys us S3 compatible object
> > storage for testing secondary). Two VMs with maybe 2 cores, 4GB ram
> > each, and I think we could knock out a big swath of the basic "does it
> > work on the supported platforms" that we're missing with a very simple
> > automated testing. We can easily donate that much.
> >
> > I agree that third parties would need to plug in their own testing
> > (solid fire, as you mention). And certainly testing full blown
> > deployment from the ground up like it sounds like we are doing is
> > great and necessary, I just want to plug a few holes and add some
> > basic sanity checking that we seem to keep getting tripped up on.
> >
> > On Wed, Sep 11, 2013 at 10:23 PM, Prasanna Santhanam <ts...@apache.org>
> > wrote:
> > > As Sebastien said, it's easy to get you the credentials for jenkins.
> > > Anyone with commit rights can request for an account. In fact one is
> > > created soon as you commit. I just need to adjust the credentials.
> > > (We'll move to git based job configurations but later)
> > >
> > > Citrix is unable to test various configurations for lack of necessary
> > > resources. for eg: It would be hard to test something that requires
> > > hardware resources like Nicira/Midokura/Solidfire. The current testbed
> > > is also limited in that it only deploys standard zone models. I have
> > > only one storage node to spare on which NFS is configured.
> > >
> > > CloudStack can be deployed and configured in so many ways that I don't
> > > think a single testbed cycling through all models is going to be
> > > effective in testing every possible configuration in time. This is why
> > > I'd like everyone of us to chip-in and use each others resources to
> > > make the infrastructure better.
> > >
> > > The RBD store at least will require sometime for us to bring up. It
> > > would be best if we could roll a few hosts from different datacenters
> > > up into jenkins. Object storage backed CS with something like Riak is
> > > another untested configuration. It is definitely tested within Citrix
> > > Labs but those testbeds are internal and cannot be exposed to the
> > > community. We've got corporate IT which wouldn't like that very much :)
> > >
> > > Ultimately, I'd want testbeds span across companies contributing to
> > > cloudstack. I wouldn't want any single company X to hold the resources
> > > and control allocation for testing even though that is not the case at
> > > all.
> > >
> > > We still need to figure out how securely these deployments can be
> > > brought into jenkins and who holds keys to the infrastructure. I'm no
> > > secure conscious sysadmin so I'm hoping for inputs from operators
> > > deploying cloudstack.
> > >
> > > On Wed, Sep 11, 2013 at 11:12:34AM -0600, Marcus Sorensen wrote:
> > >> Again, I'm not knocking Citrix. If anything, the issue is that they
> > tend to
> > >> be so generous and community oriented that it surprises me when I find
> > out
> > >> that certain donation is limited to their interests. Its perfectly
> > >> reasonable, e.g. my own donations are mostly limited to KVM.
> > >> On Sep 11, 2013 10:52 AM, "Marcus Sorensen" <sh...@gmail.com>
> > wrote:
> > >>
> > >> > I do understand that. The email I received just triggered warning
> > bells
> > >> > because it gave me the impression that the QA team as it stands isn't
> > >> > testing anything that Citrix doesn't care about, regardless of what
> > the
> > >> > community has put on the support matrix. This includes even basic
> > configs
> > >> > that the community claims to support like KVM on Ubuntu as the 4.1
> > release
> > >> > shows, and other things that we may already have infra for but just
> > haven't
> > >> > implemented.
> > >> >
> > >> > That led me to wondering how much control the community really has
> > over
> > >> > testing. Its good to know that we can roll our own nodes up into
> > Jenkins,
> > >> > and/or modify tests if the infrastructure is already there. We just
> > need to
> > >> > raise awareness as a community that there are still holes in
> > resources and
> > >> > a need for donations to provide the minimum testing required for our
> > >> > support matrix. I think David's email about release requirements is a
> > good
> > >> > step.
> > >> >
> > >> > If possible I'd like to modify the existing KVM testing to support
> > testing
> > >> > NFS, CLVM, and RBD. This can all be done with a single host (that
> > >> > presumably already exists), we just need to set up the storage on the
> > host
> > >> > and add create pool commands and volume create/delete tests. I'll
> > have to
> > >> > figure out how to go about getting admin rights on the KVM test hosts
> > to
> > >> > configure the storage types or work with someone. If we can't do that
> > due
> > >> > to company logistics, I can easily stand up a VM or two to cover all
> > of the
> > >> > KVM mgmt/host hypervisor and storage configs if I can figure out how
> > to
> > >> > integrate.
> > >> > On Sep 11, 2013 2:10 AM, "Sebastien Goasguen" <ru...@gmail.com>
> > wrote:
> > >> >
> > >> >>
> > >> >> On Sep 11, 2013, at 3:02 AM, Prasanna Santhanam <ts...@apache.org>
> > wrote:
> > >> >>
> > >> >> > CloudStack API actions are agnostic of underlying infrastructure
> > and
> > >> >> > most cases can fall into such a category as you describe. But
> > imagine
> > >> >> > this - I want to test snapshots ..
> > >> >> >
> > >> >> > so i take a snapshot and verify if it backedup correctly against a
> > >> >> > ceph object store, nfs store or iscsi store. that sort of test is
> > >> >> > going to involve more than just api actions.
> > >> >> >
> > >> >> > or say - i want to test multiple shared networks a VM gets deployed
> > >> >> > into. Do I assume the deployment has multiple shared networks? Can
> > i
> > >> >> > add my own network into the deployment?
> > >> >> >
> > >> >> > or even - I want to exhaust all the public network IPs and check if
> > >> >> > the next deployed VM picks up an IP in the new public range I've
> > >> >> > added. This sort of test assumes that all the necessary networking
> > is
> > >> >> > in place and also hurts VM deployments of all tests that run at the
> > >> >> > same time.
> > >> >> >
> > >> >> > It's a difficult balance to strike but we have to begin somewhere.
> > >> >> > Start with the basic minimum that every infra can run. infra
> > specifc
> > >> >> > tests skip if thing are unsuitable, but will run for someone who
> > wants
> > >> >> > to test that feature
> > >> >>
> > >> >> A small point here to make is that jenkins.cloudstack.org is open to
> > >> >> anyone.
> > >> >> Prasanna has created an account for me and I am (slowly) working on
> > >> >> adding tests for clients including aws.
> > >> >>
> > >> >> Anyone could use this jenkins instance, bring in slaves from "home"
> > and
> > >> >> setup tests?
> > >> >>
> > >> >> Back to the solidfire example, I think Mike could easily contribute
> > one
> > >> >> node that has a solidfire storage, then contribute Marvin tests that
> > would
> > >> >> run on jenkins.c.o and target his slave specifically. Same for KVM on
> > >> >> Ubuntu...
> > >> >>
> > >> >> -sebastien
> > >> >>
> > >> >>
> > >> >> >
> > >> >> > On Tue, Sep 10, 2013 at 11:53:15PM -0700, Ahmad Emneina wrote:
> > >> >> >> That's a good question, I'm not sure how preconditions work with
> > >> >> >> Marvin cases, but I know the tests are run generically. Say I run
> > >> >> >> copyvolumeToPrimary (not sure this test exists, hypothetical at
> > the
> > >> >> >> moment), it gets run against a slew of infrastructure
> > configurations
> > >> >> >> using local storage as well as shared (NSF, iscsi, ceph...) back
> > >> >> >> ends. So just dropping my test into a storage suite should give it
> > >> >> >> some guarantee its hitting a few different storage back-ends.
> > That's
> > >> >> >> how i understand it works today, I'll defer to Prasanna or
> > Sudha...
> > >> >> >> Or anyone else that runs tests aggressively to fill in the gaps
> > and
> > >> >> >> make corrections.
> > >> >> >>
> > >> >> >> Ahmad
> > >> >> >>
> > >> >> >> On Sep 10, 2013, at 11:43 PM, Marcus Sorensen <
> > shadowsor@gmail.com>
> > >> >> wrote:
> > >> >> >>
> > >> >> >>> But if the test requires some sort of preconfiguration, what then
> > >> >> (e.g. test NFS primary storage would need a local or remote NFS
> > >> >> configured)? do I need to roll my own, or can I touch the existing
> > test
> > >> >> infra and do the preconfigure?
> > >> >> >>>
> > >> >> >>> On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <ts...@apache.org>
> > >> >> wrote:
> > >> >> >>>> Yes - Once your test goes into the repo, it should get picked
> > in the
> > >> >> subsequent
> > >> >> >>>> run.
> > >> >> >>>>
> > >> >> >>>> Jenkins installations from various companies can be combined
> > into a
> > >> >> single
> > >> >> >>>> landing page. Jenkins itself doesn't support master/slave but it
> > >> >> does through
> > >> >> >>>> the gearman plugin. It's something I have tried using with VMs
> > but
> > >> >> not with
> > >> >> >>>> real infra - but it is entirely possible.
> > >> >> >>>>
> > >> >> >>>> On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
> > >> >> >>>>> I think there are jenkins slaves that run the nicera plugins
> > on/at
> > >> >> Schuberg
> > >> >> >>>>> Philis housed infrastructure. The Citrix jenkins nodes also
> > runs as
> > >> >> slaves
> > >> >> >>>>> that connect back to the apache owned/controlled jenkins. No
> > reason
> > >> >> why
> > >> >> >>>>> testing infra need be so consolidated, it just so happens no
> > one is
> > >> >> putting
> > >> >> >>>>> their hardware where their mouth is.
> > >> >> >>>>>
> > >> >> >>>>> I also assume if your marvin tests get accepted upstream,
> > they'll be
> > >> >> >>>>> included in the nightly runs/reports. Prasanna correct me if
> > I'm
> > >> >> wrong.
> > >> >> >>>>>
> > >> >> >>>>>
> > >> >> >>>>> On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <
> > >> >> shadowsor@gmail.com>wrote:
> > >> >> >>>>>
> > >> >> >>>>>> CloudStack Dev,
> > >> >> >>>>>> I was emailed about some of the testing questions I
> > brought up
> > >> >> >>>>>> over the last few threads, and a few things were pointed out
> > to me
> > >> >> >>>>>> that I think we should try to remedy. Primarily, that the
> > testing
> > >> >> >>>>>> environment is owned by Citrix, the QA team is primarily
> > >> >> Citrix-run,
> > >> >> >>>>>> and the testing done is focused on the use models that Citrix
> > >> >> >>>>>> develops.
> > >> >> >>>>>> I've been assured that the test infrastructure is for
> > everyone,
> > >> >> >>>>>> and I'm not at all trying to say that there's a problem with
> > Citrix
> > >> >> >>>>>> focusing their work on their own interests, but I'm not sure
> > that
> > >> >> >>>>>> anyone outside of Citrix really knows how to add their own
> > stuff to
> > >> >> >>>>>> this testing infrastructure (perhaps for lack of trying, I
> > don't
> > >> >> >>>>>> know).
> > >> >> >>>>>> I haven't really put together enough thought to know how to
> > >> >> tackle
> > >> >> >>>>>> this, but my gut tells me that we need some sort of
> > community-owned
> > >> >> >>>>>> testing roll-up, where everyone can do their own testing in
> > >> >> whatever
> > >> >> >>>>>> infrastructure and submit hourly, daily, weekly results. If
> > my test
> > >> >> >>>>>> fits into the Citrix test infrastructure and I can figure out
> > how
> > >> >> to
> > >> >> >>>>>> get it there, great. If not, I can roll my own and integrate
> > it via
> > >> >> >>>>>> some API. For example the SolidFire guys may wan to run
> > automated
> > >> >> >>>>>> regression testing. That probably won't be doable in the
> > Citrix
> > >> >> >>>>>> infrastructure, but they may want to script a daily
> > >> >> >>>>>> git-pull/build/deploy zone/create volume and it seems logical
> > that
> > >> >> >>>>>> we'd want to support it.
> > >> >> >>>>>> Thoughts? Anyone have experience with such things? Can we
> > have a
> > >> >> >>>>>> master/slave scenario with Jenkins? Perhaps the Citrix
> > environment
> > >> >> >>>>>> already supports something like this via Jenkins API?
> > >> >> >>>>>>
> > >> >> >>>>
> > >> >> >>>> --
> > >> >> >>>> Prasanna.,
> > >> >> >>>>
> > >> >> >>>> ------------------------
> > >> >> >>>> Powered by BigRock.com
> > >> >> >
> > >> >> > --
> > >> >> > Prasanna.,
> > >> >> >
> > >> >> > ------------------------
> > >> >> > Powered by BigRock.com
> > >> >> >
> > >> >>
> > >> >>
> > >
> > > --
> > > Prasanna.,
> > >
> > > ------------------------
> > > Powered by BigRock.com
> > >
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud<http://solidfire.com/solution/overview/?video=play>
> *?*
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: community testing
Posted by Marcus Sorensen <sh...@gmail.com>.
I guess I was unaware that the test infrastructure tests various
deployment configs from the ground up, rebuilding from baremetal (I
guess?). I was initially thinking "We already test KVM, let me just
run two commands on the host to set up a volume group, then tweak the
marvin test to register it as a primary storage and put a loop in to
do volume tests for each registered primary storage". The test infra
sounds much more advanced than what I was thinking, and probably
harder to adjust. At any rate, I do think that having a devcloud-type
basic sanity check for our support matrix would be relatively little
work and provide a lot of benefit in catching some of these big holes
in testing prior to creating an RC.
On Wed, Sep 11, 2013 at 11:15 PM, Marcus Sorensen <sh...@gmail.com> wrote:
> I think the test infra as described is great, but I think we're
> hurting a little more for basics. For example, we don't need a full
> infrastructure with hardware to ensure that the support matrix works.
> I could bring up a VM with CentOS and one with Ubuntu, and test NFS,
> CLVM, and RBD on each. CLVM just needs a volume group, NFS can be
> exported locally, and RBD can run on localhost node (ceph has how-tos
> for this to get your feet wet, that also buys us S3 compatible object
> storage for testing secondary). Two VMs with maybe 2 cores, 4GB ram
> each, and I think we could knock out a big swath of the basic "does it
> work on the supported platforms" that we're missing with a very simple
> automated testing. We can easily donate that much.
>
> I agree that third parties would need to plug in their own testing
> (solid fire, as you mention). And certainly testing full blown
> deployment from the ground up like it sounds like we are doing is
> great and necessary, I just want to plug a few holes and add some
> basic sanity checking that we seem to keep getting tripped up on.
>
> On Wed, Sep 11, 2013 at 10:23 PM, Prasanna Santhanam <ts...@apache.org> wrote:
>> As Sebastien said, it's easy to get you the credentials for jenkins.
>> Anyone with commit rights can request for an account. In fact one is
>> created soon as you commit. I just need to adjust the credentials.
>> (We'll move to git based job configurations but later)
>>
>> Citrix is unable to test various configurations for lack of necessary
>> resources. for eg: It would be hard to test something that requires
>> hardware resources like Nicira/Midokura/Solidfire. The current testbed
>> is also limited in that it only deploys standard zone models. I have
>> only one storage node to spare on which NFS is configured.
>>
>> CloudStack can be deployed and configured in so many ways that I don't
>> think a single testbed cycling through all models is going to be
>> effective in testing every possible configuration in time. This is why
>> I'd like everyone of us to chip-in and use each others resources to
>> make the infrastructure better.
>>
>> The RBD store at least will require sometime for us to bring up. It
>> would be best if we could roll a few hosts from different datacenters
>> up into jenkins. Object storage backed CS with something like Riak is
>> another untested configuration. It is definitely tested within Citrix
>> Labs but those testbeds are internal and cannot be exposed to the
>> community. We've got corporate IT which wouldn't like that very much :)
>>
>> Ultimately, I'd want testbeds span across companies contributing to
>> cloudstack. I wouldn't want any single company X to hold the resources
>> and control allocation for testing even though that is not the case at
>> all.
>>
>> We still need to figure out how securely these deployments can be
>> brought into jenkins and who holds keys to the infrastructure. I'm no
>> secure conscious sysadmin so I'm hoping for inputs from operators
>> deploying cloudstack.
>>
>> On Wed, Sep 11, 2013 at 11:12:34AM -0600, Marcus Sorensen wrote:
>>> Again, I'm not knocking Citrix. If anything, the issue is that they tend to
>>> be so generous and community oriented that it surprises me when I find out
>>> that certain donation is limited to their interests. Its perfectly
>>> reasonable, e.g. my own donations are mostly limited to KVM.
>>> On Sep 11, 2013 10:52 AM, "Marcus Sorensen" <sh...@gmail.com> wrote:
>>>
>>> > I do understand that. The email I received just triggered warning bells
>>> > because it gave me the impression that the QA team as it stands isn't
>>> > testing anything that Citrix doesn't care about, regardless of what the
>>> > community has put on the support matrix. This includes even basic configs
>>> > that the community claims to support like KVM on Ubuntu as the 4.1 release
>>> > shows, and other things that we may already have infra for but just haven't
>>> > implemented.
>>> >
>>> > That led me to wondering how much control the community really has over
>>> > testing. Its good to know that we can roll our own nodes up into Jenkins,
>>> > and/or modify tests if the infrastructure is already there. We just need to
>>> > raise awareness as a community that there are still holes in resources and
>>> > a need for donations to provide the minimum testing required for our
>>> > support matrix. I think David's email about release requirements is a good
>>> > step.
>>> >
>>> > If possible I'd like to modify the existing KVM testing to support testing
>>> > NFS, CLVM, and RBD. This can all be done with a single host (that
>>> > presumably already exists), we just need to set up the storage on the host
>>> > and add create pool commands and volume create/delete tests. I'll have to
>>> > figure out how to go about getting admin rights on the KVM test hosts to
>>> > configure the storage types or work with someone. If we can't do that due
>>> > to company logistics, I can easily stand up a VM or two to cover all of the
>>> > KVM mgmt/host hypervisor and storage configs if I can figure out how to
>>> > integrate.
>>> > On Sep 11, 2013 2:10 AM, "Sebastien Goasguen" <ru...@gmail.com> wrote:
>>> >
>>> >>
>>> >> On Sep 11, 2013, at 3:02 AM, Prasanna Santhanam <ts...@apache.org> wrote:
>>> >>
>>> >> > CloudStack API actions are agnostic of underlying infrastructure and
>>> >> > most cases can fall into such a category as you describe. But imagine
>>> >> > this - I want to test snapshots ..
>>> >> >
>>> >> > so i take a snapshot and verify if it backedup correctly against a
>>> >> > ceph object store, nfs store or iscsi store. that sort of test is
>>> >> > going to involve more than just api actions.
>>> >> >
>>> >> > or say - i want to test multiple shared networks a VM gets deployed
>>> >> > into. Do I assume the deployment has multiple shared networks? Can i
>>> >> > add my own network into the deployment?
>>> >> >
>>> >> > or even - I want to exhaust all the public network IPs and check if
>>> >> > the next deployed VM picks up an IP in the new public range I've
>>> >> > added. This sort of test assumes that all the necessary networking is
>>> >> > in place and also hurts VM deployments of all tests that run at the
>>> >> > same time.
>>> >> >
>>> >> > It's a difficult balance to strike but we have to begin somewhere.
>>> >> > Start with the basic minimum that every infra can run. infra specifc
>>> >> > tests skip if thing are unsuitable, but will run for someone who wants
>>> >> > to test that feature
>>> >>
>>> >> A small point here to make is that jenkins.cloudstack.org is open to
>>> >> anyone.
>>> >> Prasanna has created an account for me and I am (slowly) working on
>>> >> adding tests for clients including aws.
>>> >>
>>> >> Anyone could use this jenkins instance, bring in slaves from "home" and
>>> >> setup tests?
>>> >>
>>> >> Back to the solidfire example, I think Mike could easily contribute one
>>> >> node that has a solidfire storage, then contribute Marvin tests that would
>>> >> run on jenkins.c.o and target his slave specifically. Same for KVM on
>>> >> Ubuntu...
>>> >>
>>> >> -sebastien
>>> >>
>>> >>
>>> >> >
>>> >> > On Tue, Sep 10, 2013 at 11:53:15PM -0700, Ahmad Emneina wrote:
>>> >> >> That's a good question, I'm not sure how preconditions work with
>>> >> >> Marvin cases, but I know the tests are run generically. Say I run
>>> >> >> copyvolumeToPrimary (not sure this test exists, hypothetical at the
>>> >> >> moment), it gets run against a slew of infrastructure configurations
>>> >> >> using local storage as well as shared (NSF, iscsi, ceph...) back
>>> >> >> ends. So just dropping my test into a storage suite should give it
>>> >> >> some guarantee its hitting a few different storage back-ends. That's
>>> >> >> how i understand it works today, I'll defer to Prasanna or Sudha...
>>> >> >> Or anyone else that runs tests aggressively to fill in the gaps and
>>> >> >> make corrections.
>>> >> >>
>>> >> >> Ahmad
>>> >> >>
>>> >> >> On Sep 10, 2013, at 11:43 PM, Marcus Sorensen <sh...@gmail.com>
>>> >> wrote:
>>> >> >>
>>> >> >>> But if the test requires some sort of preconfiguration, what then
>>> >> (e.g. test NFS primary storage would need a local or remote NFS
>>> >> configured)? do I need to roll my own, or can I touch the existing test
>>> >> infra and do the preconfigure?
>>> >> >>>
>>> >> >>> On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <ts...@apache.org>
>>> >> wrote:
>>> >> >>>> Yes - Once your test goes into the repo, it should get picked in the
>>> >> subsequent
>>> >> >>>> run.
>>> >> >>>>
>>> >> >>>> Jenkins installations from various companies can be combined into a
>>> >> single
>>> >> >>>> landing page. Jenkins itself doesn't support master/slave but it
>>> >> does through
>>> >> >>>> the gearman plugin. It's something I have tried using with VMs but
>>> >> not with
>>> >> >>>> real infra - but it is entirely possible.
>>> >> >>>>
>>> >> >>>> On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
>>> >> >>>>> I think there are jenkins slaves that run the nicera plugins on/at
>>> >> Schuberg
>>> >> >>>>> Philis housed infrastructure. The Citrix jenkins nodes also runs as
>>> >> slaves
>>> >> >>>>> that connect back to the apache owned/controlled jenkins. No reason
>>> >> why
>>> >> >>>>> testing infra need be so consolidated, it just so happens no one is
>>> >> putting
>>> >> >>>>> their hardware where their mouth is.
>>> >> >>>>>
>>> >> >>>>> I also assume if your marvin tests get accepted upstream, they'll be
>>> >> >>>>> included in the nightly runs/reports. Prasanna correct me if I'm
>>> >> wrong.
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <
>>> >> shadowsor@gmail.com>wrote:
>>> >> >>>>>
>>> >> >>>>>> CloudStack Dev,
>>> >> >>>>>> I was emailed about some of the testing questions I brought up
>>> >> >>>>>> over the last few threads, and a few things were pointed out to me
>>> >> >>>>>> that I think we should try to remedy. Primarily, that the testing
>>> >> >>>>>> environment is owned by Citrix, the QA team is primarily
>>> >> Citrix-run,
>>> >> >>>>>> and the testing done is focused on the use models that Citrix
>>> >> >>>>>> develops.
>>> >> >>>>>> I've been assured that the test infrastructure is for everyone,
>>> >> >>>>>> and I'm not at all trying to say that there's a problem with Citrix
>>> >> >>>>>> focusing their work on their own interests, but I'm not sure that
>>> >> >>>>>> anyone outside of Citrix really knows how to add their own stuff to
>>> >> >>>>>> this testing infrastructure (perhaps for lack of trying, I don't
>>> >> >>>>>> know).
>>> >> >>>>>> I haven't really put together enough thought to know how to
>>> >> tackle
>>> >> >>>>>> this, but my gut tells me that we need some sort of community-owned
>>> >> >>>>>> testing roll-up, where everyone can do their own testing in
>>> >> whatever
>>> >> >>>>>> infrastructure and submit hourly, daily, weekly results. If my test
>>> >> >>>>>> fits into the Citrix test infrastructure and I can figure out how
>>> >> to
>>> >> >>>>>> get it there, great. If not, I can roll my own and integrate it via
>>> >> >>>>>> some API. For example the SolidFire guys may wan to run automated
>>> >> >>>>>> regression testing. That probably won't be doable in the Citrix
>>> >> >>>>>> infrastructure, but they may want to script a daily
>>> >> >>>>>> git-pull/build/deploy zone/create volume and it seems logical that
>>> >> >>>>>> we'd want to support it.
>>> >> >>>>>> Thoughts? Anyone have experience with such things? Can we have a
>>> >> >>>>>> master/slave scenario with Jenkins? Perhaps the Citrix environment
>>> >> >>>>>> already supports something like this via Jenkins API?
>>> >> >>>>>>
>>> >> >>>>
>>> >> >>>> --
>>> >> >>>> Prasanna.,
>>> >> >>>>
>>> >> >>>> ------------------------
>>> >> >>>> Powered by BigRock.com
>>> >> >
>>> >> > --
>>> >> > Prasanna.,
>>> >> >
>>> >> > ------------------------
>>> >> > Powered by BigRock.com
>>> >> >
>>> >>
>>> >>
>>
>> --
>> Prasanna.,
>>
>> ------------------------
>> Powered by BigRock.com
>>
Re: community testing
Posted by Mike Tutkowski <mi...@solidfire.com>.
Yeah, I would love to get a SolidFire test suite up and running and plugged
into the main builds.
On Wed, Sep 11, 2013 at 11:15 PM, Marcus Sorensen <sh...@gmail.com>wrote:
> I think the test infra as described is great, but I think we're
> hurting a little more for basics. For example, we don't need a full
> infrastructure with hardware to ensure that the support matrix works.
> I could bring up a VM with CentOS and one with Ubuntu, and test NFS,
> CLVM, and RBD on each. CLVM just needs a volume group, NFS can be
> exported locally, and RBD can run on localhost node (ceph has how-tos
> for this to get your feet wet, that also buys us S3 compatible object
> storage for testing secondary). Two VMs with maybe 2 cores, 4GB ram
> each, and I think we could knock out a big swath of the basic "does it
> work on the supported platforms" that we're missing with a very simple
> automated testing. We can easily donate that much.
>
> I agree that third parties would need to plug in their own testing
> (solid fire, as you mention). And certainly testing full blown
> deployment from the ground up like it sounds like we are doing is
> great and necessary, I just want to plug a few holes and add some
> basic sanity checking that we seem to keep getting tripped up on.
>
> On Wed, Sep 11, 2013 at 10:23 PM, Prasanna Santhanam <ts...@apache.org>
> wrote:
> > As Sebastien said, it's easy to get you the credentials for jenkins.
> > Anyone with commit rights can request for an account. In fact one is
> > created soon as you commit. I just need to adjust the credentials.
> > (We'll move to git based job configurations but later)
> >
> > Citrix is unable to test various configurations for lack of necessary
> > resources. for eg: It would be hard to test something that requires
> > hardware resources like Nicira/Midokura/Solidfire. The current testbed
> > is also limited in that it only deploys standard zone models. I have
> > only one storage node to spare on which NFS is configured.
> >
> > CloudStack can be deployed and configured in so many ways that I don't
> > think a single testbed cycling through all models is going to be
> > effective in testing every possible configuration in time. This is why
> > I'd like everyone of us to chip-in and use each others resources to
> > make the infrastructure better.
> >
> > The RBD store at least will require sometime for us to bring up. It
> > would be best if we could roll a few hosts from different datacenters
> > up into jenkins. Object storage backed CS with something like Riak is
> > another untested configuration. It is definitely tested within Citrix
> > Labs but those testbeds are internal and cannot be exposed to the
> > community. We've got corporate IT which wouldn't like that very much :)
> >
> > Ultimately, I'd want testbeds span across companies contributing to
> > cloudstack. I wouldn't want any single company X to hold the resources
> > and control allocation for testing even though that is not the case at
> > all.
> >
> > We still need to figure out how securely these deployments can be
> > brought into jenkins and who holds keys to the infrastructure. I'm no
> > secure conscious sysadmin so I'm hoping for inputs from operators
> > deploying cloudstack.
> >
> > On Wed, Sep 11, 2013 at 11:12:34AM -0600, Marcus Sorensen wrote:
> >> Again, I'm not knocking Citrix. If anything, the issue is that they
> tend to
> >> be so generous and community oriented that it surprises me when I find
> out
> >> that certain donation is limited to their interests. Its perfectly
> >> reasonable, e.g. my own donations are mostly limited to KVM.
> >> On Sep 11, 2013 10:52 AM, "Marcus Sorensen" <sh...@gmail.com>
> wrote:
> >>
> >> > I do understand that. The email I received just triggered warning
> bells
> >> > because it gave me the impression that the QA team as it stands isn't
> >> > testing anything that Citrix doesn't care about, regardless of what
> the
> >> > community has put on the support matrix. This includes even basic
> configs
> >> > that the community claims to support like KVM on Ubuntu as the 4.1
> release
> >> > shows, and other things that we may already have infra for but just
> haven't
> >> > implemented.
> >> >
> >> > That led me to wondering how much control the community really has
> over
> >> > testing. Its good to know that we can roll our own nodes up into
> Jenkins,
> >> > and/or modify tests if the infrastructure is already there. We just
> need to
> >> > raise awareness as a community that there are still holes in
> resources and
> >> > a need for donations to provide the minimum testing required for our
> >> > support matrix. I think David's email about release requirements is a
> good
> >> > step.
> >> >
> >> > If possible I'd like to modify the existing KVM testing to support
> testing
> >> > NFS, CLVM, and RBD. This can all be done with a single host (that
> >> > presumably already exists), we just need to set up the storage on the
> host
> >> > and add create pool commands and volume create/delete tests. I'll
> have to
> >> > figure out how to go about getting admin rights on the KVM test hosts
> to
> >> > configure the storage types or work with someone. If we can't do that
> due
> >> > to company logistics, I can easily stand up a VM or two to cover all
> of the
> >> > KVM mgmt/host hypervisor and storage configs if I can figure out how
> to
> >> > integrate.
> >> > On Sep 11, 2013 2:10 AM, "Sebastien Goasguen" <ru...@gmail.com>
> wrote:
> >> >
> >> >>
> >> >> On Sep 11, 2013, at 3:02 AM, Prasanna Santhanam <ts...@apache.org>
> wrote:
> >> >>
> >> >> > CloudStack API actions are agnostic of underlying infrastructure
> and
> >> >> > most cases can fall into such a category as you describe. But
> imagine
> >> >> > this - I want to test snapshots ..
> >> >> >
> >> >> > so i take a snapshot and verify if it backedup correctly against a
> >> >> > ceph object store, nfs store or iscsi store. that sort of test is
> >> >> > going to involve more than just api actions.
> >> >> >
> >> >> > or say - i want to test multiple shared networks a VM gets deployed
> >> >> > into. Do I assume the deployment has multiple shared networks? Can
> i
> >> >> > add my own network into the deployment?
> >> >> >
> >> >> > or even - I want to exhaust all the public network IPs and check if
> >> >> > the next deployed VM picks up an IP in the new public range I've
> >> >> > added. This sort of test assumes that all the necessary networking
> is
> >> >> > in place and also hurts VM deployments of all tests that run at the
> >> >> > same time.
> >> >> >
> >> >> > It's a difficult balance to strike but we have to begin somewhere.
> >> >> > Start with the basic minimum that every infra can run. infra
> specifc
> >> >> > tests skip if thing are unsuitable, but will run for someone who
> wants
> >> >> > to test that feature
> >> >>
> >> >> A small point here to make is that jenkins.cloudstack.org is open to
> >> >> anyone.
> >> >> Prasanna has created an account for me and I am (slowly) working on
> >> >> adding tests for clients including aws.
> >> >>
> >> >> Anyone could use this jenkins instance, bring in slaves from "home"
> and
> >> >> setup tests?
> >> >>
> >> >> Back to the solidfire example, I think Mike could easily contribute
> one
> >> >> node that has a solidfire storage, then contribute Marvin tests that
> would
> >> >> run on jenkins.c.o and target his slave specifically. Same for KVM on
> >> >> Ubuntu...
> >> >>
> >> >> -sebastien
> >> >>
> >> >>
> >> >> >
> >> >> > On Tue, Sep 10, 2013 at 11:53:15PM -0700, Ahmad Emneina wrote:
> >> >> >> That's a good question, I'm not sure how preconditions work with
> >> >> >> Marvin cases, but I know the tests are run generically. Say I run
> >> >> >> copyvolumeToPrimary (not sure this test exists, hypothetical at
> the
> >> >> >> moment), it gets run against a slew of infrastructure
> configurations
> >> >> >> using local storage as well as shared (NSF, iscsi, ceph...) back
> >> >> >> ends. So just dropping my test into a storage suite should give it
> >> >> >> some guarantee its hitting a few different storage back-ends.
> That's
> >> >> >> how i understand it works today, I'll defer to Prasanna or
> Sudha...
> >> >> >> Or anyone else that runs tests aggressively to fill in the gaps
> and
> >> >> >> make corrections.
> >> >> >>
> >> >> >> Ahmad
> >> >> >>
> >> >> >> On Sep 10, 2013, at 11:43 PM, Marcus Sorensen <
> shadowsor@gmail.com>
> >> >> wrote:
> >> >> >>
> >> >> >>> But if the test requires some sort of preconfiguration, what then
> >> >> (e.g. test NFS primary storage would need a local or remote NFS
> >> >> configured)? do I need to roll my own, or can I touch the existing
> test
> >> >> infra and do the preconfigure?
> >> >> >>>
> >> >> >>> On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <ts...@apache.org>
> >> >> wrote:
> >> >> >>>> Yes - Once your test goes into the repo, it should get picked
> in the
> >> >> subsequent
> >> >> >>>> run.
> >> >> >>>>
> >> >> >>>> Jenkins installations from various companies can be combined
> into a
> >> >> single
> >> >> >>>> landing page. Jenkins itself doesn't support master/slave but it
> >> >> does through
> >> >> >>>> the gearman plugin. It's something I have tried using with VMs
> but
> >> >> not with
> >> >> >>>> real infra - but it is entirely possible.
> >> >> >>>>
> >> >> >>>> On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
> >> >> >>>>> I think there are jenkins slaves that run the nicera plugins
> on/at
> >> >> Schuberg
> >> >> >>>>> Philis housed infrastructure. The Citrix jenkins nodes also
> runs as
> >> >> slaves
> >> >> >>>>> that connect back to the apache owned/controlled jenkins. No
> reason
> >> >> why
> >> >> >>>>> testing infra need be so consolidated, it just so happens no
> one is
> >> >> putting
> >> >> >>>>> their hardware where their mouth is.
> >> >> >>>>>
> >> >> >>>>> I also assume if your marvin tests get accepted upstream,
> they'll be
> >> >> >>>>> included in the nightly runs/reports. Prasanna correct me if
> I'm
> >> >> wrong.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <
> >> >> shadowsor@gmail.com>wrote:
> >> >> >>>>>
> >> >> >>>>>> CloudStack Dev,
> >> >> >>>>>> I was emailed about some of the testing questions I
> brought up
> >> >> >>>>>> over the last few threads, and a few things were pointed out
> to me
> >> >> >>>>>> that I think we should try to remedy. Primarily, that the
> testing
> >> >> >>>>>> environment is owned by Citrix, the QA team is primarily
> >> >> Citrix-run,
> >> >> >>>>>> and the testing done is focused on the use models that Citrix
> >> >> >>>>>> develops.
> >> >> >>>>>> I've been assured that the test infrastructure is for
> everyone,
> >> >> >>>>>> and I'm not at all trying to say that there's a problem with
> Citrix
> >> >> >>>>>> focusing their work on their own interests, but I'm not sure
> that
> >> >> >>>>>> anyone outside of Citrix really knows how to add their own
> stuff to
> >> >> >>>>>> this testing infrastructure (perhaps for lack of trying, I
> don't
> >> >> >>>>>> know).
> >> >> >>>>>> I haven't really put together enough thought to know how to
> >> >> tackle
> >> >> >>>>>> this, but my gut tells me that we need some sort of
> community-owned
> >> >> >>>>>> testing roll-up, where everyone can do their own testing in
> >> >> whatever
> >> >> >>>>>> infrastructure and submit hourly, daily, weekly results. If
> my test
> >> >> >>>>>> fits into the Citrix test infrastructure and I can figure out
> how
> >> >> to
> >> >> >>>>>> get it there, great. If not, I can roll my own and integrate
> it via
> >> >> >>>>>> some API. For example the SolidFire guys may wan to run
> automated
> >> >> >>>>>> regression testing. That probably won't be doable in the
> Citrix
> >> >> >>>>>> infrastructure, but they may want to script a daily
> >> >> >>>>>> git-pull/build/deploy zone/create volume and it seems logical
> that
> >> >> >>>>>> we'd want to support it.
> >> >> >>>>>> Thoughts? Anyone have experience with such things? Can we
> have a
> >> >> >>>>>> master/slave scenario with Jenkins? Perhaps the Citrix
> environment
> >> >> >>>>>> already supports something like this via Jenkins API?
> >> >> >>>>>>
> >> >> >>>>
> >> >> >>>> --
> >> >> >>>> Prasanna.,
> >> >> >>>>
> >> >> >>>> ------------------------
> >> >> >>>> Powered by BigRock.com
> >> >> >
> >> >> > --
> >> >> > Prasanna.,
> >> >> >
> >> >> > ------------------------
> >> >> > Powered by BigRock.com
> >> >> >
> >> >>
> >> >>
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
> >
>
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*
Re: community testing
Posted by Marcus Sorensen <sh...@gmail.com>.
I think the test infra as described is great, but I think we're
hurting a little more for basics. For example, we don't need a full
infrastructure with hardware to ensure that the support matrix works.
I could bring up a VM with CentOS and one with Ubuntu, and test NFS,
CLVM, and RBD on each. CLVM just needs a volume group, NFS can be
exported locally, and RBD can run on localhost node (ceph has how-tos
for this to get your feet wet, that also buys us S3 compatible object
storage for testing secondary). Two VMs with maybe 2 cores, 4GB ram
each, and I think we could knock out a big swath of the basic "does it
work on the supported platforms" that we're missing with a very simple
automated testing. We can easily donate that much.
I agree that third parties would need to plug in their own testing
(solid fire, as you mention). And certainly testing full blown
deployment from the ground up like it sounds like we are doing is
great and necessary, I just want to plug a few holes and add some
basic sanity checking that we seem to keep getting tripped up on.
On Wed, Sep 11, 2013 at 10:23 PM, Prasanna Santhanam <ts...@apache.org> wrote:
> As Sebastien said, it's easy to get you the credentials for jenkins.
> Anyone with commit rights can request for an account. In fact one is
> created soon as you commit. I just need to adjust the credentials.
> (We'll move to git based job configurations but later)
>
> Citrix is unable to test various configurations for lack of necessary
> resources. for eg: It would be hard to test something that requires
> hardware resources like Nicira/Midokura/Solidfire. The current testbed
> is also limited in that it only deploys standard zone models. I have
> only one storage node to spare on which NFS is configured.
>
> CloudStack can be deployed and configured in so many ways that I don't
> think a single testbed cycling through all models is going to be
> effective in testing every possible configuration in time. This is why
> I'd like everyone of us to chip-in and use each others resources to
> make the infrastructure better.
>
> The RBD store at least will require sometime for us to bring up. It
> would be best if we could roll a few hosts from different datacenters
> up into jenkins. Object storage backed CS with something like Riak is
> another untested configuration. It is definitely tested within Citrix
> Labs but those testbeds are internal and cannot be exposed to the
> community. We've got corporate IT which wouldn't like that very much :)
>
> Ultimately, I'd want testbeds span across companies contributing to
> cloudstack. I wouldn't want any single company X to hold the resources
> and control allocation for testing even though that is not the case at
> all.
>
> We still need to figure out how securely these deployments can be
> brought into jenkins and who holds keys to the infrastructure. I'm no
> secure conscious sysadmin so I'm hoping for inputs from operators
> deploying cloudstack.
>
> On Wed, Sep 11, 2013 at 11:12:34AM -0600, Marcus Sorensen wrote:
>> Again, I'm not knocking Citrix. If anything, the issue is that they tend to
>> be so generous and community oriented that it surprises me when I find out
>> that certain donation is limited to their interests. Its perfectly
>> reasonable, e.g. my own donations are mostly limited to KVM.
>> On Sep 11, 2013 10:52 AM, "Marcus Sorensen" <sh...@gmail.com> wrote:
>>
>> > I do understand that. The email I received just triggered warning bells
>> > because it gave me the impression that the QA team as it stands isn't
>> > testing anything that Citrix doesn't care about, regardless of what the
>> > community has put on the support matrix. This includes even basic configs
>> > that the community claims to support like KVM on Ubuntu as the 4.1 release
>> > shows, and other things that we may already have infra for but just haven't
>> > implemented.
>> >
>> > That led me to wondering how much control the community really has over
>> > testing. Its good to know that we can roll our own nodes up into Jenkins,
>> > and/or modify tests if the infrastructure is already there. We just need to
>> > raise awareness as a community that there are still holes in resources and
>> > a need for donations to provide the minimum testing required for our
>> > support matrix. I think David's email about release requirements is a good
>> > step.
>> >
>> > If possible I'd like to modify the existing KVM testing to support testing
>> > NFS, CLVM, and RBD. This can all be done with a single host (that
>> > presumably already exists), we just need to set up the storage on the host
>> > and add create pool commands and volume create/delete tests. I'll have to
>> > figure out how to go about getting admin rights on the KVM test hosts to
>> > configure the storage types or work with someone. If we can't do that due
>> > to company logistics, I can easily stand up a VM or two to cover all of the
>> > KVM mgmt/host hypervisor and storage configs if I can figure out how to
>> > integrate.
>> > On Sep 11, 2013 2:10 AM, "Sebastien Goasguen" <ru...@gmail.com> wrote:
>> >
>> >>
>> >> On Sep 11, 2013, at 3:02 AM, Prasanna Santhanam <ts...@apache.org> wrote:
>> >>
>> >> > CloudStack API actions are agnostic of underlying infrastructure and
>> >> > most cases can fall into such a category as you describe. But imagine
>> >> > this - I want to test snapshots ..
>> >> >
>> >> > so i take a snapshot and verify if it backedup correctly against a
>> >> > ceph object store, nfs store or iscsi store. that sort of test is
>> >> > going to involve more than just api actions.
>> >> >
>> >> > or say - i want to test multiple shared networks a VM gets deployed
>> >> > into. Do I assume the deployment has multiple shared networks? Can i
>> >> > add my own network into the deployment?
>> >> >
>> >> > or even - I want to exhaust all the public network IPs and check if
>> >> > the next deployed VM picks up an IP in the new public range I've
>> >> > added. This sort of test assumes that all the necessary networking is
>> >> > in place and also hurts VM deployments of all tests that run at the
>> >> > same time.
>> >> >
>> >> > It's a difficult balance to strike but we have to begin somewhere.
>> >> > Start with the basic minimum that every infra can run. infra specifc
>> >> > tests skip if thing are unsuitable, but will run for someone who wants
>> >> > to test that feature
>> >>
>> >> A small point here to make is that jenkins.cloudstack.org is open to
>> >> anyone.
>> >> Prasanna has created an account for me and I am (slowly) working on
>> >> adding tests for clients including aws.
>> >>
>> >> Anyone could use this jenkins instance, bring in slaves from "home" and
>> >> setup tests?
>> >>
>> >> Back to the solidfire example, I think Mike could easily contribute one
>> >> node that has a solidfire storage, then contribute Marvin tests that would
>> >> run on jenkins.c.o and target his slave specifically. Same for KVM on
>> >> Ubuntu...
>> >>
>> >> -sebastien
>> >>
>> >>
>> >> >
>> >> > On Tue, Sep 10, 2013 at 11:53:15PM -0700, Ahmad Emneina wrote:
>> >> >> That's a good question, I'm not sure how preconditions work with
>> >> >> Marvin cases, but I know the tests are run generically. Say I run
>> >> >> copyvolumeToPrimary (not sure this test exists, hypothetical at the
>> >> >> moment), it gets run against a slew of infrastructure configurations
>> >> >> using local storage as well as shared (NSF, iscsi, ceph...) back
>> >> >> ends. So just dropping my test into a storage suite should give it
>> >> >> some guarantee its hitting a few different storage back-ends. That's
>> >> >> how i understand it works today, I'll defer to Prasanna or Sudha...
>> >> >> Or anyone else that runs tests aggressively to fill in the gaps and
>> >> >> make corrections.
>> >> >>
>> >> >> Ahmad
>> >> >>
>> >> >> On Sep 10, 2013, at 11:43 PM, Marcus Sorensen <sh...@gmail.com>
>> >> wrote:
>> >> >>
>> >> >>> But if the test requires some sort of preconfiguration, what then
>> >> (e.g. test NFS primary storage would need a local or remote NFS
>> >> configured)? do I need to roll my own, or can I touch the existing test
>> >> infra and do the preconfigure?
>> >> >>>
>> >> >>> On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <ts...@apache.org>
>> >> wrote:
>> >> >>>> Yes - Once your test goes into the repo, it should get picked in the
>> >> subsequent
>> >> >>>> run.
>> >> >>>>
>> >> >>>> Jenkins installations from various companies can be combined into a
>> >> single
>> >> >>>> landing page. Jenkins itself doesn't support master/slave but it
>> >> does through
>> >> >>>> the gearman plugin. It's something I have tried using with VMs but
>> >> not with
>> >> >>>> real infra - but it is entirely possible.
>> >> >>>>
>> >> >>>> On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
>> >> >>>>> I think there are jenkins slaves that run the nicera plugins on/at
>> >> Schuberg
>> >> >>>>> Philis housed infrastructure. The Citrix jenkins nodes also runs as
>> >> slaves
>> >> >>>>> that connect back to the apache owned/controlled jenkins. No reason
>> >> why
>> >> >>>>> testing infra need be so consolidated, it just so happens no one is
>> >> putting
>> >> >>>>> their hardware where their mouth is.
>> >> >>>>>
>> >> >>>>> I also assume if your marvin tests get accepted upstream, they'll be
>> >> >>>>> included in the nightly runs/reports. Prasanna correct me if I'm
>> >> wrong.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <
>> >> shadowsor@gmail.com>wrote:
>> >> >>>>>
>> >> >>>>>> CloudStack Dev,
>> >> >>>>>> I was emailed about some of the testing questions I brought up
>> >> >>>>>> over the last few threads, and a few things were pointed out to me
>> >> >>>>>> that I think we should try to remedy. Primarily, that the testing
>> >> >>>>>> environment is owned by Citrix, the QA team is primarily
>> >> Citrix-run,
>> >> >>>>>> and the testing done is focused on the use models that Citrix
>> >> >>>>>> develops.
>> >> >>>>>> I've been assured that the test infrastructure is for everyone,
>> >> >>>>>> and I'm not at all trying to say that there's a problem with Citrix
>> >> >>>>>> focusing their work on their own interests, but I'm not sure that
>> >> >>>>>> anyone outside of Citrix really knows how to add their own stuff to
>> >> >>>>>> this testing infrastructure (perhaps for lack of trying, I don't
>> >> >>>>>> know).
>> >> >>>>>> I haven't really put together enough thought to know how to
>> >> tackle
>> >> >>>>>> this, but my gut tells me that we need some sort of community-owned
>> >> >>>>>> testing roll-up, where everyone can do their own testing in
>> >> whatever
>> >> >>>>>> infrastructure and submit hourly, daily, weekly results. If my test
>> >> >>>>>> fits into the Citrix test infrastructure and I can figure out how
>> >> to
>> >> >>>>>> get it there, great. If not, I can roll my own and integrate it via
>> >> >>>>>> some API. For example the SolidFire guys may wan to run automated
>> >> >>>>>> regression testing. That probably won't be doable in the Citrix
>> >> >>>>>> infrastructure, but they may want to script a daily
>> >> >>>>>> git-pull/build/deploy zone/create volume and it seems logical that
>> >> >>>>>> we'd want to support it.
>> >> >>>>>> Thoughts? Anyone have experience with such things? Can we have a
>> >> >>>>>> master/slave scenario with Jenkins? Perhaps the Citrix environment
>> >> >>>>>> already supports something like this via Jenkins API?
>> >> >>>>>>
>> >> >>>>
>> >> >>>> --
>> >> >>>> Prasanna.,
>> >> >>>>
>> >> >>>> ------------------------
>> >> >>>> Powered by BigRock.com
>> >> >
>> >> > --
>> >> > Prasanna.,
>> >> >
>> >> > ------------------------
>> >> > Powered by BigRock.com
>> >> >
>> >>
>> >>
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
Re: community testing
Posted by Prasanna Santhanam <ts...@apache.org>.
As Sebastien said, it's easy to get you the credentials for jenkins.
Anyone with commit rights can request for an account. In fact one is
created soon as you commit. I just need to adjust the credentials.
(We'll move to git based job configurations but later)
Citrix is unable to test various configurations for lack of necessary
resources. for eg: It would be hard to test something that requires
hardware resources like Nicira/Midokura/Solidfire. The current testbed
is also limited in that it only deploys standard zone models. I have
only one storage node to spare on which NFS is configured.
CloudStack can be deployed and configured in so many ways that I don't
think a single testbed cycling through all models is going to be
effective in testing every possible configuration in time. This is why
I'd like everyone of us to chip-in and use each others resources to
make the infrastructure better.
The RBD store at least will require sometime for us to bring up. It
would be best if we could roll a few hosts from different datacenters
up into jenkins. Object storage backed CS with something like Riak is
another untested configuration. It is definitely tested within Citrix
Labs but those testbeds are internal and cannot be exposed to the
community. We've got corporate IT which wouldn't like that very much :)
Ultimately, I'd want testbeds span across companies contributing to
cloudstack. I wouldn't want any single company X to hold the resources
and control allocation for testing even though that is not the case at
all.
We still need to figure out how securely these deployments can be
brought into jenkins and who holds keys to the infrastructure. I'm no
secure conscious sysadmin so I'm hoping for inputs from operators
deploying cloudstack.
On Wed, Sep 11, 2013 at 11:12:34AM -0600, Marcus Sorensen wrote:
> Again, I'm not knocking Citrix. If anything, the issue is that they tend to
> be so generous and community oriented that it surprises me when I find out
> that certain donation is limited to their interests. Its perfectly
> reasonable, e.g. my own donations are mostly limited to KVM.
> On Sep 11, 2013 10:52 AM, "Marcus Sorensen" <sh...@gmail.com> wrote:
>
> > I do understand that. The email I received just triggered warning bells
> > because it gave me the impression that the QA team as it stands isn't
> > testing anything that Citrix doesn't care about, regardless of what the
> > community has put on the support matrix. This includes even basic configs
> > that the community claims to support like KVM on Ubuntu as the 4.1 release
> > shows, and other things that we may already have infra for but just haven't
> > implemented.
> >
> > That led me to wondering how much control the community really has over
> > testing. Its good to know that we can roll our own nodes up into Jenkins,
> > and/or modify tests if the infrastructure is already there. We just need to
> > raise awareness as a community that there are still holes in resources and
> > a need for donations to provide the minimum testing required for our
> > support matrix. I think David's email about release requirements is a good
> > step.
> >
> > If possible I'd like to modify the existing KVM testing to support testing
> > NFS, CLVM, and RBD. This can all be done with a single host (that
> > presumably already exists), we just need to set up the storage on the host
> > and add create pool commands and volume create/delete tests. I'll have to
> > figure out how to go about getting admin rights on the KVM test hosts to
> > configure the storage types or work with someone. If we can't do that due
> > to company logistics, I can easily stand up a VM or two to cover all of the
> > KVM mgmt/host hypervisor and storage configs if I can figure out how to
> > integrate.
> > On Sep 11, 2013 2:10 AM, "Sebastien Goasguen" <ru...@gmail.com> wrote:
> >
> >>
> >> On Sep 11, 2013, at 3:02 AM, Prasanna Santhanam <ts...@apache.org> wrote:
> >>
> >> > CloudStack API actions are agnostic of underlying infrastructure and
> >> > most cases can fall into such a category as you describe. But imagine
> >> > this - I want to test snapshots ..
> >> >
> >> > so i take a snapshot and verify if it backedup correctly against a
> >> > ceph object store, nfs store or iscsi store. that sort of test is
> >> > going to involve more than just api actions.
> >> >
> >> > or say - i want to test multiple shared networks a VM gets deployed
> >> > into. Do I assume the deployment has multiple shared networks? Can i
> >> > add my own network into the deployment?
> >> >
> >> > or even - I want to exhaust all the public network IPs and check if
> >> > the next deployed VM picks up an IP in the new public range I've
> >> > added. This sort of test assumes that all the necessary networking is
> >> > in place and also hurts VM deployments of all tests that run at the
> >> > same time.
> >> >
> >> > It's a difficult balance to strike but we have to begin somewhere.
> >> > Start with the basic minimum that every infra can run. infra specifc
> >> > tests skip if thing are unsuitable, but will run for someone who wants
> >> > to test that feature
> >>
> >> A small point here to make is that jenkins.cloudstack.org is open to
> >> anyone.
> >> Prasanna has created an account for me and I am (slowly) working on
> >> adding tests for clients including aws.
> >>
> >> Anyone could use this jenkins instance, bring in slaves from "home" and
> >> setup tests?
> >>
> >> Back to the solidfire example, I think Mike could easily contribute one
> >> node that has a solidfire storage, then contribute Marvin tests that would
> >> run on jenkins.c.o and target his slave specifically. Same for KVM on
> >> Ubuntu...
> >>
> >> -sebastien
> >>
> >>
> >> >
> >> > On Tue, Sep 10, 2013 at 11:53:15PM -0700, Ahmad Emneina wrote:
> >> >> That's a good question, I'm not sure how preconditions work with
> >> >> Marvin cases, but I know the tests are run generically. Say I run
> >> >> copyvolumeToPrimary (not sure this test exists, hypothetical at the
> >> >> moment), it gets run against a slew of infrastructure configurations
> >> >> using local storage as well as shared (NSF, iscsi, ceph...) back
> >> >> ends. So just dropping my test into a storage suite should give it
> >> >> some guarantee its hitting a few different storage back-ends. That's
> >> >> how i understand it works today, I'll defer to Prasanna or Sudha...
> >> >> Or anyone else that runs tests aggressively to fill in the gaps and
> >> >> make corrections.
> >> >>
> >> >> Ahmad
> >> >>
> >> >> On Sep 10, 2013, at 11:43 PM, Marcus Sorensen <sh...@gmail.com>
> >> wrote:
> >> >>
> >> >>> But if the test requires some sort of preconfiguration, what then
> >> (e.g. test NFS primary storage would need a local or remote NFS
> >> configured)? do I need to roll my own, or can I touch the existing test
> >> infra and do the preconfigure?
> >> >>>
> >> >>> On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <ts...@apache.org>
> >> wrote:
> >> >>>> Yes - Once your test goes into the repo, it should get picked in the
> >> subsequent
> >> >>>> run.
> >> >>>>
> >> >>>> Jenkins installations from various companies can be combined into a
> >> single
> >> >>>> landing page. Jenkins itself doesn't support master/slave but it
> >> does through
> >> >>>> the gearman plugin. It's something I have tried using with VMs but
> >> not with
> >> >>>> real infra - but it is entirely possible.
> >> >>>>
> >> >>>> On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
> >> >>>>> I think there are jenkins slaves that run the nicera plugins on/at
> >> Schuberg
> >> >>>>> Philis housed infrastructure. The Citrix jenkins nodes also runs as
> >> slaves
> >> >>>>> that connect back to the apache owned/controlled jenkins. No reason
> >> why
> >> >>>>> testing infra need be so consolidated, it just so happens no one is
> >> putting
> >> >>>>> their hardware where their mouth is.
> >> >>>>>
> >> >>>>> I also assume if your marvin tests get accepted upstream, they'll be
> >> >>>>> included in the nightly runs/reports. Prasanna correct me if I'm
> >> wrong.
> >> >>>>>
> >> >>>>>
> >> >>>>> On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <
> >> shadowsor@gmail.com>wrote:
> >> >>>>>
> >> >>>>>> CloudStack Dev,
> >> >>>>>> I was emailed about some of the testing questions I brought up
> >> >>>>>> over the last few threads, and a few things were pointed out to me
> >> >>>>>> that I think we should try to remedy. Primarily, that the testing
> >> >>>>>> environment is owned by Citrix, the QA team is primarily
> >> Citrix-run,
> >> >>>>>> and the testing done is focused on the use models that Citrix
> >> >>>>>> develops.
> >> >>>>>> I've been assured that the test infrastructure is for everyone,
> >> >>>>>> and I'm not at all trying to say that there's a problem with Citrix
> >> >>>>>> focusing their work on their own interests, but I'm not sure that
> >> >>>>>> anyone outside of Citrix really knows how to add their own stuff to
> >> >>>>>> this testing infrastructure (perhaps for lack of trying, I don't
> >> >>>>>> know).
> >> >>>>>> I haven't really put together enough thought to know how to
> >> tackle
> >> >>>>>> this, but my gut tells me that we need some sort of community-owned
> >> >>>>>> testing roll-up, where everyone can do their own testing in
> >> whatever
> >> >>>>>> infrastructure and submit hourly, daily, weekly results. If my test
> >> >>>>>> fits into the Citrix test infrastructure and I can figure out how
> >> to
> >> >>>>>> get it there, great. If not, I can roll my own and integrate it via
> >> >>>>>> some API. For example the SolidFire guys may wan to run automated
> >> >>>>>> regression testing. That probably won't be doable in the Citrix
> >> >>>>>> infrastructure, but they may want to script a daily
> >> >>>>>> git-pull/build/deploy zone/create volume and it seems logical that
> >> >>>>>> we'd want to support it.
> >> >>>>>> Thoughts? Anyone have experience with such things? Can we have a
> >> >>>>>> master/slave scenario with Jenkins? Perhaps the Citrix environment
> >> >>>>>> already supports something like this via Jenkins API?
> >> >>>>>>
> >> >>>>
> >> >>>> --
> >> >>>> Prasanna.,
> >> >>>>
> >> >>>> ------------------------
> >> >>>> Powered by BigRock.com
> >> >
> >> > --
> >> > Prasanna.,
> >> >
> >> > ------------------------
> >> > Powered by BigRock.com
> >> >
> >>
> >>
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: community testing
Posted by Marcus Sorensen <sh...@gmail.com>.
Again, I'm not knocking Citrix. If anything, the issue is that they tend to
be so generous and community oriented that it surprises me when I find out
that certain donation is limited to their interests. Its perfectly
reasonable, e.g. my own donations are mostly limited to KVM.
On Sep 11, 2013 10:52 AM, "Marcus Sorensen" <sh...@gmail.com> wrote:
> I do understand that. The email I received just triggered warning bells
> because it gave me the impression that the QA team as it stands isn't
> testing anything that Citrix doesn't care about, regardless of what the
> community has put on the support matrix. This includes even basic configs
> that the community claims to support like KVM on Ubuntu as the 4.1 release
> shows, and other things that we may already have infra for but just haven't
> implemented.
>
> That led me to wondering how much control the community really has over
> testing. Its good to know that we can roll our own nodes up into Jenkins,
> and/or modify tests if the infrastructure is already there. We just need to
> raise awareness as a community that there are still holes in resources and
> a need for donations to provide the minimum testing required for our
> support matrix. I think David's email about release requirements is a good
> step.
>
> If possible I'd like to modify the existing KVM testing to support testing
> NFS, CLVM, and RBD. This can all be done with a single host (that
> presumably already exists), we just need to set up the storage on the host
> and add create pool commands and volume create/delete tests. I'll have to
> figure out how to go about getting admin rights on the KVM test hosts to
> configure the storage types or work with someone. If we can't do that due
> to company logistics, I can easily stand up a VM or two to cover all of the
> KVM mgmt/host hypervisor and storage configs if I can figure out how to
> integrate.
> On Sep 11, 2013 2:10 AM, "Sebastien Goasguen" <ru...@gmail.com> wrote:
>
>>
>> On Sep 11, 2013, at 3:02 AM, Prasanna Santhanam <ts...@apache.org> wrote:
>>
>> > CloudStack API actions are agnostic of underlying infrastructure and
>> > most cases can fall into such a category as you describe. But imagine
>> > this - I want to test snapshots ..
>> >
>> > so i take a snapshot and verify if it backedup correctly against a
>> > ceph object store, nfs store or iscsi store. that sort of test is
>> > going to involve more than just api actions.
>> >
>> > or say - i want to test multiple shared networks a VM gets deployed
>> > into. Do I assume the deployment has multiple shared networks? Can i
>> > add my own network into the deployment?
>> >
>> > or even - I want to exhaust all the public network IPs and check if
>> > the next deployed VM picks up an IP in the new public range I've
>> > added. This sort of test assumes that all the necessary networking is
>> > in place and also hurts VM deployments of all tests that run at the
>> > same time.
>> >
>> > It's a difficult balance to strike but we have to begin somewhere.
>> > Start with the basic minimum that every infra can run. infra specifc
>> > tests skip if thing are unsuitable, but will run for someone who wants
>> > to test that feature
>>
>> A small point here to make is that jenkins.cloudstack.org is open to
>> anyone.
>> Prasanna has created an account for me and I am (slowly) working on
>> adding tests for clients including aws.
>>
>> Anyone could use this jenkins instance, bring in slaves from "home" and
>> setup tests…
>>
>> Back to the solidfire example, I think Mike could easily contribute one
>> node that has a solidfire storage, then contribute Marvin tests that would
>> run on jenkins.c.o and target his slave specifically. Same for KVM on
>> Ubuntu...
>>
>> -sebastien
>>
>>
>> >
>> > On Tue, Sep 10, 2013 at 11:53:15PM -0700, Ahmad Emneina wrote:
>> >> That's a good question, I'm not sure how preconditions work with
>> >> Marvin cases, but I know the tests are run generically. Say I run
>> >> copyvolumeToPrimary (not sure this test exists, hypothetical at the
>> >> moment), it gets run against a slew of infrastructure configurations
>> >> using local storage as well as shared (NSF, iscsi, ceph...) back
>> >> ends. So just dropping my test into a storage suite should give it
>> >> some guarantee its hitting a few different storage back-ends. That's
>> >> how i understand it works today, I'll defer to Prasanna or Sudha...
>> >> Or anyone else that runs tests aggressively to fill in the gaps and
>> >> make corrections.
>> >>
>> >> Ahmad
>> >>
>> >> On Sep 10, 2013, at 11:43 PM, Marcus Sorensen <sh...@gmail.com>
>> wrote:
>> >>
>> >>> But if the test requires some sort of preconfiguration, what then
>> (e.g. test NFS primary storage would need a local or remote NFS
>> configured)? do I need to roll my own, or can I touch the existing test
>> infra and do the preconfigure?
>> >>>
>> >>> On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <ts...@apache.org>
>> wrote:
>> >>>> Yes - Once your test goes into the repo, it should get picked in the
>> subsequent
>> >>>> run.
>> >>>>
>> >>>> Jenkins installations from various companies can be combined into a
>> single
>> >>>> landing page. Jenkins itself doesn't support master/slave but it
>> does through
>> >>>> the gearman plugin. It's something I have tried using with VMs but
>> not with
>> >>>> real infra - but it is entirely possible.
>> >>>>
>> >>>> On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
>> >>>>> I think there are jenkins slaves that run the nicera plugins on/at
>> Schuberg
>> >>>>> Philis housed infrastructure. The Citrix jenkins nodes also runs as
>> slaves
>> >>>>> that connect back to the apache owned/controlled jenkins. No reason
>> why
>> >>>>> testing infra need be so consolidated, it just so happens no one is
>> putting
>> >>>>> their hardware where their mouth is.
>> >>>>>
>> >>>>> I also assume if your marvin tests get accepted upstream, they'll be
>> >>>>> included in the nightly runs/reports. Prasanna correct me if I'm
>> wrong.
>> >>>>>
>> >>>>>
>> >>>>> On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <
>> shadowsor@gmail.com>wrote:
>> >>>>>
>> >>>>>> CloudStack Dev,
>> >>>>>> I was emailed about some of the testing questions I brought up
>> >>>>>> over the last few threads, and a few things were pointed out to me
>> >>>>>> that I think we should try to remedy. Primarily, that the testing
>> >>>>>> environment is owned by Citrix, the QA team is primarily
>> Citrix-run,
>> >>>>>> and the testing done is focused on the use models that Citrix
>> >>>>>> develops.
>> >>>>>> I've been assured that the test infrastructure is for everyone,
>> >>>>>> and I'm not at all trying to say that there's a problem with Citrix
>> >>>>>> focusing their work on their own interests, but I'm not sure that
>> >>>>>> anyone outside of Citrix really knows how to add their own stuff to
>> >>>>>> this testing infrastructure (perhaps for lack of trying, I don't
>> >>>>>> know).
>> >>>>>> I haven't really put together enough thought to know how to
>> tackle
>> >>>>>> this, but my gut tells me that we need some sort of community-owned
>> >>>>>> testing roll-up, where everyone can do their own testing in
>> whatever
>> >>>>>> infrastructure and submit hourly, daily, weekly results. If my test
>> >>>>>> fits into the Citrix test infrastructure and I can figure out how
>> to
>> >>>>>> get it there, great. If not, I can roll my own and integrate it via
>> >>>>>> some API. For example the SolidFire guys may wan to run automated
>> >>>>>> regression testing. That probably won't be doable in the Citrix
>> >>>>>> infrastructure, but they may want to script a daily
>> >>>>>> git-pull/build/deploy zone/create volume and it seems logical that
>> >>>>>> we'd want to support it.
>> >>>>>> Thoughts? Anyone have experience with such things? Can we have a
>> >>>>>> master/slave scenario with Jenkins? Perhaps the Citrix environment
>> >>>>>> already supports something like this via Jenkins API?
>> >>>>>>
>> >>>>
>> >>>> --
>> >>>> Prasanna.,
>> >>>>
>> >>>> ------------------------
>> >>>> Powered by BigRock.com
>> >
>> > --
>> > Prasanna.,
>> >
>> > ------------------------
>> > Powered by BigRock.com
>> >
>>
>>
Re: community testing
Posted by Marcus Sorensen <sh...@gmail.com>.
I do understand that. The email I received just triggered warning bells
because it gave me the impression that the QA team as it stands isn't
testing anything that Citrix doesn't care about, regardless of what the
community has put on the support matrix. This includes even basic configs
that the community claims to support like KVM on Ubuntu as the 4.1 release
shows, and other things that we may already have infra for but just haven't
implemented.
That led me to wondering how much control the community really has over
testing. Its good to know that we can roll our own nodes up into Jenkins,
and/or modify tests if the infrastructure is already there. We just need to
raise awareness as a community that there are still holes in resources and
a need for donations to provide the minimum testing required for our
support matrix. I think David's email about release requirements is a good
step.
If possible I'd like to modify the existing KVM testing to support testing
NFS, CLVM, and RBD. This can all be done with a single host (that
presumably already exists), we just need to set up the storage on the host
and add create pool commands and volume create/delete tests. I'll have to
figure out how to go about getting admin rights on the KVM test hosts to
configure the storage types or work with someone. If we can't do that due
to company logistics, I can easily stand up a VM or two to cover all of the
KVM mgmt/host hypervisor and storage configs if I can figure out how to
integrate.
On Sep 11, 2013 2:10 AM, "Sebastien Goasguen" <ru...@gmail.com> wrote:
>
> On Sep 11, 2013, at 3:02 AM, Prasanna Santhanam <ts...@apache.org> wrote:
>
> > CloudStack API actions are agnostic of underlying infrastructure and
> > most cases can fall into such a category as you describe. But imagine
> > this - I want to test snapshots ..
> >
> > so i take a snapshot and verify if it backedup correctly against a
> > ceph object store, nfs store or iscsi store. that sort of test is
> > going to involve more than just api actions.
> >
> > or say - i want to test multiple shared networks a VM gets deployed
> > into. Do I assume the deployment has multiple shared networks? Can i
> > add my own network into the deployment?
> >
> > or even - I want to exhaust all the public network IPs and check if
> > the next deployed VM picks up an IP in the new public range I've
> > added. This sort of test assumes that all the necessary networking is
> > in place and also hurts VM deployments of all tests that run at the
> > same time.
> >
> > It's a difficult balance to strike but we have to begin somewhere.
> > Start with the basic minimum that every infra can run. infra specifc
> > tests skip if thing are unsuitable, but will run for someone who wants
> > to test that feature
>
> A small point here to make is that jenkins.cloudstack.org is open to
> anyone.
> Prasanna has created an account for me and I am (slowly) working on adding
> tests for clients including aws.
>
> Anyone could use this jenkins instance, bring in slaves from "home" and
> setup tests…
>
> Back to the solidfire example, I think Mike could easily contribute one
> node that has a solidfire storage, then contribute Marvin tests that would
> run on jenkins.c.o and target his slave specifically. Same for KVM on
> Ubuntu...
>
> -sebastien
>
>
> >
> > On Tue, Sep 10, 2013 at 11:53:15PM -0700, Ahmad Emneina wrote:
> >> That's a good question, I'm not sure how preconditions work with
> >> Marvin cases, but I know the tests are run generically. Say I run
> >> copyvolumeToPrimary (not sure this test exists, hypothetical at the
> >> moment), it gets run against a slew of infrastructure configurations
> >> using local storage as well as shared (NSF, iscsi, ceph...) back
> >> ends. So just dropping my test into a storage suite should give it
> >> some guarantee its hitting a few different storage back-ends. That's
> >> how i understand it works today, I'll defer to Prasanna or Sudha...
> >> Or anyone else that runs tests aggressively to fill in the gaps and
> >> make corrections.
> >>
> >> Ahmad
> >>
> >> On Sep 10, 2013, at 11:43 PM, Marcus Sorensen <sh...@gmail.com>
> wrote:
> >>
> >>> But if the test requires some sort of preconfiguration, what then
> (e.g. test NFS primary storage would need a local or remote NFS
> configured)? do I need to roll my own, or can I touch the existing test
> infra and do the preconfigure?
> >>>
> >>> On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
> >>>> Yes - Once your test goes into the repo, it should get picked in the
> subsequent
> >>>> run.
> >>>>
> >>>> Jenkins installations from various companies can be combined into a
> single
> >>>> landing page. Jenkins itself doesn't support master/slave but it does
> through
> >>>> the gearman plugin. It's something I have tried using with VMs but
> not with
> >>>> real infra - but it is entirely possible.
> >>>>
> >>>> On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
> >>>>> I think there are jenkins slaves that run the nicera plugins on/at
> Schuberg
> >>>>> Philis housed infrastructure. The Citrix jenkins nodes also runs as
> slaves
> >>>>> that connect back to the apache owned/controlled jenkins. No reason
> why
> >>>>> testing infra need be so consolidated, it just so happens no one is
> putting
> >>>>> their hardware where their mouth is.
> >>>>>
> >>>>> I also assume if your marvin tests get accepted upstream, they'll be
> >>>>> included in the nightly runs/reports. Prasanna correct me if I'm
> wrong.
> >>>>>
> >>>>>
> >>>>> On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <
> shadowsor@gmail.com>wrote:
> >>>>>
> >>>>>> CloudStack Dev,
> >>>>>> I was emailed about some of the testing questions I brought up
> >>>>>> over the last few threads, and a few things were pointed out to me
> >>>>>> that I think we should try to remedy. Primarily, that the testing
> >>>>>> environment is owned by Citrix, the QA team is primarily Citrix-run,
> >>>>>> and the testing done is focused on the use models that Citrix
> >>>>>> develops.
> >>>>>> I've been assured that the test infrastructure is for everyone,
> >>>>>> and I'm not at all trying to say that there's a problem with Citrix
> >>>>>> focusing their work on their own interests, but I'm not sure that
> >>>>>> anyone outside of Citrix really knows how to add their own stuff to
> >>>>>> this testing infrastructure (perhaps for lack of trying, I don't
> >>>>>> know).
> >>>>>> I haven't really put together enough thought to know how to
> tackle
> >>>>>> this, but my gut tells me that we need some sort of community-owned
> >>>>>> testing roll-up, where everyone can do their own testing in whatever
> >>>>>> infrastructure and submit hourly, daily, weekly results. If my test
> >>>>>> fits into the Citrix test infrastructure and I can figure out how to
> >>>>>> get it there, great. If not, I can roll my own and integrate it via
> >>>>>> some API. For example the SolidFire guys may wan to run automated
> >>>>>> regression testing. That probably won't be doable in the Citrix
> >>>>>> infrastructure, but they may want to script a daily
> >>>>>> git-pull/build/deploy zone/create volume and it seems logical that
> >>>>>> we'd want to support it.
> >>>>>> Thoughts? Anyone have experience with such things? Can we have a
> >>>>>> master/slave scenario with Jenkins? Perhaps the Citrix environment
> >>>>>> already supports something like this via Jenkins API?
> >>>>>>
> >>>>
> >>>> --
> >>>> Prasanna.,
> >>>>
> >>>> ------------------------
> >>>> Powered by BigRock.com
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
> >
>
>
Re: community testing
Posted by Mike Tutkowski <mi...@solidfire.com>.
Yeah, that would be fantastic if I could test the SolidFire plug-in in such
a way.
I could probably get away with a virtual storage node, as well, because I'm
not testing anything that requires the real hardware.
On Wed, Sep 11, 2013 at 2:10 AM, Sebastien Goasguen <ru...@gmail.com>wrote:
>
> On Sep 11, 2013, at 3:02 AM, Prasanna Santhanam <ts...@apache.org> wrote:
>
> > CloudStack API actions are agnostic of underlying infrastructure and
> > most cases can fall into such a category as you describe. But imagine
> > this - I want to test snapshots ..
> >
> > so i take a snapshot and verify if it backedup correctly against a
> > ceph object store, nfs store or iscsi store. that sort of test is
> > going to involve more than just api actions.
> >
> > or say - i want to test multiple shared networks a VM gets deployed
> > into. Do I assume the deployment has multiple shared networks? Can i
> > add my own network into the deployment?
> >
> > or even - I want to exhaust all the public network IPs and check if
> > the next deployed VM picks up an IP in the new public range I've
> > added. This sort of test assumes that all the necessary networking is
> > in place and also hurts VM deployments of all tests that run at the
> > same time.
> >
> > It's a difficult balance to strike but we have to begin somewhere.
> > Start with the basic minimum that every infra can run. infra specifc
> > tests skip if thing are unsuitable, but will run for someone who wants
> > to test that feature
>
> A small point here to make is that jenkins.cloudstack.org is open to
> anyone.
> Prasanna has created an account for me and I am (slowly) working on adding
> tests for clients including aws.
>
> Anyone could use this jenkins instance, bring in slaves from "home" and
> setup tests…
>
> Back to the solidfire example, I think Mike could easily contribute one
> node that has a solidfire storage, then contribute Marvin tests that would
> run on jenkins.c.o and target his slave specifically. Same for KVM on
> Ubuntu...
>
> -sebastien
>
>
> >
> > On Tue, Sep 10, 2013 at 11:53:15PM -0700, Ahmad Emneina wrote:
> >> That's a good question, I'm not sure how preconditions work with
> >> Marvin cases, but I know the tests are run generically. Say I run
> >> copyvolumeToPrimary (not sure this test exists, hypothetical at the
> >> moment), it gets run against a slew of infrastructure configurations
> >> using local storage as well as shared (NSF, iscsi, ceph...) back
> >> ends. So just dropping my test into a storage suite should give it
> >> some guarantee its hitting a few different storage back-ends. That's
> >> how i understand it works today, I'll defer to Prasanna or Sudha...
> >> Or anyone else that runs tests aggressively to fill in the gaps and
> >> make corrections.
> >>
> >> Ahmad
> >>
> >> On Sep 10, 2013, at 11:43 PM, Marcus Sorensen <sh...@gmail.com>
> wrote:
> >>
> >>> But if the test requires some sort of preconfiguration, what then
> (e.g. test NFS primary storage would need a local or remote NFS
> configured)? do I need to roll my own, or can I touch the existing test
> infra and do the preconfigure?
> >>>
> >>> On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
> >>>> Yes - Once your test goes into the repo, it should get picked in the
> subsequent
> >>>> run.
> >>>>
> >>>> Jenkins installations from various companies can be combined into a
> single
> >>>> landing page. Jenkins itself doesn't support master/slave but it does
> through
> >>>> the gearman plugin. It's something I have tried using with VMs but
> not with
> >>>> real infra - but it is entirely possible.
> >>>>
> >>>> On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
> >>>>> I think there are jenkins slaves that run the nicera plugins on/at
> Schuberg
> >>>>> Philis housed infrastructure. The Citrix jenkins nodes also runs as
> slaves
> >>>>> that connect back to the apache owned/controlled jenkins. No reason
> why
> >>>>> testing infra need be so consolidated, it just so happens no one is
> putting
> >>>>> their hardware where their mouth is.
> >>>>>
> >>>>> I also assume if your marvin tests get accepted upstream, they'll be
> >>>>> included in the nightly runs/reports. Prasanna correct me if I'm
> wrong.
> >>>>>
> >>>>>
> >>>>> On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <
> shadowsor@gmail.com>wrote:
> >>>>>
> >>>>>> CloudStack Dev,
> >>>>>> I was emailed about some of the testing questions I brought up
> >>>>>> over the last few threads, and a few things were pointed out to me
> >>>>>> that I think we should try to remedy. Primarily, that the testing
> >>>>>> environment is owned by Citrix, the QA team is primarily Citrix-run,
> >>>>>> and the testing done is focused on the use models that Citrix
> >>>>>> develops.
> >>>>>> I've been assured that the test infrastructure is for everyone,
> >>>>>> and I'm not at all trying to say that there's a problem with Citrix
> >>>>>> focusing their work on their own interests, but I'm not sure that
> >>>>>> anyone outside of Citrix really knows how to add their own stuff to
> >>>>>> this testing infrastructure (perhaps for lack of trying, I don't
> >>>>>> know).
> >>>>>> I haven't really put together enough thought to know how to
> tackle
> >>>>>> this, but my gut tells me that we need some sort of community-owned
> >>>>>> testing roll-up, where everyone can do their own testing in whatever
> >>>>>> infrastructure and submit hourly, daily, weekly results. If my test
> >>>>>> fits into the Citrix test infrastructure and I can figure out how to
> >>>>>> get it there, great. If not, I can roll my own and integrate it via
> >>>>>> some API. For example the SolidFire guys may wan to run automated
> >>>>>> regression testing. That probably won't be doable in the Citrix
> >>>>>> infrastructure, but they may want to script a daily
> >>>>>> git-pull/build/deploy zone/create volume and it seems logical that
> >>>>>> we'd want to support it.
> >>>>>> Thoughts? Anyone have experience with such things? Can we have a
> >>>>>> master/slave scenario with Jenkins? Perhaps the Citrix environment
> >>>>>> already supports something like this via Jenkins API?
> >>>>>>
> >>>>
> >>>> --
> >>>> Prasanna.,
> >>>>
> >>>> ------------------------
> >>>> Powered by BigRock.com
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
> >
>
>
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*
Re: community testing
Posted by Sebastien Goasguen <ru...@gmail.com>.
On Sep 11, 2013, at 3:02 AM, Prasanna Santhanam <ts...@apache.org> wrote:
> CloudStack API actions are agnostic of underlying infrastructure and
> most cases can fall into such a category as you describe. But imagine
> this - I want to test snapshots ..
>
> so i take a snapshot and verify if it backedup correctly against a
> ceph object store, nfs store or iscsi store. that sort of test is
> going to involve more than just api actions.
>
> or say - i want to test multiple shared networks a VM gets deployed
> into. Do I assume the deployment has multiple shared networks? Can i
> add my own network into the deployment?
>
> or even - I want to exhaust all the public network IPs and check if
> the next deployed VM picks up an IP in the new public range I've
> added. This sort of test assumes that all the necessary networking is
> in place and also hurts VM deployments of all tests that run at the
> same time.
>
> It's a difficult balance to strike but we have to begin somewhere.
> Start with the basic minimum that every infra can run. infra specifc
> tests skip if thing are unsuitable, but will run for someone who wants
> to test that feature
A small point here to make is that jenkins.cloudstack.org is open to anyone.
Prasanna has created an account for me and I am (slowly) working on adding tests for clients including aws.
Anyone could use this jenkins instance, bring in slaves from "home" and setup tests…
Back to the solidfire example, I think Mike could easily contribute one node that has a solidfire storage, then contribute Marvin tests that would run on jenkins.c.o and target his slave specifically. Same for KVM on Ubuntu...
-sebastien
>
> On Tue, Sep 10, 2013 at 11:53:15PM -0700, Ahmad Emneina wrote:
>> That's a good question, I'm not sure how preconditions work with
>> Marvin cases, but I know the tests are run generically. Say I run
>> copyvolumeToPrimary (not sure this test exists, hypothetical at the
>> moment), it gets run against a slew of infrastructure configurations
>> using local storage as well as shared (NSF, iscsi, ceph...) back
>> ends. So just dropping my test into a storage suite should give it
>> some guarantee its hitting a few different storage back-ends. That's
>> how i understand it works today, I'll defer to Prasanna or Sudha...
>> Or anyone else that runs tests aggressively to fill in the gaps and
>> make corrections.
>>
>> Ahmad
>>
>> On Sep 10, 2013, at 11:43 PM, Marcus Sorensen <sh...@gmail.com> wrote:
>>
>>> But if the test requires some sort of preconfiguration, what then (e.g. test NFS primary storage would need a local or remote NFS configured)? do I need to roll my own, or can I touch the existing test infra and do the preconfigure?
>>>
>>> On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
>>>> Yes - Once your test goes into the repo, it should get picked in the subsequent
>>>> run.
>>>>
>>>> Jenkins installations from various companies can be combined into a single
>>>> landing page. Jenkins itself doesn't support master/slave but it does through
>>>> the gearman plugin. It's something I have tried using with VMs but not with
>>>> real infra - but it is entirely possible.
>>>>
>>>> On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
>>>>> I think there are jenkins slaves that run the nicera plugins on/at Schuberg
>>>>> Philis housed infrastructure. The Citrix jenkins nodes also runs as slaves
>>>>> that connect back to the apache owned/controlled jenkins. No reason why
>>>>> testing infra need be so consolidated, it just so happens no one is putting
>>>>> their hardware where their mouth is.
>>>>>
>>>>> I also assume if your marvin tests get accepted upstream, they'll be
>>>>> included in the nightly runs/reports. Prasanna correct me if I'm wrong.
>>>>>
>>>>>
>>>>> On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <sh...@gmail.com>wrote:
>>>>>
>>>>>> CloudStack Dev,
>>>>>> I was emailed about some of the testing questions I brought up
>>>>>> over the last few threads, and a few things were pointed out to me
>>>>>> that I think we should try to remedy. Primarily, that the testing
>>>>>> environment is owned by Citrix, the QA team is primarily Citrix-run,
>>>>>> and the testing done is focused on the use models that Citrix
>>>>>> develops.
>>>>>> I've been assured that the test infrastructure is for everyone,
>>>>>> and I'm not at all trying to say that there's a problem with Citrix
>>>>>> focusing their work on their own interests, but I'm not sure that
>>>>>> anyone outside of Citrix really knows how to add their own stuff to
>>>>>> this testing infrastructure (perhaps for lack of trying, I don't
>>>>>> know).
>>>>>> I haven't really put together enough thought to know how to tackle
>>>>>> this, but my gut tells me that we need some sort of community-owned
>>>>>> testing roll-up, where everyone can do their own testing in whatever
>>>>>> infrastructure and submit hourly, daily, weekly results. If my test
>>>>>> fits into the Citrix test infrastructure and I can figure out how to
>>>>>> get it there, great. If not, I can roll my own and integrate it via
>>>>>> some API. For example the SolidFire guys may wan to run automated
>>>>>> regression testing. That probably won't be doable in the Citrix
>>>>>> infrastructure, but they may want to script a daily
>>>>>> git-pull/build/deploy zone/create volume and it seems logical that
>>>>>> we'd want to support it.
>>>>>> Thoughts? Anyone have experience with such things? Can we have a
>>>>>> master/slave scenario with Jenkins? Perhaps the Citrix environment
>>>>>> already supports something like this via Jenkins API?
>>>>>>
>>>>
>>>> --
>>>> Prasanna.,
>>>>
>>>> ------------------------
>>>> Powered by BigRock.com
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
Re: community testing
Posted by Prasanna Santhanam <ts...@apache.org>.
CloudStack API actions are agnostic of underlying infrastructure and
most cases can fall into such a category as you describe. But imagine
this - I want to test snapshots ..
so i take a snapshot and verify if it backedup correctly against a
ceph object store, nfs store or iscsi store. that sort of test is
going to involve more than just api actions.
or say - i want to test multiple shared networks a VM gets deployed
into. Do I assume the deployment has multiple shared networks? Can i
add my own network into the deployment?
or even - I want to exhaust all the public network IPs and check if
the next deployed VM picks up an IP in the new public range I've
added. This sort of test assumes that all the necessary networking is
in place and also hurts VM deployments of all tests that run at the
same time.
It's a difficult balance to strike but we have to begin somewhere.
Start with the basic minimum that every infra can run. infra specifc
tests skip if thing are unsuitable, but will run for someone who wants
to test that feature
On Tue, Sep 10, 2013 at 11:53:15PM -0700, Ahmad Emneina wrote:
> That's a good question, I'm not sure how preconditions work with
> Marvin cases, but I know the tests are run generically. Say I run
> copyvolumeToPrimary (not sure this test exists, hypothetical at the
> moment), it gets run against a slew of infrastructure configurations
> using local storage as well as shared (NSF, iscsi, ceph...) back
> ends. So just dropping my test into a storage suite should give it
> some guarantee its hitting a few different storage back-ends. That's
> how i understand it works today, I'll defer to Prasanna or Sudha...
> Or anyone else that runs tests aggressively to fill in the gaps and
> make corrections.
>
> Ahmad
>
> On Sep 10, 2013, at 11:43 PM, Marcus Sorensen <sh...@gmail.com> wrote:
>
> > But if the test requires some sort of preconfiguration, what then (e.g. test NFS primary storage would need a local or remote NFS configured)? do I need to roll my own, or can I touch the existing test infra and do the preconfigure?
> >
> > On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
> >> Yes - Once your test goes into the repo, it should get picked in the subsequent
> >> run.
> >>
> >> Jenkins installations from various companies can be combined into a single
> >> landing page. Jenkins itself doesn't support master/slave but it does through
> >> the gearman plugin. It's something I have tried using with VMs but not with
> >> real infra - but it is entirely possible.
> >>
> >> On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
> >> > I think there are jenkins slaves that run the nicera plugins on/at Schuberg
> >> > Philis housed infrastructure. The Citrix jenkins nodes also runs as slaves
> >> > that connect back to the apache owned/controlled jenkins. No reason why
> >> > testing infra need be so consolidated, it just so happens no one is putting
> >> > their hardware where their mouth is.
> >> >
> >> > I also assume if your marvin tests get accepted upstream, they'll be
> >> > included in the nightly runs/reports. Prasanna correct me if I'm wrong.
> >> >
> >> >
> >> > On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <sh...@gmail.com>wrote:
> >> >
> >> > > CloudStack Dev,
> >> > > I was emailed about some of the testing questions I brought up
> >> > > over the last few threads, and a few things were pointed out to me
> >> > > that I think we should try to remedy. Primarily, that the testing
> >> > > environment is owned by Citrix, the QA team is primarily Citrix-run,
> >> > > and the testing done is focused on the use models that Citrix
> >> > > develops.
> >> > > I've been assured that the test infrastructure is for everyone,
> >> > > and I'm not at all trying to say that there's a problem with Citrix
> >> > > focusing their work on their own interests, but I'm not sure that
> >> > > anyone outside of Citrix really knows how to add their own stuff to
> >> > > this testing infrastructure (perhaps for lack of trying, I don't
> >> > > know).
> >> > > I haven't really put together enough thought to know how to tackle
> >> > > this, but my gut tells me that we need some sort of community-owned
> >> > > testing roll-up, where everyone can do their own testing in whatever
> >> > > infrastructure and submit hourly, daily, weekly results. If my test
> >> > > fits into the Citrix test infrastructure and I can figure out how to
> >> > > get it there, great. If not, I can roll my own and integrate it via
> >> > > some API. For example the SolidFire guys may wan to run automated
> >> > > regression testing. That probably won't be doable in the Citrix
> >> > > infrastructure, but they may want to script a daily
> >> > > git-pull/build/deploy zone/create volume and it seems logical that
> >> > > we'd want to support it.
> >> > > Thoughts? Anyone have experience with such things? Can we have a
> >> > > master/slave scenario with Jenkins? Perhaps the Citrix environment
> >> > > already supports something like this via Jenkins API?
> >> > >
> >>
> >> --
> >> Prasanna.,
> >>
> >> ------------------------
> >> Powered by BigRock.com
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: community testing
Posted by Ahmad Emneina <ae...@gmail.com>.
That's a good question, I'm not sure how preconditions work with Marvin cases, but I know the tests are run generically. Say I run copyvolumeToPrimary (not sure this test exists, hypothetical at the moment), it gets run against a slew of infrastructure configurations using local storage as well as shared (NSF, iscsi, ceph...) back ends. So just dropping my test into a storage suite should give it some guarantee its hitting a few different storage back-ends. That's how i understand it works today, I'll defer to Prasanna or Sudha... Or anyone else that runs tests aggressively to fill in the gaps and make corrections.
Ahmad
On Sep 10, 2013, at 11:43 PM, Marcus Sorensen <sh...@gmail.com> wrote:
> But if the test requires some sort of preconfiguration, what then (e.g. test NFS primary storage would need a local or remote NFS configured)? do I need to roll my own, or can I touch the existing test infra and do the preconfigure?
>
> On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
>> Yes - Once your test goes into the repo, it should get picked in the subsequent
>> run.
>>
>> Jenkins installations from various companies can be combined into a single
>> landing page. Jenkins itself doesn't support master/slave but it does through
>> the gearman plugin. It's something I have tried using with VMs but not with
>> real infra - but it is entirely possible.
>>
>> On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
>> > I think there are jenkins slaves that run the nicera plugins on/at Schuberg
>> > Philis housed infrastructure. The Citrix jenkins nodes also runs as slaves
>> > that connect back to the apache owned/controlled jenkins. No reason why
>> > testing infra need be so consolidated, it just so happens no one is putting
>> > their hardware where their mouth is.
>> >
>> > I also assume if your marvin tests get accepted upstream, they'll be
>> > included in the nightly runs/reports. Prasanna correct me if I'm wrong.
>> >
>> >
>> > On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <sh...@gmail.com>wrote:
>> >
>> > > CloudStack Dev,
>> > > I was emailed about some of the testing questions I brought up
>> > > over the last few threads, and a few things were pointed out to me
>> > > that I think we should try to remedy. Primarily, that the testing
>> > > environment is owned by Citrix, the QA team is primarily Citrix-run,
>> > > and the testing done is focused on the use models that Citrix
>> > > develops.
>> > > I've been assured that the test infrastructure is for everyone,
>> > > and I'm not at all trying to say that there's a problem with Citrix
>> > > focusing their work on their own interests, but I'm not sure that
>> > > anyone outside of Citrix really knows how to add their own stuff to
>> > > this testing infrastructure (perhaps for lack of trying, I don't
>> > > know).
>> > > I haven't really put together enough thought to know how to tackle
>> > > this, but my gut tells me that we need some sort of community-owned
>> > > testing roll-up, where everyone can do their own testing in whatever
>> > > infrastructure and submit hourly, daily, weekly results. If my test
>> > > fits into the Citrix test infrastructure and I can figure out how to
>> > > get it there, great. If not, I can roll my own and integrate it via
>> > > some API. For example the SolidFire guys may wan to run automated
>> > > regression testing. That probably won't be doable in the Citrix
>> > > infrastructure, but they may want to script a daily
>> > > git-pull/build/deploy zone/create volume and it seems logical that
>> > > we'd want to support it.
>> > > Thoughts? Anyone have experience with such things? Can we have a
>> > > master/slave scenario with Jenkins? Perhaps the Citrix environment
>> > > already supports something like this via Jenkins API?
>> > >
>>
>> --
>> Prasanna.,
>>
>> ------------------------
>> Powered by BigRock.com
Re: community testing
Posted by Marcus Sorensen <sh...@gmail.com>.
But if the test requires some sort of preconfiguration, what then (e.g.
test NFS primary storage would need a local or remote NFS configured)? do I
need to roll my own, or can I touch the existing test infra and do the
preconfigure?
On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
> Yes - Once your test goes into the repo, it should get picked in the
> subsequent
> run.
>
> Jenkins installations from various companies can be combined into a single
> landing page. Jenkins itself doesn't support master/slave but it does
> through
> the gearman plugin. It's something I have tried using with VMs but not with
> real infra - but it is entirely possible.
>
> On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
> > I think there are jenkins slaves that run the nicera plugins on/at
> Schuberg
> > Philis housed infrastructure. The Citrix jenkins nodes also runs as
> slaves
> > that connect back to the apache owned/controlled jenkins. No reason why
> > testing infra need be so consolidated, it just so happens no one is
> putting
> > their hardware where their mouth is.
> >
> > I also assume if your marvin tests get accepted upstream, they'll be
> > included in the nightly runs/reports. Prasanna correct me if I'm wrong.
> >
> >
> > On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <shadowsor@gmail.com
> >wrote:
> >
> > > CloudStack Dev,
> > > I was emailed about some of the testing questions I brought up
> > > over the last few threads, and a few things were pointed out to me
> > > that I think we should try to remedy. Primarily, that the testing
> > > environment is owned by Citrix, the QA team is primarily Citrix-run,
> > > and the testing done is focused on the use models that Citrix
> > > develops.
> > > I've been assured that the test infrastructure is for everyone,
> > > and I'm not at all trying to say that there's a problem with Citrix
> > > focusing their work on their own interests, but I'm not sure that
> > > anyone outside of Citrix really knows how to add their own stuff to
> > > this testing infrastructure (perhaps for lack of trying, I don't
> > > know).
> > > I haven't really put together enough thought to know how to tackle
> > > this, but my gut tells me that we need some sort of community-owned
> > > testing roll-up, where everyone can do their own testing in whatever
> > > infrastructure and submit hourly, daily, weekly results. If my test
> > > fits into the Citrix test infrastructure and I can figure out how to
> > > get it there, great. If not, I can roll my own and integrate it via
> > > some API. For example the SolidFire guys may wan to run automated
> > > regression testing. That probably won't be doable in the Citrix
> > > infrastructure, but they may want to script a daily
> > > git-pull/build/deploy zone/create volume and it seems logical that
> > > we'd want to support it.
> > > Thoughts? Anyone have experience with such things? Can we have a
> > > master/slave scenario with Jenkins? Perhaps the Citrix environment
> > > already supports something like this via Jenkins API?
> > >
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
>
Re: community testing
Posted by Prasanna Santhanam <ts...@apache.org>.
Yes - Once your test goes into the repo, it should get picked in the subsequent
run.
Jenkins installations from various companies can be combined into a single
landing page. Jenkins itself doesn't support master/slave but it does through
the gearman plugin. It's something I have tried using with VMs but not with
real infra - but it is entirely possible.
On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote:
> I think there are jenkins slaves that run the nicera plugins on/at Schuberg
> Philis housed infrastructure. The Citrix jenkins nodes also runs as slaves
> that connect back to the apache owned/controlled jenkins. No reason why
> testing infra need be so consolidated, it just so happens no one is putting
> their hardware where their mouth is.
>
> I also assume if your marvin tests get accepted upstream, they'll be
> included in the nightly runs/reports. Prasanna correct me if I'm wrong.
>
>
> On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <sh...@gmail.com>wrote:
>
> > CloudStack Dev,
> > I was emailed about some of the testing questions I brought up
> > over the last few threads, and a few things were pointed out to me
> > that I think we should try to remedy. Primarily, that the testing
> > environment is owned by Citrix, the QA team is primarily Citrix-run,
> > and the testing done is focused on the use models that Citrix
> > develops.
> > I've been assured that the test infrastructure is for everyone,
> > and I'm not at all trying to say that there's a problem with Citrix
> > focusing their work on their own interests, but I'm not sure that
> > anyone outside of Citrix really knows how to add their own stuff to
> > this testing infrastructure (perhaps for lack of trying, I don't
> > know).
> > I haven't really put together enough thought to know how to tackle
> > this, but my gut tells me that we need some sort of community-owned
> > testing roll-up, where everyone can do their own testing in whatever
> > infrastructure and submit hourly, daily, weekly results. If my test
> > fits into the Citrix test infrastructure and I can figure out how to
> > get it there, great. If not, I can roll my own and integrate it via
> > some API. For example the SolidFire guys may wan to run automated
> > regression testing. That probably won't be doable in the Citrix
> > infrastructure, but they may want to script a daily
> > git-pull/build/deploy zone/create volume and it seems logical that
> > we'd want to support it.
> > Thoughts? Anyone have experience with such things? Can we have a
> > master/slave scenario with Jenkins? Perhaps the Citrix environment
> > already supports something like this via Jenkins API?
> >
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: community testing
Posted by Ahmad Emneina <ae...@gmail.com>.
I think there are jenkins slaves that run the nicera plugins on/at Schuberg
Philis housed infrastructure. The Citrix jenkins nodes also runs as slaves
that connect back to the apache owned/controlled jenkins. No reason why
testing infra need be so consolidated, it just so happens no one is putting
their hardware where their mouth is.
I also assume if your marvin tests get accepted upstream, they'll be
included in the nightly runs/reports. Prasanna correct me if I'm wrong.
On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen <sh...@gmail.com>wrote:
> CloudStack Dev,
> I was emailed about some of the testing questions I brought up
> over the last few threads, and a few things were pointed out to me
> that I think we should try to remedy. Primarily, that the testing
> environment is owned by Citrix, the QA team is primarily Citrix-run,
> and the testing done is focused on the use models that Citrix
> develops.
> I've been assured that the test infrastructure is for everyone,
> and I'm not at all trying to say that there's a problem with Citrix
> focusing their work on their own interests, but I'm not sure that
> anyone outside of Citrix really knows how to add their own stuff to
> this testing infrastructure (perhaps for lack of trying, I don't
> know).
> I haven't really put together enough thought to know how to tackle
> this, but my gut tells me that we need some sort of community-owned
> testing roll-up, where everyone can do their own testing in whatever
> infrastructure and submit hourly, daily, weekly results. If my test
> fits into the Citrix test infrastructure and I can figure out how to
> get it there, great. If not, I can roll my own and integrate it via
> some API. For example the SolidFire guys may wan to run automated
> regression testing. That probably won't be doable in the Citrix
> infrastructure, but they may want to script a daily
> git-pull/build/deploy zone/create volume and it seems logical that
> we'd want to support it.
> Thoughts? Anyone have experience with such things? Can we have a
> master/slave scenario with Jenkins? Perhaps the Citrix environment
> already supports something like this via Jenkins API?
>