You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Prasanna Santhanam <ts...@apache.org> on 2013/10/02 19:14:27 UTC
[DISCUSS] Breaking out Marvin from CloudStack
I would like to seperate marvin from the main cloudstack repo. Much of
marvin's development has little coupling with CloudStack.
Similar to CloudMonkey, marvin undergoes rapid changes and it is
essential to provide a smooth workflow and faster releases for those
working with it.
There are also a small set of people currently looking at marvin for
testing right now. Often, their reviews and QA effort is mixed with
those of cloudstack itself. By having a different repo I'd like to be
able to provide commit access to those working on marvin alone quickly
to help with testing.
After separating marvin
0. we will have a separate release cycle for marvin
1. we will have a new home for marvin's docs using Sphinx
2. if possible, a different criteria for providing commit access to
marvin's repos.
3. all tests of cloudstack will also move to marvin's repository
Thoughts?
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: [DISCUSS] Breaking out Marvin from CloudStack
Posted by David Nalley <da...@gnsa.us>.
> After separating marvin
> 0. we will have a separate release cycle for marvin
> 1. we will have a new home for marvin's docs using Sphinx
> 2. if possible, a different criteria for providing commit access to
> marvin's repos.
> 3. all tests of cloudstack will also move to marvin's repository
>
I personally am +1 on breaking it out because of 0 and 3. A couple of
comments though:
1. What would the home for Marvin docs be?
2. At present, the way git authz works is based on being a committer
for CloudStack - so even if we created a cloudstack-marvin repo, we
wouldn't have any way of providing access only to cloudstack-marvin.
Also being a committer is about more than getting commit privs to a
repo. Speaking only for myself, I don't see an easy way for this to
change culturally or technologically.
--David
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Sudha Ponnaganti <su...@citrix.com>.
I assume changes are being done on a separate branch. We are currently going to run automation on Master for 4.3. Just want to make sure that this would not impact current runs.
-----Original Message-----
From: Sebastien Goasguen [mailto:runseb@gmail.com]
Sent: Wednesday, October 02, 2013 11:26 AM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Breaking out Marvin from CloudStack
+1
On Oct 2, 2013, at 1:58 PM, Edison Su <Ed...@citrix.com> wrote:
> Seems Marvin won't depend on commands.xml any more, thus won't depend on any artifacts build from cloudstack. So I am +1.
>
>> -----Original Message-----
>> From: Prasanna Santhanam [mailto:tsp@apache.org]
>> Sent: Wednesday, October 02, 2013 10:14 AM
>> To: CloudStack Dev
>> Subject: [DISCUSS] Breaking out Marvin from CloudStack
>>
>> I would like to seperate marvin from the main cloudstack repo. Much
>> of marvin's development has little coupling with CloudStack.
>>
>> Similar to CloudMonkey, marvin undergoes rapid changes and it is
>> essential to provide a smooth workflow and faster releases for those working with it.
>>
>> There are also a small set of people currently looking at marvin for
>> testing right now. Often, their reviews and QA effort is mixed with
>> those of cloudstack itself. By having a different repo I'd like to be
>> able to provide commit access to those working on marvin alone quickly to help with testing.
>>
>> After separating marvin
>> 0. we will have a separate release cycle for marvin 1. we will have a
>> new home for marvin's docs using Sphinx 2. if possible, a different
>> criteria for providing commit access to marvin's repos.
>> 3. all tests of cloudstack will also move to marvin's repository
>>
>> Thoughts?
>>
>> --
>> Prasanna.,
>>
>> ------------------------
>> Powered by BigRock.com
>
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Santhosh Edukulla <sa...@citrix.com>.
+1.
All libraries of marvin and tests using them will now be at single place. Though marvin and tests under it are used to test cloudstack, segregating it from main cloudstack product code is good. Going ahead, few more changes will be required to make tests run automatically based upon the cloudstack release version on a given test bed, but that can be enhanced separately.
Regards,
Santhosh
________________________________________
From: Sebastien Goasguen [runseb@gmail.com]
Sent: Wednesday, October 02, 2013 2:25 PM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Breaking out Marvin from CloudStack
+1
On Oct 2, 2013, at 1:58 PM, Edison Su <Ed...@citrix.com> wrote:
> Seems Marvin won't depend on commands.xml any more, thus won't depend on any artifacts build from cloudstack. So I am +1.
>
>> -----Original Message-----
>> From: Prasanna Santhanam [mailto:tsp@apache.org]
>> Sent: Wednesday, October 02, 2013 10:14 AM
>> To: CloudStack Dev
>> Subject: [DISCUSS] Breaking out Marvin from CloudStack
>>
>> I would like to seperate marvin from the main cloudstack repo. Much of
>> marvin's development has little coupling with CloudStack.
>>
>> Similar to CloudMonkey, marvin undergoes rapid changes and it is essential
>> to provide a smooth workflow and faster releases for those working with it.
>>
>> There are also a small set of people currently looking at marvin for testing
>> right now. Often, their reviews and QA effort is mixed with those of
>> cloudstack itself. By having a different repo I'd like to be able to provide
>> commit access to those working on marvin alone quickly to help with testing.
>>
>> After separating marvin
>> 0. we will have a separate release cycle for marvin 1. we will have a new
>> home for marvin's docs using Sphinx 2. if possible, a different criteria for
>> providing commit access to marvin's repos.
>> 3. all tests of cloudstack will also move to marvin's repository
>>
>> Thoughts?
>>
>> --
>> Prasanna.,
>>
>> ------------------------
>> Powered by BigRock.com
>
Re: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Sebastien Goasguen <ru...@gmail.com>.
+1
On Oct 2, 2013, at 1:58 PM, Edison Su <Ed...@citrix.com> wrote:
> Seems Marvin won't depend on commands.xml any more, thus won't depend on any artifacts build from cloudstack. So I am +1.
>
>> -----Original Message-----
>> From: Prasanna Santhanam [mailto:tsp@apache.org]
>> Sent: Wednesday, October 02, 2013 10:14 AM
>> To: CloudStack Dev
>> Subject: [DISCUSS] Breaking out Marvin from CloudStack
>>
>> I would like to seperate marvin from the main cloudstack repo. Much of
>> marvin's development has little coupling with CloudStack.
>>
>> Similar to CloudMonkey, marvin undergoes rapid changes and it is essential
>> to provide a smooth workflow and faster releases for those working with it.
>>
>> There are also a small set of people currently looking at marvin for testing
>> right now. Often, their reviews and QA effort is mixed with those of
>> cloudstack itself. By having a different repo I'd like to be able to provide
>> commit access to those working on marvin alone quickly to help with testing.
>>
>> After separating marvin
>> 0. we will have a separate release cycle for marvin 1. we will have a new
>> home for marvin's docs using Sphinx 2. if possible, a different criteria for
>> providing commit access to marvin's repos.
>> 3. all tests of cloudstack will also move to marvin's repository
>>
>> Thoughts?
>>
>> --
>> Prasanna.,
>>
>> ------------------------
>> Powered by BigRock.com
>
Re: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Marcus Sorensen <sh...@gmail.com>.
Isn't there marvin code that's specific to the cloudstack API calls? I
just remember having to go in and add to marvin python code when we
wanted to test certain new cloudstack functionality. I suppose if the
tests themselves are moving as well then it might not be a big deal,
presumably the tests will only support what the accompanying marvin
code does. Still, it might end up tricky to keep track of which tests
are intended for which version of CloudStack, unless we can create (or
already have) version dependencies in the tests.
On Wed, Oct 2, 2013 at 11:58 AM, Edison Su <Ed...@citrix.com> wrote:
> Seems Marvin won't depend on commands.xml any more, thus won't depend on any artifacts build from cloudstack. So I am +1.
>
>> -----Original Message-----
>> From: Prasanna Santhanam [mailto:tsp@apache.org]
>> Sent: Wednesday, October 02, 2013 10:14 AM
>> To: CloudStack Dev
>> Subject: [DISCUSS] Breaking out Marvin from CloudStack
>>
>> I would like to seperate marvin from the main cloudstack repo. Much of
>> marvin's development has little coupling with CloudStack.
>>
>> Similar to CloudMonkey, marvin undergoes rapid changes and it is essential
>> to provide a smooth workflow and faster releases for those working with it.
>>
>> There are also a small set of people currently looking at marvin for testing
>> right now. Often, their reviews and QA effort is mixed with those of
>> cloudstack itself. By having a different repo I'd like to be able to provide
>> commit access to those working on marvin alone quickly to help with testing.
>>
>> After separating marvin
>> 0. we will have a separate release cycle for marvin 1. we will have a new
>> home for marvin's docs using Sphinx 2. if possible, a different criteria for
>> providing commit access to marvin's repos.
>> 3. all tests of cloudstack will also move to marvin's repository
>>
>> Thoughts?
>>
>> --
>> Prasanna.,
>>
>> ------------------------
>> Powered by BigRock.com
>
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Edison Su <Ed...@citrix.com>.
Seems Marvin won't depend on commands.xml any more, thus won't depend on any artifacts build from cloudstack. So I am +1.
> -----Original Message-----
> From: Prasanna Santhanam [mailto:tsp@apache.org]
> Sent: Wednesday, October 02, 2013 10:14 AM
> To: CloudStack Dev
> Subject: [DISCUSS] Breaking out Marvin from CloudStack
>
> I would like to seperate marvin from the main cloudstack repo. Much of
> marvin's development has little coupling with CloudStack.
>
> Similar to CloudMonkey, marvin undergoes rapid changes and it is essential
> to provide a smooth workflow and faster releases for those working with it.
>
> There are also a small set of people currently looking at marvin for testing
> right now. Often, their reviews and QA effort is mixed with those of
> cloudstack itself. By having a different repo I'd like to be able to provide
> commit access to those working on marvin alone quickly to help with testing.
>
> After separating marvin
> 0. we will have a separate release cycle for marvin 1. we will have a new
> home for marvin's docs using Sphinx 2. if possible, a different criteria for
> providing commit access to marvin's repos.
> 3. all tests of cloudstack will also move to marvin's repository
>
> Thoughts?
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
Re: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Prasanna Santhanam <ts...@apache.org>.
All - thanks for your detailed thoughts on this so far. It seems like
this should've come with a fair bit of notice. I will hold on this
separation for now. May be a couple more releases to see the level of
participation in the QA and if it still feels like a hindrance we'll
discuss separating it later. For now I think it should suffice to
reorganize the current folder structure of marvin to be able to find
all its bits at a single place. This will be proposed separately.
I am closing this DISCUSS thread as of now by not splitting out
marvin.
I will however address some additional points which were raised so
far. Inline.
On Fri, Oct 04, 2013 at 04:52:45PM -0400, Chip Childers wrote:
>
> Fair warning - some of this is a straw man argument to explore the
> situation, and a little bit of ranting at the end.
>
> On Fri, Oct 04, 2013 at 05:57:58PM +0530, Prasanna Santhanam wrote:
> > I'll summarize and address the concerns raised so far.
> >
> > Marvin has been in this repo for a long time for us to start writing
> > tests. The only tests I've seen coming are from a specific set of
> > people focussed on QA efforts.
>
> I agree - and that's a problem. New features should *ALL* have tests
> before they merge into master. I think that assuming that the only test
> writers are a group of folks that write the tests today is actually a
> larger problem.
>
It's a bit more prevalent than test cases alone. Docs for eg are
written by doc writers and there's a whole bit of process of doc
review and someone writing the stuff in an email and then someone else
going and correcting it. If you have repo access and notice a problem,
please help fix it as well and not create additional process. I'm not
for such segmentation but there are roles in a corporate setup that
are probably causing this presumption that marvin tests are written by
QA alone.
> > I want to reduce the impediment for
> > people who are writing tests *today*. Those looking to get started in
> > the near future won't have any new learning to do, just that their
> > code goes in an alternate repo that is pointed to right
> > infrastructure.
> >
> > Automated testing also works in a push-to-production style very often.
> > Testers need to run their tests on a deployed environment(s) quickly
> > to be able to ensure it is valid and passes. By making them go through
> > reviewboard each time for each test we massively slow down the
> > process. (tons of fixes to tests are on rb today, not just new tests).
> > We don't know if they run until they run on the environment.
>
> I want to be clear about this part - a different repo doesn't change the
> need for someone to be a committer to commit.
Yes - I thought this would be a problem as David also mentioned
earlier. There are no ACLs within a project for controlling repo
access. Except if it was possible, I would've looked at providing
access faster to those who are contributing.
> >
> > Reason for tests and framework to go together is simple. If I go look
> > at the jclouds repository today I find tests for rackspace cloud,
> > openstack cloud, cloudstack cloud, euca clouds in the jclouds
> > repository and not in the respective provider/project repository. A
> > newcomer to the marvin repository will be someone interested in
> > writing tests and he will also thus be able to find tests in the
> > marvin repository.
> >
> > This also allows for more heterogenous testing of cloudstack. No one
> > needs to be tied down to a framework / tool to write integration
> > tests. If python is not your forte, use Chip's ruby client, or perhaps
> > in the near future Chiradeep's stackmate to write your test, or even
> > jclouds.
>
> But that's actually true today, right? I mean if I wanted to write an
> integration test using some other method, I'd do that... but would it
> be useful for others? Probably not! That's because the way that we do
> testing of this type is via Marvin. The Citrix infra wouldn't be setup
> for whatever other framework I used, and the community as a whole would
> get less benefit than if I was consistent.
>
Somehow I sense that as a problem of our test writing. The tests are
written to assume a certain infrastructure. But a lot of the API is
also admin only which requires certain infra to be in place. The other
tools jclouds live tests for example cannot assume admin access to the
cloud because they are only testing the user api.
Tests written by other tools would only be provided an endpoint and
API/Secret keys and test whatever they can. Specific tools required
for those tests can be provisioned automatically on the machine that
runs the tests.
> >
> > Now the question of supporting older version of marvin against newer
> > versions of cloudstack. Marvin now fully auto-generates itself (see
> > the design in the proposal) based on endpoint. So you have the
> > marvin version that will work with your endpoint only. As for being
> > backwards compatible (also addressed in the design doc) - no old tests
> > are broken, they will still run perfectly fine.
> >
>
> That's marvin, not the tests, right? If I add a new feature for 4.3,
> should that test appear in a 4.2 run? Aren't we aiming for (I know
> we're not there) a situation where *all* tests pass before we release?
> IMO not versioning the tests (not the Marvin framework) with the target
> of the tests is confusing.
We would mirror the branches of ACS release with the marvin repo to
control what gets tested against what. Also, If an API is not
available on a certain version of CloudStack and the test requires it,
we can have the framework skip such a test.
>
> > The infrastructure (currently) only looks at the changes in the test
> > directory before performing a run. It doesn't care whether server/ was
> > changed or plugins/x/y/z was changed. That's because the tests are
> > unrelated to what is in the rest of the repository. In fact you can't
> > even run them without a deployed cloud. So I don't see why idle code
> > should lie in the repo.
> >
> > Integration tests are essential, they will keep coming as long as
> > Citrix QA is invested in the effort, but they need to come faster into
> > the repos and that will be addressed by the separation IMO.
>
> How?
Rewarding committership quickly. non sequitir given we cannot control
who gets committership on which repo within a single project.
> >
> > What I would like to see provided by CloudStack is the ability to
> > upgrade all our test environments, staging environments, UATs, what
> > have you in continuous integration and have tests run on the
> > upgraded setup. That allows incrementally testing CloudStack the way
> > users do it. The current design of installing everything from scratch,
> > redoing the testbed for each automated test run is mostly a workaround
> > for that inability. If we had this ability tests written in marvin can
> > be run against live setups at all times as and when features merge to
> > master.
>
> This is a valid argument worth exploring. However, it doesn't require a
> separate repo, right? I mean, the repo could be cloned twice if
> necessary.
>
There is a different sense I'm getting off late where large refactors
are done without enough unit-tests and we bank on our bvt to allow for
a merge. The project requires both unittests and integration tests IMO.
--
Prasanna.,
------------------------
Powered by BigRock.com
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Animesh Chaturvedi <an...@citrix.com>.
> -----Original Message-----
> From: Chip Childers [mailto:chip.childers@sungard.com]
> Sent: Friday, October 04, 2013 1:53 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [DISCUSS] Breaking out Marvin from CloudStack
>
>
> Fair warning - some of this is a straw man argument to explore the
> situation, and a little bit of ranting at the end.
>
> On Fri, Oct 04, 2013 at 05:57:58PM +0530, Prasanna Santhanam wrote:
> > I'll summarize and address the concerns raised so far.
> >
> > Marvin has been in this repo for a long time for us to start writing
> > tests. The only tests I've seen coming are from a specific set of
> > people focussed on QA efforts.
>
> I agree - and that's a problem. New features should *ALL* have tests
> before they merge into master. I think that assuming that the only test
> writers are a group of folks that write the tests today is actually a
> larger problem.
>
> > I want to reduce the impediment for
> > people who are writing tests *today*. Those looking to get started in
> > the near future won't have any new learning to do, just that their
> > code goes in an alternate repo that is pointed to right
> > infrastructure.
> >
> > Automated testing also works in a push-to-production style very often.
> > Testers need to run their tests on a deployed environment(s) quickly
> > to be able to ensure it is valid and passes. By making them go through
> > reviewboard each time for each test we massively slow down the
> > process. (tons of fixes to tests are on rb today, not just new tests).
> > We don't know if they run until they run on the environment.
>
> I want to be clear about this part - a different repo doesn't change the
> need for someone to be a committer to commit.
>
[Animesh>] Test code contribution is similar to code contribution and the issue is not reviewboard but folks not responding to reviews. We need to address them as community. If we have to get new committers we should bring them forward.
> There's a thread on private@ that you should weigh in on here. If we
> have more people that should be committers, then let's get *that* done.
>
> >
> > Reason for tests and framework to go together is simple. If I go look
> > at the jclouds repository today I find tests for rackspace cloud,
> > openstack cloud, cloudstack cloud, euca clouds in the jclouds
> > repository and not in the respective provider/project repository. A
> > newcomer to the marvin repository will be someone interested in
> > writing tests and he will also thus be able to find tests in the
> > marvin repository.
> >
> > This also allows for more heterogenous testing of cloudstack. No one
> > needs to be tied down to a framework / tool to write integration
> > tests. If python is not your forte, use Chip's ruby client, or perhaps
> > in the near future Chiradeep's stackmate to write your test, or even
> > jclouds.
>
> But that's actually true today, right? I mean if I wanted to write an
> integration test using some other method, I'd do that... but would it
> be useful for others? Probably not! That's because the way that we do
> testing of this type is via Marvin. The Citrix infra wouldn't be setup
> for whatever other framework I used, and the community as a whole would
> get less benefit than if I was consistent.
>
[Animesh>] I agree having consistent tooling shortens the learning cycle for new contributors
> >
> > Now the question of supporting older version of marvin against newer
> > versions of cloudstack. Marvin now fully auto-generates itself (see
> > the design in the proposal) based on endpoint. So you have the marvin
> > version that will work with your endpoint only. As for being backwards
> > compatible (also addressed in the design doc) - no old tests are
> > broken, they will still run perfectly fine.
> >
>
> That's marvin, not the tests, right? If I add a new feature for 4.3,
> should that test appear in a 4.2 run? Aren't we aiming for (I know
> we're not there) a situation where *all* tests pass before we release?
> IMO not versioning the tests (not the Marvin framework) with the target
> of the tests is confusing.
>
[Animesh>] IMO "Things that change together, should live together", separate repo for tests does not sit well
> > The infrastructure (currently) only looks at the changes in the test
> > directory before performing a run. It doesn't care whether server/ was
> > changed or plugins/x/y/z was changed. That's because the tests are
> > unrelated to what is in the rest of the repository. In fact you can't
> > even run them without a deployed cloud. So I don't see why idle code
> > should lie in the repo.
> >
> > Integration tests are essential, they will keep coming as long as
> > Citrix QA is invested in the effort, but they need to come faster into
> > the repos and that will be addressed by the separation IMO.
>
> How?
[Animesh>] Like Chip, I don't see breaking tests into separate repo will increase their velocity.
>
> > Managing
> > the feature submitted to cloudstack against tests submitted to marvin
> > is not a hard thing to do. We simply mirror the release branches in
> > marvin and submit tests there. In fact I wonder why we didn't have
> > this question when docs were separated? It doesn't work any
> > differently really.
>
> Well, it turns out that docs are absolutely not being done before code
> comes into master. I dislike this fact, but live with it.
>
> >
> > What I would like to see provided by CloudStack is the ability to
> > upgrade all our test environments, staging environments, UATs, what
> > have you in continuous integration and have tests run on the upgraded
> > setup. That allows incrementally testing CloudStack the way users do
> > it. The current design of installing everything from scratch, redoing
> > the testbed for each automated test run is mostly a workaround for
> > that inability. If we had this ability tests written in marvin can be
> > run against live setups at all times as and when features merge to
> > master.
>
> This is a valid argument worth exploring. However, it doesn't require a
> separate repo, right? I mean, the repo could be cloned twice if
> necessary.
>
> >
> > It has always been my goal to reduce the barrier to writing tests for
> > cloudstack and that's entirely the goal of the refactor as well. So I
> > hope I have spent my time well. Since I can't be after people to write
> > tests, and setup mandates, I aim to address the technical difficulty
> > in writing tests through this refactor and separation.
>
> Some of my arguments above are, frankly, straw man arguments. I see
> many of your points as being valid as well. Don't take my perspective
> as being against any of your goals, just as discussion points about the
> right way to make the goals work.
>
> I'll say one thing though... I'm exceptionally frustrated with the
> features / commits coming into master with little to no tests (unit OR
> integration). I'm at a loss at this point, and frankly considering a
> possible reality that this community (generally) just doesn't care about
> making it easy to reproduce tests. The lack of interest and ignorance
> about the pain that this causes is infuriating to me.
>
> Prasanna, I can tell that you have some / similar frustrations (but I
> won't put the words into your mouth), and are trying to find ways to
> make it easier for others to step in and help out where feature
> developers are apparently showing no interest. This is an admirable
> goal.
>
> So... perhaps we have to live with the complete lack of interest from
> the feature devs, and take the approach of breaking out the code. That
> doesn't make *access permission* easier, but it does make a target repo
> easier to focus on test scenarios.
>
> As I said above, let's talk on private@ about people that are doing the
> tests that deserve the commit bit. At least we can help them help
> everyone else...
>
> >
> > PS: David, by hosted docs I mean something like
> > http://pythonhosted.org/cloudmonkey. Lives in the repo but read from
> > the webpage.
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
> >
> >
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Santhosh Edukulla [mailto:santhosh.edukulla@citrix.com]
> Sent: Tuesday, October 08, 2013 1:28 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [DISCUSS] Breaking out Marvin from CloudStack
>
> Comments Inline.
>
> -----Original Message-----
> From: Edison Su [mailto:Edison.su@citrix.com]
> Sent: Tuesday, October 08, 2013 4:18 AM
> To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> Subject: RE: [DISCUSS] Breaking out Marvin from CloudStack
>
> Few questions:
> 1. About the "more object-oriented" CloudStack API python binding: Is the
> proposed api good enough?
> For example,
> The current hand written create virtual machine looks like:
> class VirtualMachine(object):
> ....
> @classmethod
> def create(cls, apiclient, services, templateid=None, accountid=None,
> domainid=None, zoneid=None, networkids=None,
> serviceofferingid=None,
> securitygroupids=None, projectid=None, startvm=None,
> diskofferingid=None, affinitygroupnames=None, group=None,
> hostid=None, keypair=None, mode='basic', method='GET'):
>
> the proposed api may look like:
>
> class VirtualMachine(object):
> def create(self, apiclient, accountId, templateId, **kwargs)
>
> The proposed api will look better than previous one, and it's automatically
> generated, so easy to maintain. But as a consumer of the api, how do people
> know what kind of parameters should be passed in? Will you have an online
> document for your api? Or you assume people will look at the api docs
> generated by CloudStack?
> Or why not make the api itself as self-contained? For example, add docs
> before create method:
>
> class VirtualMachine(object):
> '''
> Args:
> accountId: what ever
> templateId: whatever
> networkids: whatever
> '''
> '''
> Response:
> '''
> def create(self, apiclient, accountId, templateId, **kwargs)
>
> All the api documents should be included in api discovery already, so it should
> be easy to add them in your api binding.
> >> [Santhosh]: Each verb as an action on entity, will have provision as earlier
> to have all required and as well optional arguments. Regarding doc strings, If
> the API docs are having this facilitation, we will add them as corresponding
> doc strings during generation of python binding and as well entities. As you
> rightly mentioned, it will good to add this . We will make sure to get it.
> Adding adequate doc strings applies even while writing test feature\lib as
> well, it will improve ease ,readability,usage etc. Anyways a wiki page, and
> additional pydoc documents posted online will be there.
That's great! The way you separate required parameters from optional parameters on the method signature, is quite a good idea.
>
> 2. Regarding to data factories. From the proposed factories, in each test case,
> does test writer still need to write the code to get data, such as writing code
> to get account during the setupclass?
> I looked at some of the existing test cases, most of them have the same code
> snippet:
> class Services:
> def __init__(self):
> self.services = {
> "account": {
> "email": "test@test.com<ma...@test.com>",
> "firstname": "Test",
> "lastname": "User",
> "username": "test",
> "password": "password",
> },
> "virtual_machine": {
> "displayname": "Test VM",
> "username": "root",
> "password": "password",
> "ssh_port": 22,
> "hypervisor": 'XenServer',
> "privateport": 22,
> "publicport": 22,
> "protocol": 'TCP',
> },
>
> With the data factories, the code will look like the following?
>
> Class TestFoo:
> Def setupClass():
> Account = UserAccount(apiclient)
> VM = UserVM(apiClient)
>
> And if I want to customize the default data factories, I should be able to use
> something like: UserAccount(apiclient, username='myfoo')?
> And the data factories should be able to customized based on test
> environment, right?
> For example, the current iso test cases are hardcoded to test against
> http://people.apache.org/~tsp/dummy.iso, but it won't work for devcloud,
> or in an internal network. The ISO data factory should be able to return an url
> based on different test environment, thus iso test cases can be reused.
>
> >>[Santhosh] : Currently, as you mentioned, Services class is part of many
> test modules, this is basically data part for the test. We are separating this
> with factory approach. Thus, segregating data from test. Compare the earlier
> mention of Services class in earlier test code without Service class in the
> below test code.
>
> class TestVpcLifeCycle(cloudstackTestCase):
>
> def setUp(self):
> self.apiclient = super(TestVpcLifeCycle,
> self).getClsTestClient().getApiClient()
> self.zoneid = get_zone(self.apiclient).id
> self.templateid = get_template(self.apiclient).id
> self.serviceofferingid = get_service_offering(self.apiclient).id
> self.account = UserAccount( apiclient=self.apiclient ) ---> Data factory
> creation
>
>
> @attr(tags='debug')
> def test_deployvm(self):
> vm = VpcVirtualMachine(
> apiclient=self.apiclient,
> account=self.account.name,
> domainid=self.account.domainid,
> zoneid=self.zoneid,
> templateid=self.templateid,
> serviceofferingid=self.serviceofferingid
> )
> vm.state | should | equal_to('Running')
>
> There could be many ways of data driving\In this case providing simple data
> to the test cases. Factory approach though better is driven to create factories
> with default sequences once and reused. Yes, user can customize a data
> factory as you mentioned. customizations specific to environment are not
> available though, but doable while creating a factory instance. We can take a
> care to look at it. Basic data factory for an entity by default will have values
> based upon environment available and then user can customize accordingly,
> this can be added to proposal as edit. But using factories logic, do you see
> any issues as a whole in long run? Any disadvantages you foresee? Or using
> plain data driving approach through sample xml\csv file for a given test
> module should suffice i.e., provide a facilitation as part of current testclient
> itself as dynamic data object creation as similar to the one provided by db
> connection as part of test client currently, the new data object therefore as
> part of testclient will then replace the current Service Class\object creation
> for a given test module. Loading the values from xml\csv in to data object for
> test can be controlled through a config value( path to data values file(xml\csv)
> for a given test module ) and are loaded during the object creation? This
> object through testclient is available for test module to be used as a whole?
To me, a simple data driven approach sounds suffice. For example, anything like
Get_iso_url() which will return an usable url from a json based test data configuration file,
Or get_account() which will return an default account, is much better than put the data in the test cases itself.
Just keep in mind, whenever their general requirements in each test, put them in either a helper library, or inject them into each test case.
>
> Thanks!
> Santhosh
> > -----Original Message-----
> > From: Santhosh Edukulla [mailto:santhosh.edukulla@citrix.com]
> > Sent: Monday, October 07, 2013 7:06 AM
> > To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> > Subject: RE: [DISCUSS] Breaking out Marvin from CloudStack
> >
> > Team,
> >
> > Apart\Away from breaking out marvin from cloudstack, please check the
> > other new details provided as part of the new proposal for marvin
> > refactoring. Your inputs out of experience are invaluable. Any new
> > feature tests for CS will be followed with the new approach, provided
> > if we agree to all. Pasting the proposal link one more time below.
> >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+Refactor
> >
> > Regards,
> > Santhosh
> > ________________________________________
> > From: Daan Hoogland [daan.hoogland@gmail.com]
> > Sent: Sunday, October 06, 2013 3:05 PM
> > To: dev
> > Subject: Re: [DISCUSS] Breaking out Marvin from CloudStack
> >
> > On Sun, Oct 6, 2013 at 8:10 PM, Animesh Chaturvedi <
> > animesh.chaturvedi@citrix.com<ma...@citrix.com>>
> wrote:
> >
> > > > Yes and we will need to work down a backlog of scenarios before we
> > > > ever can rely on guys like me doing that. Not because they won't
> > > > but because there is to much to write tests for edging on the new
> > > > features they write. Just because those tests aren't there yet. I
> > > > think giving Citrix QA a repo to work on is fine but I would like
> > > > to see it merged back at some point and a continued possibility to
> > > > write
> > them in the main tree.
> > > >
> > > [Animesh>] While I don't agree to a separate repo for tests (marvin
> > > framework is ok) I do want to call out the proposal is not for
> > > giving Citrix QA a repo to work on and I don't think Prasanna meant that
> way.
> > >
> >
> >
> > I have to apologize for the formulations I choose to express my
> > thoughts with. I did not mean to talk of a department of a certain
> > company donating human effort for testing to the community. I was
> > talking of the frustration of the individuals working and how the
> > separate repo would smoothen their workflow. The new repo is in the
> > apache domain, no question whether the work in their is done by one
> person or 100 companies.
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Santhosh Edukulla <sa...@citrix.com>.
Comments Inline.
-----Original Message-----
From: Edison Su [mailto:Edison.su@citrix.com]
Sent: Tuesday, October 08, 2013 4:18 AM
To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
Subject: RE: [DISCUSS] Breaking out Marvin from CloudStack
Few questions:
1. About the "more object-oriented" CloudStack API python binding: Is the proposed api good enough?
For example,
The current hand written create virtual machine looks like:
class VirtualMachine(object):
....
@classmethod
def create(cls, apiclient, services, templateid=None, accountid=None,
domainid=None, zoneid=None, networkids=None, serviceofferingid=None,
securitygroupids=None, projectid=None, startvm=None,
diskofferingid=None, affinitygroupnames=None, group=None,
hostid=None, keypair=None, mode='basic', method='GET'):
the proposed api may look like:
class VirtualMachine(object):
def create(self, apiclient, accountId, templateId, **kwargs)
The proposed api will look better than previous one, and it's automatically generated, so easy to maintain. But as a consumer of the api, how do people know what kind of parameters should be passed in? Will you have an online document for your api? Or you assume people will look at the api docs generated by CloudStack?
Or why not make the api itself as self-contained? For example, add docs before create method:
class VirtualMachine(object):
'''
Args:
accountId: what ever
templateId: whatever
networkids: whatever
'''
'''
Response:
'''
def create(self, apiclient, accountId, templateId, **kwargs)
All the api documents should be included in api discovery already, so it should be easy to add them in your api binding.
>> [Santhosh]: Each verb as an action on entity, will have provision as earlier to have all required and as well optional arguments. Regarding doc strings, If the API docs are having this facilitation, we will add them as corresponding doc strings during generation of python binding and as well entities. As you rightly mentioned, it will good to add this . We will make sure to get it. Adding adequate doc strings applies even while writing test feature\lib as well, it will improve ease ,readability,usage etc. Anyways a wiki page, and additional pydoc documents posted online will be there.
2. Regarding to data factories. From the proposed factories, in each test case, does test writer still need to write the code to get data, such as writing code to get account during the setupclass?
I looked at some of the existing test cases, most of them have the same code snippet:
class Services:
def __init__(self):
self.services = {
"account": {
"email": "test@test.com<ma...@test.com>",
"firstname": "Test",
"lastname": "User",
"username": "test",
"password": "password",
},
"virtual_machine": {
"displayname": "Test VM",
"username": "root",
"password": "password",
"ssh_port": 22,
"hypervisor": 'XenServer',
"privateport": 22,
"publicport": 22,
"protocol": 'TCP',
},
With the data factories, the code will look like the following?
Class TestFoo:
Def setupClass():
Account = UserAccount(apiclient)
VM = UserVM(apiClient)
And if I want to customize the default data factories, I should be able to use something like: UserAccount(apiclient, username='myfoo')?
And the data factories should be able to customized based on test environment, right?
For example, the current iso test cases are hardcoded to test against http://people.apache.org/~tsp/dummy.iso, but it won't work for devcloud, or in an internal network. The ISO data factory should be able to return an url based on different test environment, thus iso test cases can be reused.
>>[Santhosh] : Currently, as you mentioned, Services class is part of many test modules, this is basically data part for the test. We are separating this with factory approach. Thus, segregating data from test. Compare the earlier mention of Services class in earlier test code without Service class in the below test code.
class TestVpcLifeCycle(cloudstackTestCase):
def setUp(self):
self.apiclient = super(TestVpcLifeCycle, self).getClsTestClient().getApiClient()
self.zoneid = get_zone(self.apiclient).id
self.templateid = get_template(self.apiclient).id
self.serviceofferingid = get_service_offering(self.apiclient).id
self.account = UserAccount( apiclient=self.apiclient ) ---> Data factory creation
@attr(tags='debug')
def test_deployvm(self):
vm = VpcVirtualMachine(
apiclient=self.apiclient,
account=self.account.name,
domainid=self.account.domainid,
zoneid=self.zoneid,
templateid=self.templateid,
serviceofferingid=self.serviceofferingid
)
vm.state | should | equal_to('Running')
There could be many ways of data driving\In this case providing simple data to the test cases. Factory approach though better is driven to create factories with default sequences once and reused. Yes, user can customize a data factory as you mentioned. customizations specific to environment are not available though, but doable while creating a factory instance. We can take a care to look at it. Basic data factory for an entity by default will have values based upon environment available and then user can customize accordingly, this can be added to proposal as edit. But using factories logic, do you see any issues as a whole in long run? Any disadvantages you foresee? Or using plain data driving approach through sample xml\csv file for a given test module should suffice i.e., provide a facilitation as part of current testclient itself as dynamic data object creation as similar to the one provided by db connection as part of test client currently, the new data object therefore as part of testclient will then replace the current Service Class\object creation for a given test module. Loading the values from xml\csv in to data object for test can be controlled through a config value( path to data values file(xml\csv) for a given test module ) and are loaded during the object creation? This object through testclient is available for test module to be used as a whole?
Thanks!
Santhosh
> -----Original Message-----
> From: Santhosh Edukulla [mailto:santhosh.edukulla@citrix.com]
> Sent: Monday, October 07, 2013 7:06 AM
> To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> Subject: RE: [DISCUSS] Breaking out Marvin from CloudStack
>
> Team,
>
> Apart\Away from breaking out marvin from cloudstack, please check the
> other new details provided as part of the new proposal for marvin
> refactoring. Your inputs out of experience are invaluable. Any new
> feature tests for CS will be followed with the new approach, provided
> if we agree to all. Pasting the proposal link one more time below.
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+Refactor
>
> Regards,
> Santhosh
> ________________________________________
> From: Daan Hoogland [daan.hoogland@gmail.com]
> Sent: Sunday, October 06, 2013 3:05 PM
> To: dev
> Subject: Re: [DISCUSS] Breaking out Marvin from CloudStack
>
> On Sun, Oct 6, 2013 at 8:10 PM, Animesh Chaturvedi <
> animesh.chaturvedi@citrix.com<ma...@citrix.com>> wrote:
>
> > > Yes and we will need to work down a backlog of scenarios before we
> > > ever can rely on guys like me doing that. Not because they won't
> > > but because there is to much to write tests for edging on the new
> > > features they write. Just because those tests aren't there yet. I
> > > think giving Citrix QA a repo to work on is fine but I would like
> > > to see it merged back at some point and a continued possibility to
> > > write
> them in the main tree.
> > >
> > [Animesh>] While I don't agree to a separate repo for tests (marvin
> > framework is ok) I do want to call out the proposal is not for
> > giving Citrix QA a repo to work on and I don't think Prasanna meant that way.
> >
>
>
> I have to apologize for the formulations I choose to express my
> thoughts with. I did not mean to talk of a department of a certain
> company donating human effort for testing to the community. I was
> talking of the frustration of the individuals working and how the
> separate repo would smoothen their workflow. The new repo is in the
> apache domain, no question whether the work in their is done by one person or 100 companies.
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Edison Su <Ed...@citrix.com>.
Few questions:
1. About the "more object-oriented" CloudStack API python binding: Is the proposed api good enough?
For example,
The current hand written create virtual machine looks like:
class VirtualMachine(object):
....
@classmethod
def create(cls, apiclient, services, templateid=None, accountid=None,
domainid=None, zoneid=None, networkids=None, serviceofferingid=None,
securitygroupids=None, projectid=None, startvm=None,
diskofferingid=None, affinitygroupnames=None, group=None,
hostid=None, keypair=None, mode='basic', method='GET'):
the proposed api may look like:
class VirtualMachine(object):
def create(self, apiclient, accountId, templateId, **kwargs)
The proposed api will look better than previous one, and it's automatically generated, so easy to maintain. But as a consumer of the api, how do people know what kind of parameters should be passed in? Will you have an online document for your api? Or you assume people will look at the api docs generated by CloudStack?
Or why not make the api itself as self-contained? For example, add docs before create method:
class VirtualMachine(object):
'''
Args:
accountId: what ever
templateId: whatever
networkids: whatever
'''
'''
Response:
'''
def create(self, apiclient, accountId, templateId, **kwargs)
All the api documents should be included in api discovery already, so it should be easy to add them in your api binding.
2. Regarding to data factories. From the proposed factories, in each test case, does test writer still need to write the code to get data, such as writing code to get account during the setupclass?
I looked at some of the existing test cases, most of them have the same code snippet:
class Services:
def __init__(self):
self.services = {
"account": {
"email": "test@test.com",
"firstname": "Test",
"lastname": "User",
"username": "test",
"password": "password",
},
"virtual_machine": {
"displayname": "Test VM",
"username": "root",
"password": "password",
"ssh_port": 22,
"hypervisor": 'XenServer',
"privateport": 22,
"publicport": 22,
"protocol": 'TCP',
},
With the data factories, the code will look like the following?
Class TestFoo:
Def setupClass():
Account = UserAccount(apiclient)
VM = UserVM(apiClient)
And if I want to customize the default data factories, I should be able to use something like: UserAccount(apiclient, username='myfoo')?
And the data factories should be able to customized based on test environment, right?
For example, the current iso test cases are hardcoded to test against http://people.apache.org/~tsp/dummy.iso, but it won't work for devcloud, or in an internal network. The ISO data factory should be able to return an url based on different test environment, thus iso test cases can be reused.
> -----Original Message-----
> From: Santhosh Edukulla [mailto:santhosh.edukulla@citrix.com]
> Sent: Monday, October 07, 2013 7:06 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [DISCUSS] Breaking out Marvin from CloudStack
>
> Team,
>
> Apart\Away from breaking out marvin from cloudstack, please check the
> other new details provided as part of the new proposal for marvin
> refactoring. Your inputs out of experience are invaluable. Any new feature
> tests for CS will be followed with the new approach, provided if we agree to
> all. Pasting the proposal link one more time below.
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+Refactor
>
> Regards,
> Santhosh
> ________________________________________
> From: Daan Hoogland [daan.hoogland@gmail.com]
> Sent: Sunday, October 06, 2013 3:05 PM
> To: dev
> Subject: Re: [DISCUSS] Breaking out Marvin from CloudStack
>
> On Sun, Oct 6, 2013 at 8:10 PM, Animesh Chaturvedi <
> animesh.chaturvedi@citrix.com> wrote:
>
> > > Yes and we will need to work down a backlog of scenarios before we
> > > ever can rely on guys like me doing that. Not because they won't but
> > > because there is to much to write tests for edging on the new
> > > features they write. Just because those tests aren't there yet. I
> > > think giving Citrix QA a repo to work on is fine but I would like to
> > > see it merged back at some point and a continued possibility to write
> them in the main tree.
> > >
> > [Animesh>] While I don't agree to a separate repo for tests (marvin
> > framework is ok) I do want to call out the proposal is not for giving
> > Citrix QA a repo to work on and I don't think Prasanna meant that way.
> >
>
>
> I have to apologize for the formulations I choose to express my thoughts
> with. I did not mean to talk of a department of a certain company donating
> human effort for testing to the community. I was talking of the frustration of
> the individuals working and how the separate repo would smoothen their
> workflow. The new repo is in the apache domain, no question whether the
> work in their is done by one person or 100 companies.
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Santhosh Edukulla <sa...@citrix.com>.
Team,
Apart\Away from breaking out marvin from cloudstack, please check the other new details provided as part of the new proposal for marvin refactoring. Your inputs out of experience are invaluable. Any new feature tests for CS will be followed with the new approach, provided if we agree to all. Pasting the proposal link one more time below.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+Refactor
Regards,
Santhosh
________________________________________
From: Daan Hoogland [daan.hoogland@gmail.com]
Sent: Sunday, October 06, 2013 3:05 PM
To: dev
Subject: Re: [DISCUSS] Breaking out Marvin from CloudStack
On Sun, Oct 6, 2013 at 8:10 PM, Animesh Chaturvedi <
animesh.chaturvedi@citrix.com> wrote:
> > Yes and we will need to work down a backlog of scenarios before we ever
> > can rely on guys like me doing that. Not because they won't but because
> > there is to much to write tests for edging on the new features they
> > write. Just because those tests aren't there yet. I think giving Citrix
> > QA a repo to work on is fine but I would like to see it merged back at
> > some point and a continued possibility to write them in the main tree.
> >
> [Animesh>] While I don't agree to a separate repo for tests (marvin
> framework is ok) I do want to call out the proposal is not for giving
> Citrix QA a repo to work on and I don't think Prasanna meant that way.
>
I have to apologize for the formulations I choose to express my thoughts
with. I did not mean to talk of a department of a certain company donating
human effort for testing to the community. I was talking of the frustration
of the individuals working and how the separate repo would smoothen their
workflow. The new repo is in the apache domain, no question whether the
work in their is done by one person or 100 companies.
Re: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Daan Hoogland <da...@gmail.com>.
On Sun, Oct 6, 2013 at 8:10 PM, Animesh Chaturvedi <
animesh.chaturvedi@citrix.com> wrote:
> > Yes and we will need to work down a backlog of scenarios before we ever
> > can rely on guys like me doing that. Not because they won't but because
> > there is to much to write tests for edging on the new features they
> > write. Just because those tests aren't there yet. I think giving Citrix
> > QA a repo to work on is fine but I would like to see it merged back at
> > some point and a continued possibility to write them in the main tree.
> >
> [Animesh>] While I don't agree to a separate repo for tests (marvin
> framework is ok) I do want to call out the proposal is not for giving
> Citrix QA a repo to work on and I don't think Prasanna meant that way.
>
I have to apologize for the formulations I choose to express my thoughts
with. I did not mean to talk of a department of a certain company donating
human effort for testing to the community. I was talking of the frustration
of the individuals working and how the separate repo would smoothen their
workflow. The new repo is in the apache domain, no question whether the
work in their is done by one person or 100 companies.
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Animesh Chaturvedi <an...@citrix.com>.
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Saturday, October 05, 2013 3:22 AM
> To: dev
> Subject: Re: [DISCUSS] Breaking out Marvin from CloudStack
>
> H Chip,
>
> As a feature-dev driven by a 150-man-strong-cloud-operator-base that is
> not interested in anything but me showing that they can work with what
> they asked for:
>
> On Fri, Oct 4, 2013 at 10:52 PM, Chip Childers
> <ch...@sungard.com>wrote:
> ...
>
> > On Fri, Oct 04, 2013 at 05:57:58PM +0530, Prasanna Santhanam wrote:
> > > I'll summarize and address the concerns raised so far.
> > >
> > > Marvin has been in this repo for a long time for us to start writing
> > > tests. The only tests I've seen coming are from a specific set of
> > > people focussed on QA efforts.
> >
> > I agree - and that's a problem. New features should *ALL* have tests
> > before they merge into master. I think that assuming that the only
> > test writers are a group of folks that write the tests today is
> > actually a larger problem.
> >
>
> Yes and we will need to work down a backlog of scenarios before we ever
> can rely on guys like me doing that. Not because they won't but because
> there is to much to write tests for edging on the new features they
> write. Just because those tests aren't there yet. I think giving Citrix
> QA a repo to work on is fine but I would like to see it merged back at
> some point and a continued possibility to write them in the main tree.
>
[Animesh>] While I don't agree to a separate repo for tests (marvin framework is ok) I do want to call out the proposal is not for giving Citrix QA a repo to work on and I don't think Prasanna meant that way.
> >
> > > I want to reduce the impediment for
> > > people who are writing tests *today*. Those looking to get started
> > > in the near future won't have any new learning to do, just that
> > > their code goes in an alternate repo that is pointed to right
> > > infrastructure.
> >
> @prasanna: education of the developers is also something to consider. I
> know I could contribute more to integration and at Schuberg Philis we
> have some work going on in that direction. It would be nice to see easy
> integration of this with the Citrix work.
>
>
> >
> ...
>
> > >
> > > Reason for tests and framework to go together is simple.
> >
> Guys, why not both. some general tests that will be valid for any ACS
> version can go in the marvin code base. Others that change over time
> should be tied to the version of ACS they apply to.
>
> Test can be moved from one repo to the other if needed.
>
> ...
>
> >
> > That's marvin, not the tests, right? If I add a new feature for 4.3,
> > should that test appear in a 4.2 run? Aren't we aiming for (I know
> > we're not there) a situation where *all* tests pass before we release?
> > IMO not versioning the tests (not the Marvin framework) with the
> > target of the tests is confusing.
> >
> That I agree with completely.
>
>
> > > The infrastructure (currently) only looks at the changes in the test
> > > directory before performing a run. It doesn't care whether server/
> > > was changed or plugins/x/y/z was changed. That's because the tests
> > > are unrelated to what is in the rest of the repository. In fact you
> > > can't even run them without a deployed cloud. So I don't see why
> > > idle code should lie in the repo.
> >
> You can trigger the building of a cloud when code changes and run the
> tests against that.
>
> > >
> > > Integration tests are essential, they will keep coming as long as
> > > Citrix QA is invested in the effort, but they need to come faster
> > > into the repos and that will be addressed by the separation IMO.
> >
> > How?
> >
> > > Managing
> > > the feature submitted to cloudstack against tests submitted to
> > > marvin is not a hard thing to do. We simply mirror the release
> > > branches in marvin and submit tests there. In fact I wonder why we
> > > didn't have this question when docs were separated? It doesn't work
> > > any differently really.
> >
> > Well, it turns out that docs are absolutely not being done before code
> > comes into master. I dislike this fact, but live with it.
> >
> I don't see why a feature has to be published when it is implemented. So
> I don't agree on the docs. The great advantage of hidden features is
> that you can take them out again more easily when they don't meet the
> promise they made when implemented. So Docs shouldn't be written (or at
> least published) until features are used in real live imho.
>
>
> > > What I would like to see provided by CloudStack is the ability to
> > > upgrade all our test environments, staging environments, UATs, what
> > > have you in continuous integration and have tests run on the
> > > upgraded setup. That allows incrementally testing CloudStack the way
> > > users do it. The current design of installing everything from
> > > scratch, redoing the testbed for each automated test run is mostly a
> > > workaround for that inability. If we had this ability tests written
> > > in marvin can be run against live setups at all times as and when
> > > features merge to master.
> >
> Yes and it would still be great if we could build these installations of
> arbitrary version as required to de upgrades from any version to any
> higher version and run the tests on those. For that marvin (or some tool
> using
> marvin) must be able to preinstall a setup on an version x upgrade to y
> and then run the required test.
>
> > It has always been my goal to reduce the barrier to writing tests for
> > > cloudstack and that's entirely the goal of the refactor as well. So
> > > I hope I have spent my time well. Since I can't be after people to
> > > write tests, and setup mandates, I aim to address the technical
> > > difficulty in writing tests through this refactor and separation.
> >
>
> But you do agree that making all of us write tests, unit as well as
> integration, is the ultimate goal, don't you.
>
>
> > I'll say one thing though... I'm exceptionally frustrated with the
> > features / commits coming into master with little to no tests (unit OR
> > integration). I'm at a loss at this point, and frankly considering a
> > possible reality that this community (generally) just doesn't care
> > about making it easy to reproduce tests. The lack of interest and
> > ignorance about the pain that this causes is infuriating to me.
> >
> Here I can't speak for all of us featurists but I want my features to
> work after I drop them. And I have the good sense to see that unit tests
> are not sufficient for that. I feel guilt as well as infuriation. Taking
> a side here against my wish: It is hard to learn 2 million lines of code
> implement a feature see it work on a version and then try to guarantee
> to the rest of the world that it will not break a twig down under. We
> should. We should also make this as easy as we can. So for now we
> shouldn't ever accept a submission without unit tests and strive to work
> towards a situation where we don't have to accept submissions without
> integration tests any longer.
>
> Let's be practical about it and make sure that both paths (marvin and
> cloudstack repo) are open and stimulate the use of both.
>
> One other thing: I refer to my self as the guilty party somehow. This is
> nonsense but it helps me focus on the goal. One goal is to have people
> outside Citrix write integration tests. I wrote integration tests for
> some features that where a rip-off of some integration tests that Hugo
> wrote for his. We run them once in a while (as in every release but not
> every RC) I am yet to learn the Citrix way of doing this and will focus
> on that. You can not blame a featurist like my self to first focus on
> the code to get something done. Now that o have a grasp of a few percent
> of those two million lines I am happy to learn more about marvin and
> rewrite our tests.
>
> patients with me please, I am willing,
> Daan, (and I am sure loads of others)
Re: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Daan Hoogland <da...@gmail.com>.
H Chip,
As a feature-dev driven by a 150-man-strong-cloud-operator-base that is not
interested in anything but me showing that they can work with what they
asked for:
On Fri, Oct 4, 2013 at 10:52 PM, Chip Childers <ch...@sungard.com>wrote:
...
> On Fri, Oct 04, 2013 at 05:57:58PM +0530, Prasanna Santhanam wrote:
> > I'll summarize and address the concerns raised so far.
> >
> > Marvin has been in this repo for a long time for us to start writing
> > tests. The only tests I've seen coming are from a specific set of
> > people focussed on QA efforts.
>
> I agree - and that's a problem. New features should *ALL* have tests
> before they merge into master. I think that assuming that the only test
> writers are a group of folks that write the tests today is actually a
> larger problem.
>
Yes and we will need to work down a backlog of scenarios before we ever can
rely on guys like me doing that. Not because they won't but because there
is to much to write tests for edging on the new features they write. Just
because those tests aren't there yet. I think giving Citrix QA a repo to
work on is fine but I would like to see it merged back at some point and a
continued possibility to write them in the main tree.
>
> > I want to reduce the impediment for
> > people who are writing tests *today*. Those looking to get started in
> > the near future won't have any new learning to do, just that their
> > code goes in an alternate repo that is pointed to right
> > infrastructure.
>
@prasanna: education of the developers is also something to consider. I
know I could contribute more to integration and at Schuberg Philis we have
some work going on in that direction. It would be nice to see easy
integration of this with the Citrix work.
>
...
> >
> > Reason for tests and framework to go together is simple.
>
Guys, why not both. some general tests that will be valid for any ACS
version can go in the marvin code base. Others that change over time should
be tied to the version of ACS they apply to.
Test can be moved from one repo to the other if needed.
...
>
> That's marvin, not the tests, right? If I add a new feature for 4.3,
> should that test appear in a 4.2 run? Aren't we aiming for (I know
> we're not there) a situation where *all* tests pass before we release?
> IMO not versioning the tests (not the Marvin framework) with the target
> of the tests is confusing.
>
That I agree with completely.
> > The infrastructure (currently) only looks at the changes in the test
> > directory before performing a run. It doesn't care whether server/ was
> > changed or plugins/x/y/z was changed. That's because the tests are
> > unrelated to what is in the rest of the repository. In fact you can't
> > even run them without a deployed cloud. So I don't see why idle code
> > should lie in the repo.
>
You can trigger the building of a cloud when code changes and run the tests
against that.
> >
> > Integration tests are essential, they will keep coming as long as
> > Citrix QA is invested in the effort, but they need to come faster into
> > the repos and that will be addressed by the separation IMO.
>
> How?
>
> > Managing
> > the feature submitted to cloudstack against tests submitted to marvin
> > is not a hard thing to do. We simply mirror the release branches in
> > marvin and submit tests there. In fact I wonder why we didn't have
> > this question when docs were separated? It doesn't work any
> > differently really.
>
> Well, it turns out that docs are absolutely not being done before code
> comes into master. I dislike this fact, but live with it.
>
I don't see why a feature has to be published when it is implemented. So I
don't agree on the docs. The great advantage of hidden features is that you
can take them out again more easily when they don't meet the promise they
made when implemented. So Docs shouldn't be written (or at least published)
until features are used in real live imho.
> > What I would like to see provided by CloudStack is the ability to
> > upgrade all our test environments, staging environments, UATs, what
> > have you in continuous integration and have tests run on the
> > upgraded setup. That allows incrementally testing CloudStack the way
> > users do it. The current design of installing everything from scratch,
> > redoing the testbed for each automated test run is mostly a workaround
> > for that inability. If we had this ability tests written in marvin can
> > be run against live setups at all times as and when features merge to
> > master.
>
Yes and it would still be great if we could build these installations of
arbitrary version as required to de upgrades from any version to any higher
version and run the tests on those. For that marvin (or some tool using
marvin) must be able to preinstall a setup on an version x upgrade to y and
then run the required test.
> It has always been my goal to reduce the barrier to writing tests for
> > cloudstack and that's entirely the goal of the refactor as well. So I
> > hope I have spent my time well. Since I can't be after people to write
> > tests, and setup mandates, I aim to address the technical difficulty
> > in writing tests through this refactor and separation.
>
But you do agree that making all of us write tests, unit as well as
integration, is the ultimate goal, don't you.
> I'll say one thing though... I'm exceptionally frustrated with the
> features / commits coming into master with little to no tests (unit OR
> integration). I'm at a loss at this point, and frankly considering a
> possible reality that this community (generally) just doesn't care about
> making it easy to reproduce tests. The lack of interest and ignorance
> about the pain that this causes is infuriating to me.
>
Here I can't speak for all of us featurists but I want my features to work
after I drop them. And I have the good sense to see that unit tests are not
sufficient for that. I feel guilt as well as infuriation. Taking a side
here against my wish: It is hard to learn 2 million lines of code implement
a feature see it work on a version and then try to guarantee to the rest of
the world that it will not break a twig down under. We should. We should
also make this as easy as we can. So for now we shouldn't ever accept a
submission without unit tests and strive to work towards a situation where
we don't have to accept submissions without integration tests any longer.
Let's be practical about it and make sure that both paths (marvin and
cloudstack repo) are open and stimulate the use of both.
One other thing: I refer to my self as the guilty party somehow. This is
nonsense but it helps me focus on the goal. One goal is to have people
outside Citrix write integration tests. I wrote integration tests for some
features that where a rip-off of some integration tests that Hugo wrote for
his. We run them once in a while (as in every release but not every RC) I
am yet to learn the Citrix way of doing this and will focus on that. You
can not blame a featurist like my self to first focus on the code to get
something done. Now that o have a grasp of a few percent of those two
million lines I am happy to learn more about marvin and rewrite our tests.
patients with me please, I am willing,
Daan, (and I am sure loads of others)
Re: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Chip Childers <ch...@sungard.com>.
Fair warning - some of this is a straw man argument to explore the
situation, and a little bit of ranting at the end.
On Fri, Oct 04, 2013 at 05:57:58PM +0530, Prasanna Santhanam wrote:
> I'll summarize and address the concerns raised so far.
>
> Marvin has been in this repo for a long time for us to start writing
> tests. The only tests I've seen coming are from a specific set of
> people focussed on QA efforts.
I agree - and that's a problem. New features should *ALL* have tests
before they merge into master. I think that assuming that the only test
writers are a group of folks that write the tests today is actually a
larger problem.
> I want to reduce the impediment for
> people who are writing tests *today*. Those looking to get started in
> the near future won't have any new learning to do, just that their
> code goes in an alternate repo that is pointed to right
> infrastructure.
>
> Automated testing also works in a push-to-production style very often.
> Testers need to run their tests on a deployed environment(s) quickly
> to be able to ensure it is valid and passes. By making them go through
> reviewboard each time for each test we massively slow down the
> process. (tons of fixes to tests are on rb today, not just new tests).
> We don't know if they run until they run on the environment.
I want to be clear about this part - a different repo doesn't change the
need for someone to be a committer to commit.
There's a thread on private@ that you should weigh in on here. If
we have more people that should be committers, then let's get *that*
done.
>
> Reason for tests and framework to go together is simple. If I go look
> at the jclouds repository today I find tests for rackspace cloud,
> openstack cloud, cloudstack cloud, euca clouds in the jclouds
> repository and not in the respective provider/project repository. A
> newcomer to the marvin repository will be someone interested in
> writing tests and he will also thus be able to find tests in the
> marvin repository.
>
> This also allows for more heterogenous testing of cloudstack. No one
> needs to be tied down to a framework / tool to write integration
> tests. If python is not your forte, use Chip's ruby client, or perhaps
> in the near future Chiradeep's stackmate to write your test, or even
> jclouds.
But that's actually true today, right? I mean if I wanted to write an
integration test using some other method, I'd do that... but would it
be useful for others? Probably not! That's because the way that we do
testing of this type is via Marvin. The Citrix infra wouldn't be setup
for whatever other framework I used, and the community as a whole would
get less benefit than if I was consistent.
>
> Now the question of supporting older version of marvin against newer
> versions of cloudstack. Marvin now fully auto-generates itself (see
> the design in the proposal) based on endpoint. So you have the
> marvin version that will work with your endpoint only. As for being
> backwards compatible (also addressed in the design doc) - no old tests
> are broken, they will still run perfectly fine.
>
That's marvin, not the tests, right? If I add a new feature for 4.3,
should that test appear in a 4.2 run? Aren't we aiming for (I know
we're not there) a situation where *all* tests pass before we release?
IMO not versioning the tests (not the Marvin framework) with the target
of the tests is confusing.
> The infrastructure (currently) only looks at the changes in the test
> directory before performing a run. It doesn't care whether server/ was
> changed or plugins/x/y/z was changed. That's because the tests are
> unrelated to what is in the rest of the repository. In fact you can't
> even run them without a deployed cloud. So I don't see why idle code
> should lie in the repo.
>
> Integration tests are essential, they will keep coming as long as
> Citrix QA is invested in the effort, but they need to come faster into
> the repos and that will be addressed by the separation IMO.
How?
> Managing
> the feature submitted to cloudstack against tests submitted to marvin
> is not a hard thing to do. We simply mirror the release branches in
> marvin and submit tests there. In fact I wonder why we didn't have
> this question when docs were separated? It doesn't work any
> differently really.
Well, it turns out that docs are absolutely not being done before code
comes into master. I dislike this fact, but live with it.
>
> What I would like to see provided by CloudStack is the ability to
> upgrade all our test environments, staging environments, UATs, what
> have you in continuous integration and have tests run on the
> upgraded setup. That allows incrementally testing CloudStack the way
> users do it. The current design of installing everything from scratch,
> redoing the testbed for each automated test run is mostly a workaround
> for that inability. If we had this ability tests written in marvin can
> be run against live setups at all times as and when features merge to
> master.
This is a valid argument worth exploring. However, it doesn't require a
separate repo, right? I mean, the repo could be cloned twice if
necessary.
>
> It has always been my goal to reduce the barrier to writing tests for
> cloudstack and that's entirely the goal of the refactor as well. So I
> hope I have spent my time well. Since I can't be after people to write
> tests, and setup mandates, I aim to address the technical difficulty
> in writing tests through this refactor and separation.
Some of my arguments above are, frankly, straw man arguments. I see
many of your points as being valid as well. Don't take my perspective
as being against any of your goals, just as discussion points about the
right way to make the goals work.
I'll say one thing though... I'm exceptionally frustrated with the
features / commits coming into master with little to no tests (unit OR
integration). I'm at a loss at this point, and frankly considering a
possible reality that this community (generally) just doesn't care about
making it easy to reproduce tests. The lack of interest and ignorance
about the pain that this causes is infuriating to me.
Prasanna, I can tell that you have some / similar frustrations (but I
won't put the words into your mouth), and are trying to find ways to
make it easier for others to step in and help out where feature developers
are apparently showing no interest. This is an admirable goal.
So... perhaps we have to live with the complete lack of interest from
the feature devs, and take the approach of breaking out the code. That
doesn't make *access permission* easier, but it does make a target repo
easier to focus on test scenarios.
As I said above, let's talk on private@ about people that are doing the
tests that deserve the commit bit. At least we can help them help
everyone else...
>
> PS: David, by hosted docs I mean something like
> http://pythonhosted.org/cloudmonkey. Lives in the repo but read from
> the webpage.
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
>
Re: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Daan Hoogland <da...@gmail.com>.
Prasanna,
I share Chips concern. But let me start by saying that given the effort
Citrix puts in the integration testing, making it easy for your guys should
be somewhat a priority. However, can we do the separation is such a way
that integration tests as part of plugin and other enhancements is still
trivial? for instance make the maven scripts checkout a stable marvin
version and run tests using this framework. What Chip wants, i think, is
that we enforce a integration test py in every contribution not only to
validate the feature but also to ensure its continued functioning while the
contributors may have wondered off in their $dayjob.
I followed the reviews submitted and know that the ones in the marvin
corner of the world are a big chunk of it ans have few reviewers/committers
in comparison. Can your refactor be organized so that integration tests can
go in the marvin and in the cloudstack repo?
thanks,
Daan
On Fri, Oct 4, 2013 at 2:27 PM, Prasanna Santhanam <ts...@apache.org> wrote:
> I'll summarize and address the concerns raised so far.
>
> Marvin has been in this repo for a long time for us to start writing
> tests. The only tests I've seen coming are from a specific set of
> people focussed on QA efforts. I want to reduce the impediment for
> people who are writing tests *today*. Those looking to get started in
> the near future won't have any new learning to do, just that their
> code goes in an alternate repo that is pointed to right
> infrastructure.
>
> Automated testing also works in a push-to-production style very often.
> Testers need to run their tests on a deployed environment(s) quickly
> to be able to ensure it is valid and passes. By making them go through
> reviewboard each time for each test we massively slow down the
> process. (tons of fixes to tests are on rb today, not just new tests).
> We don't know if they run until they run on the environment.
>
> Reason for tests and framework to go together is simple. If I go look
> at the jclouds repository today I find tests for rackspace cloud,
> openstack cloud, cloudstack cloud, euca clouds in the jclouds
> repository and not in the respective provider/project repository. A
> newcomer to the marvin repository will be someone interested in
> writing tests and he will also thus be able to find tests in the
> marvin repository.
>
> This also allows for more heterogenous testing of cloudstack. No one
> needs to be tied down to a framework / tool to write integration
> tests. If python is not your forte, use Chip's ruby client, or perhaps
> in the near future Chiradeep's stackmate to write your test, or even
> jclouds.
>
> Now the question of supporting older version of marvin against newer
> versions of cloudstack. Marvin now fully auto-generates itself (see
> the design in the proposal) based on endpoint. So you have the
> marvin version that will work with your endpoint only. As for being
> backwards compatible (also addressed in the design doc) - no old tests
> are broken, they will still run perfectly fine.
>
> The infrastructure (currently) only looks at the changes in the test
> directory before performing a run. It doesn't care whether server/ was
> changed or plugins/x/y/z was changed. That's because the tests are
> unrelated to what is in the rest of the repository. In fact you can't
> even run them without a deployed cloud. So I don't see why idle code
> should lie in the repo.
>
> Integration tests are essential, they will keep coming as long as
> Citrix QA is invested in the effort, but they need to come faster into
> the repos and that will be addressed by the separation IMO. Managing
> the feature submitted to cloudstack against tests submitted to marvin
> is not a hard thing to do. We simply mirror the release branches in
> marvin and submit tests there. In fact I wonder why we didn't have
> this question when docs were separated? It doesn't work any
> differently really.
>
> What I would like to see provided by CloudStack is the ability to
> upgrade all our test environments, staging environments, UATs, what
> have you in continuous integration and have tests run on the
> upgraded setup. That allows incrementally testing CloudStack the way
> users do it. The current design of installing everything from scratch,
> redoing the testbed for each automated test run is mostly a workaround
> for that inability. If we had this ability tests written in marvin can
> be run against live setups at all times as and when features merge to
> master.
>
> It has always been my goal to reduce the barrier to writing tests for
> cloudstack and that's entirely the goal of the refactor as well. So I
> hope I have spent my time well. Since I can't be after people to write
> tests, and setup mandates, I aim to address the technical difficulty
> in writing tests through this refactor and separation.
>
> PS: David, by hosted docs I mean something like
> http://pythonhosted.org/cloudmonkey. Lives in the repo but read from
> the webpage.
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
>
Re: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Prasanna Santhanam <ts...@apache.org>.
I'll summarize and address the concerns raised so far.
Marvin has been in this repo for a long time for us to start writing
tests. The only tests I've seen coming are from a specific set of
people focussed on QA efforts. I want to reduce the impediment for
people who are writing tests *today*. Those looking to get started in
the near future won't have any new learning to do, just that their
code goes in an alternate repo that is pointed to right
infrastructure.
Automated testing also works in a push-to-production style very often.
Testers need to run their tests on a deployed environment(s) quickly
to be able to ensure it is valid and passes. By making them go through
reviewboard each time for each test we massively slow down the
process. (tons of fixes to tests are on rb today, not just new tests).
We don't know if they run until they run on the environment.
Reason for tests and framework to go together is simple. If I go look
at the jclouds repository today I find tests for rackspace cloud,
openstack cloud, cloudstack cloud, euca clouds in the jclouds
repository and not in the respective provider/project repository. A
newcomer to the marvin repository will be someone interested in
writing tests and he will also thus be able to find tests in the
marvin repository.
This also allows for more heterogenous testing of cloudstack. No one
needs to be tied down to a framework / tool to write integration
tests. If python is not your forte, use Chip's ruby client, or perhaps
in the near future Chiradeep's stackmate to write your test, or even
jclouds.
Now the question of supporting older version of marvin against newer
versions of cloudstack. Marvin now fully auto-generates itself (see
the design in the proposal) based on endpoint. So you have the
marvin version that will work with your endpoint only. As for being
backwards compatible (also addressed in the design doc) - no old tests
are broken, they will still run perfectly fine.
The infrastructure (currently) only looks at the changes in the test
directory before performing a run. It doesn't care whether server/ was
changed or plugins/x/y/z was changed. That's because the tests are
unrelated to what is in the rest of the repository. In fact you can't
even run them without a deployed cloud. So I don't see why idle code
should lie in the repo.
Integration tests are essential, they will keep coming as long as
Citrix QA is invested in the effort, but they need to come faster into
the repos and that will be addressed by the separation IMO. Managing
the feature submitted to cloudstack against tests submitted to marvin
is not a hard thing to do. We simply mirror the release branches in
marvin and submit tests there. In fact I wonder why we didn't have
this question when docs were separated? It doesn't work any
differently really.
What I would like to see provided by CloudStack is the ability to
upgrade all our test environments, staging environments, UATs, what
have you in continuous integration and have tests run on the
upgraded setup. That allows incrementally testing CloudStack the way
users do it. The current design of installing everything from scratch,
redoing the testbed for each automated test run is mostly a workaround
for that inability. If we had this ability tests written in marvin can
be run against live setups at all times as and when features merge to
master.
It has always been my goal to reduce the barrier to writing tests for
cloudstack and that's entirely the goal of the refactor as well. So I
hope I have spent my time well. Since I can't be after people to write
tests, and setup mandates, I aim to address the technical difficulty
in writing tests through this refactor and separation.
PS: David, by hosted docs I mean something like
http://pythonhosted.org/cloudmonkey. Lives in the repo but read from
the webpage.
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Chip Childers <ch...@sungard.com>.
On Thu, Oct 03, 2013 at 05:27:27PM +0000, Santhosh Edukulla wrote:
> >The problem with tests being broken out from the main repo is that we
> > end up having a really hard time confirming that new features include
> > appropriate tests. Plus, tests are tied to features, which are tied to
> > the code. They should be versioned together.
>
> I believe this makes more sense for unit tests though, where tests need to be there with the code and both go hand in hand with release.
>
> As mentioned earlier, keeping or separating automation code\tests with the main product will not make much of an impact as a whole.
Right, it makes absolutely 100% sense for unit tests (in fact, we would
be idiots to break them out).
My point still stands. Integration tests should be part of building a
feature. No tests, no commit to master. I don't want to have to cross
reference between two different repos, when a simple feature branch diff
(with an up-to-date feature branch) will make it clear that a merge
includes good changes *and* integration tests.
-chip
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Santhosh Edukulla <sa...@citrix.com>.
>The problem with tests being broken out from the main repo is that we
end up having a really hard time confirming that new features include
appropriate tests. Plus, tests are tied to features, which are tied to
the code. They should be versioned together.
I believe this makes more sense for unit tests though, where tests need to be there with the code and both go hand in hand with release.
As mentioned earlier, keeping or separating automation code\tests with the main product will not make much of an impact as a whole.
Regards,
Santhosh
________________________________________
From: Chip Childers [chip.childers@sungard.com]
Sent: Thursday, October 03, 2013 11:07 AM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Breaking out Marvin from CloudStack
On Thu, Oct 03, 2013 at 05:55:14AM +0000, Santhosh Edukulla wrote:
> I believe both can exist independently. The framework should have more flexibility and facilitation to run features\test cases based upon a given release and version on a given test bed. It should be intelligent enough to run those cases and only those cases pertaining to that release with few config variables. Keeping aside branch\repo terminology, for me all marvin both framework and tests should be at one place and separate from main product code.
The problem with tests being broken out from the main repo is that we
end up having a really hard time confirming that new features include
appropriate tests. Plus, tests are tied to features, which are tied to
the code. They should be versioned together.
As you said though, the framework is the framework... that's a tool.
Frankly, with auto-discovery going forward, Marvin becomes no different
from any other testing framework software that has to be installed to do
useful things with the code in the main repo. Ex: maven!
-chip
Re: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Chip Childers <ch...@sungard.com>.
On Thu, Oct 03, 2013 at 05:55:14AM +0000, Santhosh Edukulla wrote:
> I believe both can exist independently. The framework should have more flexibility and facilitation to run features\test cases based upon a given release and version on a given test bed. It should be intelligent enough to run those cases and only those cases pertaining to that release with few config variables. Keeping aside branch\repo terminology, for me all marvin both framework and tests should be at one place and separate from main product code.
The problem with tests being broken out from the main repo is that we
end up having a really hard time confirming that new features include
appropriate tests. Plus, tests are tied to features, which are tied to
the code. They should be versioned together.
As you said though, the framework is the framework... that's a tool.
Frankly, with auto-discovery going forward, Marvin becomes no different
from any other testing framework software that has to be installed to do
useful things with the code in the main repo. Ex: maven!
-chip
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Santhosh Edukulla <sa...@citrix.com>.
I believe both can exist independently. The framework should have more flexibility and facilitation to run features\test cases based upon a given release and version on a given test bed. It should be intelligent enough to run those cases and only those cases pertaining to that release with few config variables. Keeping aside branch\repo terminology, for me all marvin both framework and tests should be at one place and separate from main product code.
1. Even keeping tests as part of cs code, require the user to checkout the marvin because of its dependencies. If a test\feature is using any marvin library call still needs to have marvin and in that way he\she checksout marvin anyway.
2. As a user, this is what i will do. On a given testbed,
a) I will checkout the cloudstack code
b) I will checkout the marvin code
c) For running automation, I will add few configuration parameters on that test bed.
d) Then run the automation,
EX: sh run_marvin.sh "4.2" "bvt". This will run bvt using tests pertaining to 4.2 release on that test bed.
EX: sh run_marvin.sh "4.1" "reg" "f1,f2,f3". This will run full regression suite only with features f1,f2,f3
EX: sh run_marvin.sh "4.x" "reg" "all". This will run full regression suite for me. These are specific and generic examples, the control of running test code can be made more specific through config file\lib of marvin or through cmd line arguments etc. That we can see to change later.
3. In a way marvin as a framework\library and tests are to some extent dependent. Marvin is meant currently to test cs API's, so libraries written are to enhance writing tests easy and in a way are more specific to cs product as a whole. New features keeps getting added or features getting changed , all code keeps changing, people writing new featuresets can add their feature to the marvin branch containing tests as well and enhance any library call when they see requires change.
4. As an example reference to this, we used to have multiple product branches with varied patch and maintenance releases, with automation code maintained separately having tests as well. We used to do as what we mentioned in step2. It worked for us.
5. Only, thing is we need to make this transition and enhancing smoother so that it will not effect current team activities of running automation.
Again, this is just a thought, Even keeping both cs and marvin as well will not make much difference though i believe. Both, tests and library on which these tests are dependent should be under one single folder i believe.
Regards,
Santhosh
________________________________________
From: Alex Huang [Alex.Huang@citrix.com]
Sent: Wednesday, October 02, 2013 4:42 PM
To: dev@cloudstack.apache.org
Subject: RE: [DISCUSS] Breaking out Marvin from CloudStack
Yeah...I'm more amendable to this proposal. I just don't see tests being separated from the source release. In fact, I see a lot of problems with matching versions and releases.
I still question the value of separating out a framework where both ends (tests and the server it tests) stay in the same repo but I guess there should be no harm. I do think our time can be better spent elsewhere (for example, writing the tests) but, if others see it as necessary, I wouldn't be against it.
@Edison
I can do what you said now. To me that's more or less maven changes and build changes. Not a repo/separate release question.
--Alex
> IMO, we should consider Marvin the "framework" to be the thing to break
> out, and the tests should be different from the framework.
>
> Now that leads to the question: to test or not to test (in the main repo)?
>
> I'd suggest that *tests* belong in the main repo, because they are tied to the
> software's capabilities and versions.
>
> The Marvin framework, on the other hand, since the re-work that Prasanna
> did, is mostly distinct (and uses API discovery).
>
> Anyone else agree?
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Alex Huang <Al...@citrix.com>.
Yeah...I'm more amendable to this proposal. I just don't see tests being separated from the source release. In fact, I see a lot of problems with matching versions and releases.
I still question the value of separating out a framework where both ends (tests and the server it tests) stay in the same repo but I guess there should be no harm. I do think our time can be better spent elsewhere (for example, writing the tests) but, if others see it as necessary, I wouldn't be against it.
@Edison
I can do what you said now. To me that's more or less maven changes and build changes. Not a repo/separate release question.
--Alex
> IMO, we should consider Marvin the "framework" to be the thing to break
> out, and the tests should be different from the framework.
>
> Now that leads to the question: to test or not to test (in the main repo)?
>
> I'd suggest that *tests* belong in the main repo, because they are tied to the
> software's capabilities and versions.
>
> The Marvin framework, on the other hand, since the re-work that Prasanna
> did, is mostly distinct (and uses API discovery).
>
> Anyone else agree?
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Sudha Ponnaganti <su...@citrix.com>.
Agree to keep tests with main repo as suggested as tests are tied to releases.
Backward compatibility of framework need to be considered when breaking it apart as we still use the framework on older branches for validation of patches.
-----Original Message-----
From: Animesh Chaturvedi [mailto:animesh.chaturvedi@citrix.com]
Sent: Wednesday, October 02, 2013 2:08 PM
To: dev@cloudstack.apache.org
Subject: RE: [DISCUSS] Breaking out Marvin from CloudStack
> -----Original Message-----
> From: Chip Childers [mailto:chipchilders@apache.org]
> Sent: Wednesday, October 02, 2013 1:10 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [DISCUSS] Breaking out Marvin from CloudStack
>
> On Wed, Oct 02, 2013 at 07:38:33PM +0000, Alex Huang wrote:
> > I don't really understand what purpose would this serve. Would we
> ever use newer marvin against older CloudStack or vice versa? What's
> the benefit?
> >
> > I can understand it for cloudmonkey because cloudmonkey is an admin
> cli tool and reving it differently is not a bad idea. I just don't
> see it for marvin and, especially for the tests.
> >
> > --Alex
>
> IMO, we should consider Marvin the "framework" to be the thing to
> break out, and the tests should be different from the framework.
>
> Now that leads to the question: to test or not to test (in the main
> repo)?
>
> I'd suggest that *tests* belong in the main repo, because they are
> tied to the software's capabilities and versions.
>
> The Marvin framework, on the other hand, since the re-work that
> Prasanna did, is mostly distinct (and uses API discovery).
>
> Anyone else agree?
[Animesh>] I agree tests belong in the main repo. Breaking up the marvin framework gives flexibility in its usage beyond cloudstack and separate release cycle.
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Animesh Chaturvedi <an...@citrix.com>.
> -----Original Message-----
> From: Chip Childers [mailto:chipchilders@apache.org]
> Sent: Wednesday, October 02, 2013 1:10 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [DISCUSS] Breaking out Marvin from CloudStack
>
> On Wed, Oct 02, 2013 at 07:38:33PM +0000, Alex Huang wrote:
> > I don't really understand what purpose would this serve. Would we
> ever use newer marvin against older CloudStack or vice versa? What's
> the benefit?
> >
> > I can understand it for cloudmonkey because cloudmonkey is an admin
> cli tool and reving it differently is not a bad idea. I just don't see
> it for marvin and, especially for the tests.
> >
> > --Alex
>
> IMO, we should consider Marvin the "framework" to be the thing to break
> out, and the tests should be different from the framework.
>
> Now that leads to the question: to test or not to test (in the main
> repo)?
>
> I'd suggest that *tests* belong in the main repo, because they are tied
> to the software's capabilities and versions.
>
> The Marvin framework, on the other hand, since the re-work that Prasanna
> did, is mostly distinct (and uses API discovery).
>
> Anyone else agree?
[Animesh>] I agree tests belong in the main repo. Breaking up the marvin framework gives flexibility in its usage beyond cloudstack and separate release cycle.
Re: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Chiradeep Vittal <Ch...@citrix.com>.
Chip, that makes perfect sense to me.
On 10/2/13 1:10 PM, "Chip Childers" <ch...@apache.org> wrote:
>On Wed, Oct 02, 2013 at 07:38:33PM +0000, Alex Huang wrote:
>> I don't really understand what purpose would this serve. Would we ever
>>use newer marvin against older CloudStack or vice versa? What's the
>>benefit?
>>
>> I can understand it for cloudmonkey because cloudmonkey is an admin cli
>>tool and reving it differently is not a bad idea. I just don't see it
>>for marvin and, especially for the tests.
>>
>> --Alex
>
>IMO, we should consider Marvin the "framework" to be the thing to break
>out, and the tests should be different from the framework.
>
>Now that leads to the question: to test or not to test (in the main
>repo)?
>
>I'd suggest that *tests* belong in the main repo, because they are tied
>to the software's capabilities and versions.
>
>The Marvin framework, on the other hand, since the re-work that Prasanna
>did, is mostly distinct (and uses API discovery).
>
>Anyone else agree?
Re: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Chip Childers <ch...@apache.org>.
On Wed, Oct 02, 2013 at 07:38:33PM +0000, Alex Huang wrote:
> I don't really understand what purpose would this serve. Would we ever use newer marvin against older CloudStack or vice versa? What's the benefit?
>
> I can understand it for cloudmonkey because cloudmonkey is an admin cli tool and reving it differently is not a bad idea. I just don't see it for marvin and, especially for the tests.
>
> --Alex
IMO, we should consider Marvin the "framework" to be the thing to break
out, and the tests should be different from the framework.
Now that leads to the question: to test or not to test (in the main
repo)?
I'd suggest that *tests* belong in the main repo, because they are tied
to the software's capabilities and versions.
The Marvin framework, on the other hand, since the re-work that Prasanna
did, is mostly distinct (and uses API discovery).
Anyone else agree?
Re: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Sebastien Goasguen <ru...@gmail.com>.
On Oct 2, 2013, at 4:05 PM, Edison Su <Ed...@citrix.com> wrote:
> any user can just simply "pip install marvin", without download/build cloudstack source code.
>
yes, but you have the same issue as we had with cloudmonkey. When can we release a new marvin ?
breaking it up allows to have a different release cycle.
>> -----Original Message-----
>> From: Alex Huang [mailto:Alex.Huang@citrix.com]
>> Sent: Wednesday, October 02, 2013 12:39 PM
>> To: dev@cloudstack.apache.org
>> Subject: RE: [DISCUSS] Breaking out Marvin from CloudStack
>>
>> I don't really understand what purpose would this serve. Would we ever use
>> newer marvin against older CloudStack or vice versa? What's the benefit?
>>
>> I can understand it for cloudmonkey because cloudmonkey is an admin cli
>> tool and reving it differently is not a bad idea. I just don't see it for marvin
>> and, especially for the tests.
>>
>> --Alex
>>
>>> -----Original Message-----
>>> From: Prasanna Santhanam [mailto:tsp@apache.org]
>>> Sent: Wednesday, October 2, 2013 10:14 AM
>>> To: CloudStack Dev
>>> Subject: [DISCUSS] Breaking out Marvin from CloudStack
>>>
>>> I would like to seperate marvin from the main cloudstack repo. Much of
>>> marvin's development has little coupling with CloudStack.
>>>
>>> Similar to CloudMonkey, marvin undergoes rapid changes and it is
>>> essential to provide a smooth workflow and faster releases for those
>> working with it.
>>>
>>> There are also a small set of people currently looking at marvin for
>>> testing right now. Often, their reviews and QA effort is mixed with
>>> those of cloudstack itself. By having a different repo I'd like to be
>>> able to provide commit access to those working on marvin alone quickly to
>> help with testing.
>>>
>>> After separating marvin
>>> 0. we will have a separate release cycle for marvin 1. we will have a
>>> new home for marvin's docs using Sphinx 2. if possible, a different
>>> criteria for providing commit access to marvin's repos.
>>> 3. all tests of cloudstack will also move to marvin's repository
>>>
>>> Thoughts?
>>>
>>> --
>>> Prasanna.,
>>>
>>> ------------------------
>>> Powered by BigRock.com
>
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Edison Su <Ed...@citrix.com>.
any user can just simply "pip install marvin", without download/build cloudstack source code.
> -----Original Message-----
> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> Sent: Wednesday, October 02, 2013 12:39 PM
> To: dev@cloudstack.apache.org
> Subject: RE: [DISCUSS] Breaking out Marvin from CloudStack
>
> I don't really understand what purpose would this serve. Would we ever use
> newer marvin against older CloudStack or vice versa? What's the benefit?
>
> I can understand it for cloudmonkey because cloudmonkey is an admin cli
> tool and reving it differently is not a bad idea. I just don't see it for marvin
> and, especially for the tests.
>
> --Alex
>
> > -----Original Message-----
> > From: Prasanna Santhanam [mailto:tsp@apache.org]
> > Sent: Wednesday, October 2, 2013 10:14 AM
> > To: CloudStack Dev
> > Subject: [DISCUSS] Breaking out Marvin from CloudStack
> >
> > I would like to seperate marvin from the main cloudstack repo. Much of
> > marvin's development has little coupling with CloudStack.
> >
> > Similar to CloudMonkey, marvin undergoes rapid changes and it is
> > essential to provide a smooth workflow and faster releases for those
> working with it.
> >
> > There are also a small set of people currently looking at marvin for
> > testing right now. Often, their reviews and QA effort is mixed with
> > those of cloudstack itself. By having a different repo I'd like to be
> > able to provide commit access to those working on marvin alone quickly to
> help with testing.
> >
> > After separating marvin
> > 0. we will have a separate release cycle for marvin 1. we will have a
> > new home for marvin's docs using Sphinx 2. if possible, a different
> > criteria for providing commit access to marvin's repos.
> > 3. all tests of cloudstack will also move to marvin's repository
> >
> > Thoughts?
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
RE: [DISCUSS] Breaking out Marvin from CloudStack
Posted by Alex Huang <Al...@citrix.com>.
I don't really understand what purpose would this serve. Would we ever use newer marvin against older CloudStack or vice versa? What's the benefit?
I can understand it for cloudmonkey because cloudmonkey is an admin cli tool and reving it differently is not a bad idea. I just don't see it for marvin and, especially for the tests.
--Alex
> -----Original Message-----
> From: Prasanna Santhanam [mailto:tsp@apache.org]
> Sent: Wednesday, October 2, 2013 10:14 AM
> To: CloudStack Dev
> Subject: [DISCUSS] Breaking out Marvin from CloudStack
>
> I would like to seperate marvin from the main cloudstack repo. Much of
> marvin's development has little coupling with CloudStack.
>
> Similar to CloudMonkey, marvin undergoes rapid changes and it is essential
> to provide a smooth workflow and faster releases for those working with it.
>
> There are also a small set of people currently looking at marvin for testing
> right now. Often, their reviews and QA effort is mixed with those of
> cloudstack itself. By having a different repo I'd like to be able to provide
> commit access to those working on marvin alone quickly to help with testing.
>
> After separating marvin
> 0. we will have a separate release cycle for marvin 1. we will have a new
> home for marvin's docs using Sphinx 2. if possible, a different criteria for
> providing commit access to marvin's repos.
> 3. all tests of cloudstack will also move to marvin's repository
>
> Thoughts?
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com