You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Prasanna Santhanam <ts...@apache.org> on 2013/06/15 07:42:37 UTC

Infra Issues from the IRC meeting (Wed, Jun 12)

Saw a few infra topics being discussed in the meeting and there was
talk of bringing it to the list so I'm taking this opportunity to
explain some things behind it.

> 17:22:44 [Animesh]: I will send out my weekly reminder today on status and include Sudha's test results 
> 17:23:28 [topcloud]: one thing that concerns me is that the bvt continues to be at < 100% pass rate
> 17:23:41 [topcloud]: is there anything we're doing about this?
yes - I've fixed most tests. Some have existed because of bugs in
packaging, systemvm templates that I see patches for now. 

> 17:25:20 [chipc]: topcloud: was BVT ever at 100% ?
> 17:25:32 [chipc]: (real question, not sarcasm)
It was - 100% - when the project was first proposed. But more tests
have come in since then.

> 17:26:41 [chipc]: once we get it back to 100%, I say we block all changes when it drops to <100%
> 17:26:49 [topcloud]: +1
+1 - this is what I've been driving towards but haven't announced some
changes I've made in the past weeks because it's pointless to have
tests fail soon as I announce we are 100%. We shouldn't wait for one
run, but at least 10 to ensure that the bvt is indeed stable enough to
be trusted as a 'gating' system for master stability.

There's also a couple of issues here -
1. Does everyone know where the tests run?
2. Do people know how to spot the failures?
3. Do people know how to find the logs for the failures?

If the answer is no to all this, I have more documentation on my
hands.

> 17:28:07 [Animesh]: agreed bvt also shows progress towards release readines
> 17:28:07 [chipc]: topcloud: +1
> 17:28:32 [chipc]: Animesh: BVT should show that master is stable, regardless of release timeframes
> 17:28:33 [chipc]: IMO that is
> 17:28:44 [chipc]: master should only see good  /tested code
Which is why the BVT runs on master at all times on
jenkins.buildacloud.org. There is also ability to run it against a
feature branch but I would rather defer that to the release manager
for now since it's tied with hardware resources and jenkins schedules.
That feature should strictly be reserved for architecture changes in
MERGE requests.

> 17:43:27 [topcloud]: sorry...to bring back this topic but is bvt running on apache infra?
> 17:43:35 [chipc]: no
> 17:43:57 [topcloud]: chipc: is there any talk about bringing it into apache infra?
It was brought up with the ASF infra back in January and the
suggestion was to donate hardware to the ASF to manage. So if we're
prepared to do that, great! But it certainly can't just be Citrix :)

I'd prefer individual project related test hardware and resources
to stay in the control of the project. Infrastructure is constantly
changed to allow features and enhancements to be tested so it's best
to have it in the core group. Which is why jenkins.buildacloud.org
came to existence. This is similar to how cloudbees operates or
(*gasp*) openstack-infra [1][2] operates.

Ideally, I'd like those interested in infra activities to form a
separate group for cloudstack-infra related topics. The group's focus
will be to maintain, test and add to the infrastructure of this
project. But that's in the future. Without such a group, building an
IaaS cloud is not much fun :)

> 17:44:17 [topcloud]: i can't imagine apache wanting bvt to only run inside citrix all the time.
It doesn't run within Citrix. It runs in a DC in Fremont. There are
other environments within Citrix however that run their own tests for
their needs - eg: object_store tests, cloudplatform tests, customer
related issue tests etc.

/me beginning to think more doc work is on my way :/

> 17:46:27 [chipc]: but generally, the ASF build infra is a bit overloaded
+1000

> 17:46:51 [jzb]: topcloud: when you say "in Citrix" - it's still visible outside Citrix, yes?
> 17:46:52 [chipc]: so frankly, CTXS donating an environment to run it, publicly visible to everyone, is quite helpful
> 17:46:58 [chipc]: jzb: it is
We need more people to donate hardware/virtual resources for testing :) 
CTXS has been gracious to provide quite a few resources already IMO.

> 17:47:18 [chipc]: actually, I think it is...  
> 17:47:34 [topcloud]: jzb: yeah it's still visible but it really should be runnable by everyone.
Not quite. It's a gating system. It runs automatically and shouldn't
be runnable by everyone at will. I'm still waiting to implement
devcloud tests based on that gerrit converstaion (which went nowhere)
we had many months back. DevCloud stuff can be run at will.

> 17:47:37 [jzb]: I'm all for building up Apache infra, but I also
> think having vendors donate publicly visible resources that are
> usable by the community is acceptable.
+1

> 17:47:53 [jzb]: in fact, we probably ought to be hitting up some of
> our ISP friends for more. 
+1 - who are our ISP friends? Would like to get help on this.

> 17:49:49 [ke4qqq]: so tsp (along with abayer and roman) are working
> on a publicly accessible jenkins instance in fremont
This is basically to dogfood all instances via a hosted puppetmaster.
I've just started so will take time to finish this.

Thoughts are most welcome!

[1] https://jenkins.openstack.org/
[2] http://ci.openstack.org/
-- 
Prasanna.,

------------------------
Powered by BigRock.com


RE: Infra Issues from the IRC meeting (Wed, Jun 12)

Posted by Sudha Ponnaganti <su...@citrix.com>.
Daily automated email with BVT test results would be helpful. 

-----Original Message-----
From: Prasanna Santhanam [mailto:tsp@apache.org] 
Sent: Friday, June 14, 2013 10:43 PM
To: dev@cloudstack.apache.org
Subject: Infra Issues from the IRC meeting (Wed, Jun 12)

Saw a few infra topics being discussed in the meeting and there was talk of bringing it to the list so I'm taking this opportunity to explain some things behind it.

> 17:22:44 [Animesh]: I will send out my weekly reminder today on status 
> and include Sudha's test results
> 17:23:28 [topcloud]: one thing that concerns me is that the bvt 
> continues to be at < 100% pass rate
> 17:23:41 [topcloud]: is there anything we're doing about this?
yes - I've fixed most tests. Some have existed because of bugs in packaging, systemvm templates that I see patches for now. 

> 17:25:20 [chipc]: topcloud: was BVT ever at 100% ?
> 17:25:32 [chipc]: (real question, not sarcasm)
It was - 100% - when the project was first proposed. But more tests have come in since then.

> 17:26:41 [chipc]: once we get it back to 100%, I say we block all 
> changes when it drops to <100%
> 17:26:49 [topcloud]: +1
+1 - this is what I've been driving towards but haven't announced some
changes I've made in the past weeks because it's pointless to have tests fail soon as I announce we are 100%. We shouldn't wait for one run, but at least 10 to ensure that the bvt is indeed stable enough to be trusted as a 'gating' system for master stability.

There's also a couple of issues here -
1. Does everyone know where the tests run?
2. Do people know how to spot the failures?
3. Do people know how to find the logs for the failures?

If the answer is no to all this, I have more documentation on my hands.

> 17:28:07 [Animesh]: agreed bvt also shows progress towards release 
> readines
> 17:28:07 [chipc]: topcloud: +1
> 17:28:32 [chipc]: Animesh: BVT should show that master is stable, 
> regardless of release timeframes
> 17:28:33 [chipc]: IMO that is
> 17:28:44 [chipc]: master should only see good  /tested code
Which is why the BVT runs on master at all times on jenkins.buildacloud.org. There is also ability to run it against a feature branch but I would rather defer that to the release manager for now since it's tied with hardware resources and jenkins schedules.
That feature should strictly be reserved for architecture changes in MERGE requests.

> 17:43:27 [topcloud]: sorry...to bring back this topic but is bvt running on apache infra?
> 17:43:35 [chipc]: no
> 17:43:57 [topcloud]: chipc: is there any talk about bringing it into apache infra?
It was brought up with the ASF infra back in January and the suggestion was to donate hardware to the ASF to manage. So if we're prepared to do that, great! But it certainly can't just be Citrix :)

I'd prefer individual project related test hardware and resources to stay in the control of the project. Infrastructure is constantly changed to allow features and enhancements to be tested so it's best to have it in the core group. Which is why jenkins.buildacloud.org came to existence. This is similar to how cloudbees operates or
(*gasp*) openstack-infra [1][2] operates.

Ideally, I'd like those interested in infra activities to form a separate group for cloudstack-infra related topics. The group's focus will be to maintain, test and add to the infrastructure of this project. But that's in the future. Without such a group, building an IaaS cloud is not much fun :)

> 17:44:17 [topcloud]: i can't imagine apache wanting bvt to only run inside citrix all the time.
It doesn't run within Citrix. It runs in a DC in Fremont. There are other environments within Citrix however that run their own tests for their needs - eg: object_store tests, cloudplatform tests, customer related issue tests etc.

/me beginning to think more doc work is on my way :/

> 17:46:27 [chipc]: but generally, the ASF build infra is a bit 
> overloaded
+1000

> 17:46:51 [jzb]: topcloud: when you say "in Citrix" - it's still visible outside Citrix, yes?
> 17:46:52 [chipc]: so frankly, CTXS donating an environment to run it, 
> publicly visible to everyone, is quite helpful
> 17:46:58 [chipc]: jzb: it is
We need more people to donate hardware/virtual resources for testing :) CTXS has been gracious to provide quite a few resources already IMO.

> 17:47:18 [chipc]: actually, I think it is...  
> 17:47:34 [topcloud]: jzb: yeah it's still visible but it really should be runnable by everyone.
Not quite. It's a gating system. It runs automatically and shouldn't be runnable by everyone at will. I'm still waiting to implement devcloud tests based on that gerrit converstaion (which went nowhere) we had many months back. DevCloud stuff can be run at will.

> 17:47:37 [jzb]: I'm all for building up Apache infra, but I also think 
> having vendors donate publicly visible resources that are usable by 
> the community is acceptable.
+1

> 17:47:53 [jzb]: in fact, we probably ought to be hitting up some of 
> our ISP friends for more.
+1 - who are our ISP friends? Would like to get help on this.

> 17:49:49 [ke4qqq]: so tsp (along with abayer and roman) are working on 
> a publicly accessible jenkins instance in fremont
This is basically to dogfood all instances via a hosted puppetmaster.
I've just started so will take time to finish this.

Thoughts are most welcome!

[1] https://jenkins.openstack.org/
[2] http://ci.openstack.org/
--
Prasanna.,

------------------------
Powered by BigRock.com


Re: Infra Issues from the IRC meeting (Wed, Jun 12)

Posted by Prasanna Santhanam <ts...@apache.org>.
On Sat, Jun 15, 2013 at 01:48:57PM -0400, Chip Childers wrote:
> > There's also a couple of issues here -
> > 1. Does everyone know where the tests run?
> Nope
> > 2. Do people know how to spot the failures?
> Nope
> > 3. Do people know how to find the logs for the failures?
> Nope
> > 
> > If the answer is no to all this, I have more documentation on my
> > hands.

I'll have the documentation draft up soon. Thanks for pointing this
out. All the logs show up under the test-matrix(-extended) job on the
cloudstack-qa view. You can drill down from the "Test Result" shown by
jenkins to see the stacktrace of the failure. For the management
server log, it's a little hidden - it goes under the profile
(hypervisor, ms-distro). For now I'm pulling in management server
logs. Will expose the kvm agent debug logs too.

> > Ideally, I'd like those interested in infra activities to form a
> > separate group for cloudstack-infra related topics. The group's focus
> > will be to maintain, test and add to the infrastructure of this
> > project. But that's in the future. Without such a group, building an
> > IaaS cloud is not much fun :)
> 
> +1 - and at least for now, perhaps we start getting more organized
> around this via dev@cs.a.o using [INFRA] tags.

Will start using the tag as a start.

> 
> Some thoughts I have are: I know that some stuff is being put to use for
> the project in Fremont, but I don't know what it is.  I also don't
> know what hardware donations might be helpful for the environment, so
> that perhaps I could help find something.
> 

Since every $company deploys cloudstack a different way, ideally the
environment should be a small mirror of what is used in production by
$company. That environment can be behind a firewall. What is required
is a jenkins slave that can be either hooked in through jnlp or SSH to
the jenkins.buildacloud.org instance. It will be labelled as a test
slave there and when we need to run tests, we can utilize it for
running tests.  The auth keys can be shared among those interested to
work towards maintaining that infra.

> In all seriousness, if there is a need, I could take up the question at
> $dayjob to provide some testing resources within one of our labs as
> well.  I actually think this would be easier to do then a "donation" of
> hardware that's not really a "donation" to the ASF.  The question is:
> *what's needed* that we don't have already?
> 

Right - donations are (IIUC) only reqd if the ASF infra is going to
manage this. But if there's a group of people within the project
managing this infra and not have it flout any infra rules we're good
to go and get started independantly on this.

We have a single dedicated enviornment that I cycle through deployment
styles that are oft used within Citrix. But obviously others are using
it differently. With perhaps RBD /Ceph, Object stores, OVS, Nicira,
etc. These are not tested.

For specifics on setup and internal resources like - NFS, code
repositories, images repositories, pypi mirrors/caches, log gathering
etc - we can start a separate thread if there is interest.

> > 
> > > 17:44:17 [topcloud]: i can't imagine apache wanting bvt to only run inside citrix all the time.
> > It doesn't run within Citrix. It runs in a DC in Fremont. There are
> > other environments within Citrix however that run their own tests for
> > their needs - eg: object_store tests, cloudplatform tests, customer
> > related issue tests etc.
> > 
> > /me beginning to think more doc work is on my way :/
> 
> Well, really, the key is for us to all know about which infra is being
> shared for the use of the project.  Stuff that's inside a corp that we
> can't all see isn't worth documenting for the project itself.
> 
But it should be if the infra is exposing all troubleshooting tools,
logs to fix cloudstack bugs. If it's running custom builds etc, then I
agree it would not be of much use.

-- 
Prasanna.,

------------------------
Powered by BigRock.com


Re: Infra Issues from the IRC meeting (Wed, Jun 12)

Posted by Chip Childers <ch...@sungard.com>.
On Sat, Jun 15, 2013 at 11:12:37AM +0530, Prasanna Santhanam wrote:
> Saw a few infra topics being discussed in the meeting and there was
> talk of bringing it to the list so I'm taking this opportunity to
> explain some things behind it.
> 
> > 17:22:44 [Animesh]: I will send out my weekly reminder today on status and include Sudha's test results 
> > 17:23:28 [topcloud]: one thing that concerns me is that the bvt continues to be at < 100% pass rate
> > 17:23:41 [topcloud]: is there anything we're doing about this?
> yes - I've fixed most tests. Some have existed because of bugs in
> packaging, systemvm templates that I see patches for now. 
> 
> > 17:25:20 [chipc]: topcloud: was BVT ever at 100% ?
> > 17:25:32 [chipc]: (real question, not sarcasm)
> It was - 100% - when the project was first proposed. But more tests
> have come in since then.
> 
> > 17:26:41 [chipc]: once we get it back to 100%, I say we block all changes when it drops to <100%
> > 17:26:49 [topcloud]: +1
> +1 - this is what I've been driving towards but haven't announced some
> changes I've made in the past weeks because it's pointless to have
> tests fail soon as I announce we are 100%. We shouldn't wait for one
> run, but at least 10 to ensure that the bvt is indeed stable enough to
> be trusted as a 'gating' system for master stability.

+1 - that makes sense

> 
> There's also a couple of issues here -
> 1. Does everyone know where the tests run?

Nope

> 2. Do people know how to spot the failures?

Nope

> 3. Do people know how to find the logs for the failures?

Nope

> 
> If the answer is no to all this, I have more documentation on my
> hands.

Well, perhaps I just haven't looked either!  I see some results when I
search for BVT on the wiki (
https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDSTACK&spaceSearch=true&queryString=BVT
), but they don't obviously tell me where to look.

> 
> > 17:28:07 [Animesh]: agreed bvt also shows progress towards release readines
> > 17:28:07 [chipc]: topcloud: +1
> > 17:28:32 [chipc]: Animesh: BVT should show that master is stable, regardless of release timeframes
> > 17:28:33 [chipc]: IMO that is
> > 17:28:44 [chipc]: master should only see good  /tested code
> Which is why the BVT runs on master at all times on
> jenkins.buildacloud.org. There is also ability to run it against a
> feature branch but I would rather defer that to the release manager
> for now since it's tied with hardware resources and jenkins schedules.
> That feature should strictly be reserved for architecture changes in
> MERGE requests.
> 
> > 17:43:27 [topcloud]: sorry...to bring back this topic but is bvt running on apache infra?
> > 17:43:35 [chipc]: no
> > 17:43:57 [topcloud]: chipc: is there any talk about bringing it into apache infra?
> It was brought up with the ASF infra back in January and the
> suggestion was to donate hardware to the ASF to manage. So if we're
> prepared to do that, great! But it certainly can't just be Citrix :)
> 
> I'd prefer individual project related test hardware and resources
> to stay in the control of the project. Infrastructure is constantly
> changed to allow features and enhancements to be tested so it's best
> to have it in the core group. Which is why jenkins.buildacloud.org
> came to existence. This is similar to how cloudbees operates or
> (*gasp*) openstack-infra [1][2] operates.
> 
> Ideally, I'd like those interested in infra activities to form a
> separate group for cloudstack-infra related topics. The group's focus
> will be to maintain, test and add to the infrastructure of this
> project. But that's in the future. Without such a group, building an
> IaaS cloud is not much fun :)

+1 - and at least for now, perhaps we start getting more organized
around this via dev@cs.a.o using [INFRA] tags.

Some thoughts I have are: I know that some stuff is being put to use for
the project in Fremont, but I don't know what it is.  I also don't
know what hardware donations might be helpful for the environment, so
that perhaps I could help find something.

In all seriousness, if there is a need, I could take up the question at
$dayjob to provide some testing resources within one of our labs as
well.  I actually think this would be easier to do then a "donation" of
hardware that's not really a "donation" to the ASF.  The question is:
*what's needed* that we don't have already?

> 
> > 17:44:17 [topcloud]: i can't imagine apache wanting bvt to only run inside citrix all the time.
> It doesn't run within Citrix. It runs in a DC in Fremont. There are
> other environments within Citrix however that run their own tests for
> their needs - eg: object_store tests, cloudplatform tests, customer
> related issue tests etc.
> 
> /me beginning to think more doc work is on my way :/

Well, really, the key is for us to all know about which infra is being
shared for the use of the project.  Stuff that's inside a corp that we
can't all see isn't worth documenting for the project itself.

> 
> > 17:46:27 [chipc]: but generally, the ASF build infra is a bit overloaded
> +1000
> 
> > 17:46:51 [jzb]: topcloud: when you say "in Citrix" - it's still visible outside Citrix, yes?
> > 17:46:52 [chipc]: so frankly, CTXS donating an environment to run it, publicly visible to everyone, is quite helpful
> > 17:46:58 [chipc]: jzb: it is
> We need more people to donate hardware/virtual resources for testing :) 
> CTXS has been gracious to provide quite a few resources already IMO.
> 
> > 17:47:18 [chipc]: actually, I think it is...  
> > 17:47:34 [topcloud]: jzb: yeah it's still visible but it really should be runnable by everyone.
> Not quite. It's a gating system. It runs automatically and shouldn't
> be runnable by everyone at will. I'm still waiting to implement
> devcloud tests based on that gerrit converstaion (which went nowhere)
> we had many months back. DevCloud stuff can be run at will.
> 
> > 17:47:37 [jzb]: I'm all for building up Apache infra, but I also
> > think having vendors donate publicly visible resources that are
> > usable by the community is acceptable.
> +1
> 
> > 17:47:53 [jzb]: in fact, we probably ought to be hitting up some of
> > our ISP friends for more. 
> +1 - who are our ISP friends? Would like to get help on this.

Several of us. ;-)

> 
> > 17:49:49 [ke4qqq]: so tsp (along with abayer and roman) are working
> > on a publicly accessible jenkins instance in fremont
> This is basically to dogfood all instances via a hosted puppetmaster.
> I've just started so will take time to finish this.
> 
> Thoughts are most welcome!
> 
> [1] https://jenkins.openstack.org/
> [2] http://ci.openstack.org/
> -- 
> Prasanna.,
> 
> ------------------------
> Powered by BigRock.com
> 
>