You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Evans Ye <ev...@apache.org> on 2015/07/09 16:49:07 UTC
Re: Brand new Bigtop AWS account
Sorry I didn't make it clear at the first place.
What I really want to ask is whether we need to have all the smoke
tests/integration tests to be tested and passed before we roll out 1.0.
For example, there might be 2 or 3 failed test in hadoop integration test.
But failed task doesn't mean that the component really has problems.
May I know what did we do in 0.8 release?
2015-07-01 2:43 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> On Mon, Jun 29, 2015 at 09:46PM, Evans Ye wrote:
> > I think Nate answered most of the questions, so thanks a lot!
> > To me the most critical task is to have 1.0 released, while CI overhaul
> is
> > part of the task.
> > From my point of view, there're two thinks holding us back from release.
> >
> > 1. The 1.0 branch is still at bleeding age to support all the claimed
> OSs.
> > The primary issues are in debian(should be fixed now) and oepnsuse
> working
> > as bigtop build slaves.
>
> Sorry guys - I was away for a while and still am catching up here. Evans,
> are
> you saying that there're issues that need to be fixed in the trunk and
> branch-1.0 as well? Or these are just CI related and do no require
> additional
> commits into the branch?
>
> The release criteria for us is pretty much what we define it to be ;) If we
> see 'good-enough' quality of the produced packages/tests/deployment
> mechanism
> - we can start the vote for the release any moment. Mind you - the packages
> aren't required for the release, as any ASF release is source code only.
> We do
> binaries as a extra service to our users.
>
> Cos
>
> > 2. Migrating the CI master to EC2 under EMR donated account. Since I
> can't
> > get in the old CI master, there's no way for me to retrieve current job
> > configs and restore them back to new CI master. Roman probably has
> backups
> > but he might be busying on other things.
> >
> > 3. I'll let cos add more since I'm not quite familiar with release
> > criteria, but I'd love to help.
> >
> > I guess we can't have 2 solved in short period, so I might start to work
> on
> > setting up repositories on old CI master directly.
> >
> > Thanks,
> > Evans
> >
> > 2015-06-27 3:46 GMT+08:00 <na...@reactor8.com>:
> >
> > > Still work in progress, will let Evans add some more details as he has
> been
> > > the one kickin but on CI tasks. Besides some more setup and planned
> > > enhancements focus is getting 1.0 out the door, then turning towards
> more
> > > foundational CI work for 1.x and beyond.
> > >
> > > Think we will be well served having more official bigtop repos
> serving, I
> > > setup a prototype off 0.8, serving from S3.., few things for us to
> figure
> > > out build/process wise to properly hook the CI into S3/repo publishing
> > >
> > > Evans created this jira for general tracking:
> > > https://issues.apache.org/jira/browse/BIGTOP-1897
> > >
> > > Not all of which are required for new CI to be ready, for example newly
> > > added one for optimizing build process to reuse artifacts can start
> working
> > > on in parallel to new CI and 1.0 release but get completed after both
> are
> > > taken care of
> > >
> > > -----Original Message-----
> > > From: Zeng, Tom [mailto:tomzeng@amazon.com]
> > > Sent: Thursday, June 25, 2015 3:36 PM
> > > To: Evans Ye; Konstantin Boudnik
> > > Cc: dev@bigtop.apache.org
> > > Subject: Re: Brand new Bigtop AWS account
> > >
> > > Hi Evans and Cos, what's the status of this, do we have the build CI
> > > environment set up yet?
> > >
> > > Thanks,
> > > Tom
> > >
> > > From: Evans Ye <ev...@apache.org>>
> > > Date: Sunday, April 19, 2015 at 11:12 AM
> > > To: "dev@bigtop.apache.org<ma...@bigtop.apache.org>"
> > > <de...@bigtop.apache.org>>, "Zeng, Tom"
> > > <to...@amazon.com>>
> > > Subject: Re: Brand new Bigtop AWS account
> > >
> > > Hi guys,
> > >
> > > I've copied security group settings from the old account to the new one
> > > provided by EMR team (thanks Tom!).
> > > I've also checked our existing jenkins and there are some useful jobs
> > > already defined. It seems that I only need to have some slightly
> > > modification on them to get it start working.
> > > My plan is to create a m3.xlarge instance and directly plug it on our
> > > existing jenkins as a slave so that I can start to make some progress.
> > > I'll keep you posted. Suggestions are welcome as well. :)
> > >
> > > Thanks,
> > > Evans
> > >
> > >
> > > 2015-04-09 13:52 GMT+08:00 Konstantin Boudnik
> > > <co...@apache.org>>:
> > > Cycling back: today we got a credit coupon from Amazon EMR team so we
> have
> > > everything to start building new CI infra. Please contact me directly
> if
> > > you
> > > plan to help with setting it up, so I can create an account with proper
> > > permissions for you.
> > >
> > > Special thanks are going to Tom Zeng and his team for making it happen!
> > > Thank you very much guys!
> > >
> > > Cos
> > >
> > > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
> > > > Guys,
> > > >
> > > > I want to start a separate thread to track the CI preparations for
> the
> > > > release next month (fingers crossed). Clearly, we can make a release
> > > > without CI, but it'd way easier to test and create binary artifacts
> if
> > > > we have a working environment for official validation. Roman has done
> > > > a lot in this direction (many thanks!), but there are still a few
> > > > rough edges, which might be easy to finish of.
> > > >
> > > > I want to figure out a couple of things:
> > > > - what's the state of CI and how much still needs to be done (Rvs?
> Could
> > > you
> > > > share any first hand feedback?)
> > > > - who would be able to help with the CI completion? I can commit
> some of
> > > my
> > > > cycles, but it'd be great to have few more hands on that. Clearly,
> > > some
> > > > Jenkins-foo and prior CI skills won't hurt ;)
> > > >
> > > > Please chime in if you can help. Thanks a lot!
> > > > Cos
> > > >
> > >
> > >
> > >
>
Re: Brand new Bigtop AWS account
Posted by Jay Vyas <ja...@gmail.com>.
I was just answering the question of "do all tests need to pass for release"... Imo not really :).
The issue number for hive tests? That was an issue in the past https://issues.apache.org/jira/plugins/servlet/mobile#issue/BIGTOP-1450 fixed it .
> On Jul 11, 2015, at 4:12 AM, Olaf Flebbe <of...@oflebbe.de> wrote:
>
> Jay,
>
> What's the JIRA Issue number?
>
> Olaf
>
>> Am 10.07.2015 um 23:30 schrieb Jay Vyas <ja...@gmail.com>:
>>
>> My thought is no. Hive tests were broken in 0.8 and we released hive. We continue evolving on head so it's ok. But in pretty lax about this stuff
>>
>> Sent from my iPhone
>>
>>> On Jul 9, 2015, at 10:49 AM, Evans Ye <ev...@apache.org> wrote:
>>>
>>> Sorry I didn't make it clear at the first place.
>>> What I really want to ask is whether we need to have all the smoke
>>> tests/integration tests to be tested and passed before we roll out 1.0.
>>> For example, there might be 2 or 3 failed test in hadoop integration test.
>>> But failed task doesn't mean that the component really has problems.
>>> May I know what did we do in 0.8 release?
>>>
>>> 2015-07-01 2:43 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>>>
>>>>> On Mon, Jun 29, 2015 at 09:46PM, Evans Ye wrote:
>>>>> I think Nate answered most of the questions, so thanks a lot!
>>>>> To me the most critical task is to have 1.0 released, while CI overhaul
>>>> is
>>>>> part of the task.
>>>>> From my point of view, there're two thinks holding us back from release.
>>>>>
>>>>> 1. The 1.0 branch is still at bleeding age to support all the claimed
>>>> OSs.
>>>>> The primary issues are in debian(should be fixed now) and oepnsuse
>>>> working
>>>>> as bigtop build slaves.
>>>>
>>>> Sorry guys - I was away for a while and still am catching up here. Evans,
>>>> are
>>>> you saying that there're issues that need to be fixed in the trunk and
>>>> branch-1.0 as well? Or these are just CI related and do no require
>>>> additional
>>>> commits into the branch?
>>>>
>>>> The release criteria for us is pretty much what we define it to be ;) If we
>>>> see 'good-enough' quality of the produced packages/tests/deployment
>>>> mechanism
>>>> - we can start the vote for the release any moment. Mind you - the packages
>>>> aren't required for the release, as any ASF release is source code only.
>>>> We do
>>>> binaries as a extra service to our users.
>>>>
>>>> Cos
>>>>
>>>>> 2. Migrating the CI master to EC2 under EMR donated account. Since I
>>>> can't
>>>>> get in the old CI master, there's no way for me to retrieve current job
>>>>> configs and restore them back to new CI master. Roman probably has
>>>> backups
>>>>> but he might be busying on other things.
>>>>>
>>>>> 3. I'll let cos add more since I'm not quite familiar with release
>>>>> criteria, but I'd love to help.
>>>>>
>>>>> I guess we can't have 2 solved in short period, so I might start to work
>>>> on
>>>>> setting up repositories on old CI master directly.
>>>>>
>>>>> Thanks,
>>>>> Evans
>>>>>
>>>>> 2015-06-27 3:46 GMT+08:00 <na...@reactor8.com>:
>>>>>
>>>>>> Still work in progress, will let Evans add some more details as he has
>>>> been
>>>>>> the one kickin but on CI tasks. Besides some more setup and planned
>>>>>> enhancements focus is getting 1.0 out the door, then turning towards
>>>> more
>>>>>> foundational CI work for 1.x and beyond.
>>>>>>
>>>>>> Think we will be well served having more official bigtop repos
>>>> serving, I
>>>>>> setup a prototype off 0.8, serving from S3.., few things for us to
>>>> figure
>>>>>> out build/process wise to properly hook the CI into S3/repo publishing
>>>>>>
>>>>>> Evans created this jira for general tracking:
>>>>>> https://issues.apache.org/jira/browse/BIGTOP-1897
>>>>>>
>>>>>> Not all of which are required for new CI to be ready, for example newly
>>>>>> added one for optimizing build process to reuse artifacts can start
>>>> working
>>>>>> on in parallel to new CI and 1.0 release but get completed after both
>>>> are
>>>>>> taken care of
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Zeng, Tom [mailto:tomzeng@amazon.com]
>>>>>> Sent: Thursday, June 25, 2015 3:36 PM
>>>>>> To: Evans Ye; Konstantin Boudnik
>>>>>> Cc: dev@bigtop.apache.org
>>>>>> Subject: Re: Brand new Bigtop AWS account
>>>>>>
>>>>>> Hi Evans and Cos, what's the status of this, do we have the build CI
>>>>>> environment set up yet?
>>>>>>
>>>>>> Thanks,
>>>>>> Tom
>>>>>>
>>>>>> From: Evans Ye <ev...@apache.org>>
>>>>>> Date: Sunday, April 19, 2015 at 11:12 AM
>>>>>> To: "dev@bigtop.apache.org<ma...@bigtop.apache.org>"
>>>>>> <de...@bigtop.apache.org>>, "Zeng, Tom"
>>>>>> <to...@amazon.com>>
>>>>>> Subject: Re: Brand new Bigtop AWS account
>>>>>>
>>>>>> Hi guys,
>>>>>>
>>>>>> I've copied security group settings from the old account to the new one
>>>>>> provided by EMR team (thanks Tom!).
>>>>>> I've also checked our existing jenkins and there are some useful jobs
>>>>>> already defined. It seems that I only need to have some slightly
>>>>>> modification on them to get it start working.
>>>>>> My plan is to create a m3.xlarge instance and directly plug it on our
>>>>>> existing jenkins as a slave so that I can start to make some progress.
>>>>>> I'll keep you posted. Suggestions are welcome as well. :)
>>>>>>
>>>>>> Thanks,
>>>>>> Evans
>>>>>>
>>>>>>
>>>>>> 2015-04-09 13:52 GMT+08:00 Konstantin Boudnik
>>>>>> <co...@apache.org>>:
>>>>>> Cycling back: today we got a credit coupon from Amazon EMR team so we
>>>> have
>>>>>> everything to start building new CI infra. Please contact me directly
>>>> if
>>>>>> you
>>>>>> plan to help with setting it up, so I can create an account with proper
>>>>>> permissions for you.
>>>>>>
>>>>>> Special thanks are going to Tom Zeng and his team for making it happen!
>>>>>> Thank you very much guys!
>>>>>>
>>>>>> Cos
>>>>>>
>>>>>>> On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
>>>>>>> Guys,
>>>>>>>
>>>>>>> I want to start a separate thread to track the CI preparations for
>>>> the
>>>>>>> release next month (fingers crossed). Clearly, we can make a release
>>>>>>> without CI, but it'd way easier to test and create binary artifacts
>>>> if
>>>>>>> we have a working environment for official validation. Roman has done
>>>>>>> a lot in this direction (many thanks!), but there are still a few
>>>>>>> rough edges, which might be easy to finish of.
>>>>>>>
>>>>>>> I want to figure out a couple of things:
>>>>>>> - what's the state of CI and how much still needs to be done (Rvs?
>>>> Could
>>>>>> you
>>>>>>> share any first hand feedback?)
>>>>>>> - who would be able to help with the CI completion? I can commit
>>>> some of
>>>>>> my
>>>>>>> cycles, but it'd be great to have few more hands on that. Clearly,
>>>>>> some
>>>>>>> Jenkins-foo and prior CI skills won't hurt ;)
>>>>>>>
>>>>>>> Please chime in if you can help. Thanks a lot!
>>>>>>> Cos
>
Re: Brand new Bigtop AWS account
Posted by Olaf Flebbe <of...@oflebbe.de>.
Jay,
What's the JIRA Issue number?
Olaf
> Am 10.07.2015 um 23:30 schrieb Jay Vyas <ja...@gmail.com>:
>
> My thought is no. Hive tests were broken in 0.8 and we released hive. We continue evolving on head so it's ok. But in pretty lax about this stuff
>
> Sent from my iPhone
>
>> On Jul 9, 2015, at 10:49 AM, Evans Ye <ev...@apache.org> wrote:
>>
>> Sorry I didn't make it clear at the first place.
>> What I really want to ask is whether we need to have all the smoke
>> tests/integration tests to be tested and passed before we roll out 1.0.
>> For example, there might be 2 or 3 failed test in hadoop integration test.
>> But failed task doesn't mean that the component really has problems.
>> May I know what did we do in 0.8 release?
>>
>> 2015-07-01 2:43 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>>
>>>> On Mon, Jun 29, 2015 at 09:46PM, Evans Ye wrote:
>>>> I think Nate answered most of the questions, so thanks a lot!
>>>> To me the most critical task is to have 1.0 released, while CI overhaul
>>> is
>>>> part of the task.
>>>> From my point of view, there're two thinks holding us back from release.
>>>>
>>>> 1. The 1.0 branch is still at bleeding age to support all the claimed
>>> OSs.
>>>> The primary issues are in debian(should be fixed now) and oepnsuse
>>> working
>>>> as bigtop build slaves.
>>>
>>> Sorry guys - I was away for a while and still am catching up here. Evans,
>>> are
>>> you saying that there're issues that need to be fixed in the trunk and
>>> branch-1.0 as well? Or these are just CI related and do no require
>>> additional
>>> commits into the branch?
>>>
>>> The release criteria for us is pretty much what we define it to be ;) If we
>>> see 'good-enough' quality of the produced packages/tests/deployment
>>> mechanism
>>> - we can start the vote for the release any moment. Mind you - the packages
>>> aren't required for the release, as any ASF release is source code only.
>>> We do
>>> binaries as a extra service to our users.
>>>
>>> Cos
>>>
>>>> 2. Migrating the CI master to EC2 under EMR donated account. Since I
>>> can't
>>>> get in the old CI master, there's no way for me to retrieve current job
>>>> configs and restore them back to new CI master. Roman probably has
>>> backups
>>>> but he might be busying on other things.
>>>>
>>>> 3. I'll let cos add more since I'm not quite familiar with release
>>>> criteria, but I'd love to help.
>>>>
>>>> I guess we can't have 2 solved in short period, so I might start to work
>>> on
>>>> setting up repositories on old CI master directly.
>>>>
>>>> Thanks,
>>>> Evans
>>>>
>>>> 2015-06-27 3:46 GMT+08:00 <na...@reactor8.com>:
>>>>
>>>>> Still work in progress, will let Evans add some more details as he has
>>> been
>>>>> the one kickin but on CI tasks. Besides some more setup and planned
>>>>> enhancements focus is getting 1.0 out the door, then turning towards
>>> more
>>>>> foundational CI work for 1.x and beyond.
>>>>>
>>>>> Think we will be well served having more official bigtop repos
>>> serving, I
>>>>> setup a prototype off 0.8, serving from S3.., few things for us to
>>> figure
>>>>> out build/process wise to properly hook the CI into S3/repo publishing
>>>>>
>>>>> Evans created this jira for general tracking:
>>>>> https://issues.apache.org/jira/browse/BIGTOP-1897
>>>>>
>>>>> Not all of which are required for new CI to be ready, for example newly
>>>>> added one for optimizing build process to reuse artifacts can start
>>> working
>>>>> on in parallel to new CI and 1.0 release but get completed after both
>>> are
>>>>> taken care of
>>>>>
>>>>> -----Original Message-----
>>>>> From: Zeng, Tom [mailto:tomzeng@amazon.com]
>>>>> Sent: Thursday, June 25, 2015 3:36 PM
>>>>> To: Evans Ye; Konstantin Boudnik
>>>>> Cc: dev@bigtop.apache.org
>>>>> Subject: Re: Brand new Bigtop AWS account
>>>>>
>>>>> Hi Evans and Cos, what's the status of this, do we have the build CI
>>>>> environment set up yet?
>>>>>
>>>>> Thanks,
>>>>> Tom
>>>>>
>>>>> From: Evans Ye <ev...@apache.org>>
>>>>> Date: Sunday, April 19, 2015 at 11:12 AM
>>>>> To: "dev@bigtop.apache.org<ma...@bigtop.apache.org>"
>>>>> <de...@bigtop.apache.org>>, "Zeng, Tom"
>>>>> <to...@amazon.com>>
>>>>> Subject: Re: Brand new Bigtop AWS account
>>>>>
>>>>> Hi guys,
>>>>>
>>>>> I've copied security group settings from the old account to the new one
>>>>> provided by EMR team (thanks Tom!).
>>>>> I've also checked our existing jenkins and there are some useful jobs
>>>>> already defined. It seems that I only need to have some slightly
>>>>> modification on them to get it start working.
>>>>> My plan is to create a m3.xlarge instance and directly plug it on our
>>>>> existing jenkins as a slave so that I can start to make some progress.
>>>>> I'll keep you posted. Suggestions are welcome as well. :)
>>>>>
>>>>> Thanks,
>>>>> Evans
>>>>>
>>>>>
>>>>> 2015-04-09 13:52 GMT+08:00 Konstantin Boudnik
>>>>> <co...@apache.org>>:
>>>>> Cycling back: today we got a credit coupon from Amazon EMR team so we
>>> have
>>>>> everything to start building new CI infra. Please contact me directly
>>> if
>>>>> you
>>>>> plan to help with setting it up, so I can create an account with proper
>>>>> permissions for you.
>>>>>
>>>>> Special thanks are going to Tom Zeng and his team for making it happen!
>>>>> Thank you very much guys!
>>>>>
>>>>> Cos
>>>>>
>>>>>> On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
>>>>>> Guys,
>>>>>>
>>>>>> I want to start a separate thread to track the CI preparations for
>>> the
>>>>>> release next month (fingers crossed). Clearly, we can make a release
>>>>>> without CI, but it'd way easier to test and create binary artifacts
>>> if
>>>>>> we have a working environment for official validation. Roman has done
>>>>>> a lot in this direction (many thanks!), but there are still a few
>>>>>> rough edges, which might be easy to finish of.
>>>>>>
>>>>>> I want to figure out a couple of things:
>>>>>> - what's the state of CI and how much still needs to be done (Rvs?
>>> Could
>>>>> you
>>>>>> share any first hand feedback?)
>>>>>> - who would be able to help with the CI completion? I can commit
>>> some of
>>>>> my
>>>>>> cycles, but it'd be great to have few more hands on that. Clearly,
>>>>> some
>>>>>> Jenkins-foo and prior CI skills won't hurt ;)
>>>>>>
>>>>>> Please chime in if you can help. Thanks a lot!
>>>>>> Cos
>>>
RE: Brand new Bigtop AWS account
Posted by na...@reactor8.com.
Maybe another thing we can add to post 1.0 and ongoing CI evolution is
documenting to some extent must have and nice to have tests for each
component, eventually having CI green/red have meaning interms of a
bundle/release of bigtop
-----Original Message-----
From: Jay Vyas [mailto:jayunit100.apache@gmail.com]
Sent: Friday, July 10, 2015 2:30 PM
To: dev@bigtop.apache.org
Subject: Re: Brand new Bigtop AWS account
My thought is no. Hive tests were broken in 0.8 and we released hive. We
continue evolving on head so it's ok. But in pretty lax about this stuff
Sent from my iPhone
> On Jul 9, 2015, at 10:49 AM, Evans Ye <ev...@apache.org> wrote:
>
> Sorry I didn't make it clear at the first place.
> What I really want to ask is whether we need to have all the smoke
> tests/integration tests to be tested and passed before we roll out 1.0.
> For example, there might be 2 or 3 failed test in hadoop integration test.
> But failed task doesn't mean that the component really has problems.
> May I know what did we do in 0.8 release?
>
> 2015-07-01 2:43 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>
>>> On Mon, Jun 29, 2015 at 09:46PM, Evans Ye wrote:
>>> I think Nate answered most of the questions, so thanks a lot!
>>> To me the most critical task is to have 1.0 released, while CI
>>> overhaul
>> is
>>> part of the task.
>>> From my point of view, there're two thinks holding us back from release.
>>>
>>> 1. The 1.0 branch is still at bleeding age to support all the
>>> claimed
>> OSs.
>>> The primary issues are in debian(should be fixed now) and oepnsuse
>> working
>>> as bigtop build slaves.
>>
>> Sorry guys - I was away for a while and still am catching up here.
>> Evans, are you saying that there're issues that need to be fixed in
>> the trunk and
>> branch-1.0 as well? Or these are just CI related and do no require
>> additional commits into the branch?
>>
>> The release criteria for us is pretty much what we define it to be ;)
>> If we see 'good-enough' quality of the produced
>> packages/tests/deployment mechanism
>> - we can start the vote for the release any moment. Mind you - the
>> packages aren't required for the release, as any ASF release is source
code only.
>> We do
>> binaries as a extra service to our users.
>>
>> Cos
>>
>>> 2. Migrating the CI master to EC2 under EMR donated account. Since I
>> can't
>>> get in the old CI master, there's no way for me to retrieve current
>>> job configs and restore them back to new CI master. Roman probably
>>> has
>> backups
>>> but he might be busying on other things.
>>>
>>> 3. I'll let cos add more since I'm not quite familiar with release
>>> criteria, but I'd love to help.
>>>
>>> I guess we can't have 2 solved in short period, so I might start to
>>> work
>> on
>>> setting up repositories on old CI master directly.
>>>
>>> Thanks,
>>> Evans
>>>
>>> 2015-06-27 3:46 GMT+08:00 <na...@reactor8.com>:
>>>
>>>> Still work in progress, will let Evans add some more details as he
>>>> has
>> been
>>>> the one kickin but on CI tasks. Besides some more setup and
>>>> planned enhancements focus is getting 1.0 out the door, then
>>>> turning towards
>> more
>>>> foundational CI work for 1.x and beyond.
>>>>
>>>> Think we will be well served having more official bigtop repos
>> serving, I
>>>> setup a prototype off 0.8, serving from S3.., few things for us to
>> figure
>>>> out build/process wise to properly hook the CI into S3/repo
>>>> publishing
>>>>
>>>> Evans created this jira for general tracking:
>>>> https://issues.apache.org/jira/browse/BIGTOP-1897
>>>>
>>>> Not all of which are required for new CI to be ready, for example
>>>> newly added one for optimizing build process to reuse artifacts can
>>>> start
>> working
>>>> on in parallel to new CI and 1.0 release but get completed after
>>>> both
>> are
>>>> taken care of
>>>>
>>>> -----Original Message-----
>>>> From: Zeng, Tom [mailto:tomzeng@amazon.com]
>>>> Sent: Thursday, June 25, 2015 3:36 PM
>>>> To: Evans Ye; Konstantin Boudnik
>>>> Cc: dev@bigtop.apache.org
>>>> Subject: Re: Brand new Bigtop AWS account
>>>>
>>>> Hi Evans and Cos, what's the status of this, do we have the build
>>>> CI environment set up yet?
>>>>
>>>> Thanks,
>>>> Tom
>>>>
>>>> From: Evans Ye <ev...@apache.org>>
>>>> Date: Sunday, April 19, 2015 at 11:12 AM
>>>> To: "dev@bigtop.apache.org<ma...@bigtop.apache.org>"
>>>> <de...@bigtop.apache.org>>, "Zeng, Tom"
>>>> <to...@amazon.com>>
>>>> Subject: Re: Brand new Bigtop AWS account
>>>>
>>>> Hi guys,
>>>>
>>>> I've copied security group settings from the old account to the new
>>>> one provided by EMR team (thanks Tom!).
>>>> I've also checked our existing jenkins and there are some useful
>>>> jobs already defined. It seems that I only need to have some
>>>> slightly modification on them to get it start working.
>>>> My plan is to create a m3.xlarge instance and directly plug it on
>>>> our existing jenkins as a slave so that I can start to make some
progress.
>>>> I'll keep you posted. Suggestions are welcome as well. :)
>>>>
>>>> Thanks,
>>>> Evans
>>>>
>>>>
>>>> 2015-04-09 13:52 GMT+08:00 Konstantin Boudnik
>>>> <co...@apache.org>>:
>>>> Cycling back: today we got a credit coupon from Amazon EMR team so
>>>> we
>> have
>>>> everything to start building new CI infra. Please contact me
>>>> directly
>> if
>>>> you
>>>> plan to help with setting it up, so I can create an account with
>>>> proper permissions for you.
>>>>
>>>> Special thanks are going to Tom Zeng and his team for making it happen!
>>>> Thank you very much guys!
>>>>
>>>> Cos
>>>>
>>>>> On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
>>>>> Guys,
>>>>>
>>>>> I want to start a separate thread to track the CI preparations for
>> the
>>>>> release next month (fingers crossed). Clearly, we can make a
>>>>> release without CI, but it'd way easier to test and create binary
>>>>> artifacts
>> if
>>>>> we have a working environment for official validation. Roman has
>>>>> done a lot in this direction (many thanks!), but there are still a
>>>>> few rough edges, which might be easy to finish of.
>>>>>
>>>>> I want to figure out a couple of things:
>>>>> - what's the state of CI and how much still needs to be done (Rvs?
>> Could
>>>> you
>>>>> share any first hand feedback?)
>>>>> - who would be able to help with the CI completion? I can commit
>> some of
>>>> my
>>>>> cycles, but it'd be great to have few more hands on that.
>>>>> Clearly,
>>>> some
>>>>> Jenkins-foo and prior CI skills won't hurt ;)
>>>>>
>>>>> Please chime in if you can help. Thanks a lot!
>>>>> Cos
>>
Re: Brand new Bigtop AWS account
Posted by Jay Vyas <ja...@gmail.com>.
My thought is no. Hive tests were broken in 0.8 and we released hive. We continue evolving on head so it's ok. But in pretty lax about this stuff
Sent from my iPhone
> On Jul 9, 2015, at 10:49 AM, Evans Ye <ev...@apache.org> wrote:
>
> Sorry I didn't make it clear at the first place.
> What I really want to ask is whether we need to have all the smoke
> tests/integration tests to be tested and passed before we roll out 1.0.
> For example, there might be 2 or 3 failed test in hadoop integration test.
> But failed task doesn't mean that the component really has problems.
> May I know what did we do in 0.8 release?
>
> 2015-07-01 2:43 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>
>>> On Mon, Jun 29, 2015 at 09:46PM, Evans Ye wrote:
>>> I think Nate answered most of the questions, so thanks a lot!
>>> To me the most critical task is to have 1.0 released, while CI overhaul
>> is
>>> part of the task.
>>> From my point of view, there're two thinks holding us back from release.
>>>
>>> 1. The 1.0 branch is still at bleeding age to support all the claimed
>> OSs.
>>> The primary issues are in debian(should be fixed now) and oepnsuse
>> working
>>> as bigtop build slaves.
>>
>> Sorry guys - I was away for a while and still am catching up here. Evans,
>> are
>> you saying that there're issues that need to be fixed in the trunk and
>> branch-1.0 as well? Or these are just CI related and do no require
>> additional
>> commits into the branch?
>>
>> The release criteria for us is pretty much what we define it to be ;) If we
>> see 'good-enough' quality of the produced packages/tests/deployment
>> mechanism
>> - we can start the vote for the release any moment. Mind you - the packages
>> aren't required for the release, as any ASF release is source code only.
>> We do
>> binaries as a extra service to our users.
>>
>> Cos
>>
>>> 2. Migrating the CI master to EC2 under EMR donated account. Since I
>> can't
>>> get in the old CI master, there's no way for me to retrieve current job
>>> configs and restore them back to new CI master. Roman probably has
>> backups
>>> but he might be busying on other things.
>>>
>>> 3. I'll let cos add more since I'm not quite familiar with release
>>> criteria, but I'd love to help.
>>>
>>> I guess we can't have 2 solved in short period, so I might start to work
>> on
>>> setting up repositories on old CI master directly.
>>>
>>> Thanks,
>>> Evans
>>>
>>> 2015-06-27 3:46 GMT+08:00 <na...@reactor8.com>:
>>>
>>>> Still work in progress, will let Evans add some more details as he has
>> been
>>>> the one kickin but on CI tasks. Besides some more setup and planned
>>>> enhancements focus is getting 1.0 out the door, then turning towards
>> more
>>>> foundational CI work for 1.x and beyond.
>>>>
>>>> Think we will be well served having more official bigtop repos
>> serving, I
>>>> setup a prototype off 0.8, serving from S3.., few things for us to
>> figure
>>>> out build/process wise to properly hook the CI into S3/repo publishing
>>>>
>>>> Evans created this jira for general tracking:
>>>> https://issues.apache.org/jira/browse/BIGTOP-1897
>>>>
>>>> Not all of which are required for new CI to be ready, for example newly
>>>> added one for optimizing build process to reuse artifacts can start
>> working
>>>> on in parallel to new CI and 1.0 release but get completed after both
>> are
>>>> taken care of
>>>>
>>>> -----Original Message-----
>>>> From: Zeng, Tom [mailto:tomzeng@amazon.com]
>>>> Sent: Thursday, June 25, 2015 3:36 PM
>>>> To: Evans Ye; Konstantin Boudnik
>>>> Cc: dev@bigtop.apache.org
>>>> Subject: Re: Brand new Bigtop AWS account
>>>>
>>>> Hi Evans and Cos, what's the status of this, do we have the build CI
>>>> environment set up yet?
>>>>
>>>> Thanks,
>>>> Tom
>>>>
>>>> From: Evans Ye <ev...@apache.org>>
>>>> Date: Sunday, April 19, 2015 at 11:12 AM
>>>> To: "dev@bigtop.apache.org<ma...@bigtop.apache.org>"
>>>> <de...@bigtop.apache.org>>, "Zeng, Tom"
>>>> <to...@amazon.com>>
>>>> Subject: Re: Brand new Bigtop AWS account
>>>>
>>>> Hi guys,
>>>>
>>>> I've copied security group settings from the old account to the new one
>>>> provided by EMR team (thanks Tom!).
>>>> I've also checked our existing jenkins and there are some useful jobs
>>>> already defined. It seems that I only need to have some slightly
>>>> modification on them to get it start working.
>>>> My plan is to create a m3.xlarge instance and directly plug it on our
>>>> existing jenkins as a slave so that I can start to make some progress.
>>>> I'll keep you posted. Suggestions are welcome as well. :)
>>>>
>>>> Thanks,
>>>> Evans
>>>>
>>>>
>>>> 2015-04-09 13:52 GMT+08:00 Konstantin Boudnik
>>>> <co...@apache.org>>:
>>>> Cycling back: today we got a credit coupon from Amazon EMR team so we
>> have
>>>> everything to start building new CI infra. Please contact me directly
>> if
>>>> you
>>>> plan to help with setting it up, so I can create an account with proper
>>>> permissions for you.
>>>>
>>>> Special thanks are going to Tom Zeng and his team for making it happen!
>>>> Thank you very much guys!
>>>>
>>>> Cos
>>>>
>>>>> On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
>>>>> Guys,
>>>>>
>>>>> I want to start a separate thread to track the CI preparations for
>> the
>>>>> release next month (fingers crossed). Clearly, we can make a release
>>>>> without CI, but it'd way easier to test and create binary artifacts
>> if
>>>>> we have a working environment for official validation. Roman has done
>>>>> a lot in this direction (many thanks!), but there are still a few
>>>>> rough edges, which might be easy to finish of.
>>>>>
>>>>> I want to figure out a couple of things:
>>>>> - what's the state of CI and how much still needs to be done (Rvs?
>> Could
>>>> you
>>>>> share any first hand feedback?)
>>>>> - who would be able to help with the CI completion? I can commit
>> some of
>>>> my
>>>>> cycles, but it'd be great to have few more hands on that. Clearly,
>>>> some
>>>>> Jenkins-foo and prior CI skills won't hurt ;)
>>>>>
>>>>> Please chime in if you can help. Thanks a lot!
>>>>> Cos
>>
Re: Brand new Bigtop AWS account
Posted by Evans Ye <ev...@apache.org>.
Got it. I know where we are and where we can head for. Thanks for
answering. :)
2015年7月14日 上午9:29於 "Jay Vyas" <ja...@gmail.com>寫道：
> Agree with cos. Tests are utilities that we ship but not always perfect
> given the rapid pace of change
>
> > On Jul 13, 2015, at 6:03 PM, Konstantin Boudnik <co...@apache.org> wrote:
> >
> > I don't think we ever had _all_ tests green considering how crappy Hive
> tests
> > are, etc. I'd say we can punt on a few failing ones. I know I sound
> terrible
> > ;(
> >
> > Cos
> >
> >> On Thu, Jul 09, 2015 at 10:49PM, Evans Ye wrote:
> >> Sorry I didn't make it clear at the first place.
> >> What I really want to ask is whether we need to have all the smoke
> >> tests/integration tests to be tested and passed before we roll out 1.0.
> >> For example, there might be 2 or 3 failed test in hadoop integration
> test.
> >> But failed task doesn't mean that the component really has problems.
> >> May I know what did we do in 0.8 release?
> >>
> >> 2015-07-01 2:43 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> >>
> >>>> On Mon, Jun 29, 2015 at 09:46PM, Evans Ye wrote:
> >>>> I think Nate answered most of the questions, so thanks a lot!
> >>>> To me the most critical task is to have 1.0 released, while CI
> overhaul
> >>> is
> >>>> part of the task.
> >>>> From my point of view, there're two thinks holding us back from
> release.
> >>>>
> >>>> 1. The 1.0 branch is still at bleeding age to support all the claimed
> >>> OSs.
> >>>> The primary issues are in debian(should be fixed now) and oepnsuse
> >>> working
> >>>> as bigtop build slaves.
> >>>
> >>> Sorry guys - I was away for a while and still am catching up here.
> Evans,
> >>> are
> >>> you saying that there're issues that need to be fixed in the trunk and
> >>> branch-1.0 as well? Or these are just CI related and do no require
> >>> additional
> >>> commits into the branch?
> >>>
> >>> The release criteria for us is pretty much what we define it to be ;)
> If we
> >>> see 'good-enough' quality of the produced packages/tests/deployment
> >>> mechanism
> >>> - we can start the vote for the release any moment. Mind you - the
> packages
> >>> aren't required for the release, as any ASF release is source code
> only.
> >>> We do
> >>> binaries as a extra service to our users.
> >>>
> >>> Cos
> >>>
> >>>> 2. Migrating the CI master to EC2 under EMR donated account. Since I
> >>> can't
> >>>> get in the old CI master, there's no way for me to retrieve current
> job
> >>>> configs and restore them back to new CI master. Roman probably has
> >>> backups
> >>>> but he might be busying on other things.
> >>>>
> >>>> 3. I'll let cos add more since I'm not quite familiar with release
> >>>> criteria, but I'd love to help.
> >>>>
> >>>> I guess we can't have 2 solved in short period, so I might start to
> work
> >>> on
> >>>> setting up repositories on old CI master directly.
> >>>>
> >>>> Thanks,
> >>>> Evans
> >>>>
> >>>> 2015-06-27 3:46 GMT+08:00 <na...@reactor8.com>:
> >>>>
> >>>>> Still work in progress, will let Evans add some more details as he
> has
> >>> been
> >>>>> the one kickin but on CI tasks. Besides some more setup and planned
> >>>>> enhancements focus is getting 1.0 out the door, then turning towards
> >>> more
> >>>>> foundational CI work for 1.x and beyond.
> >>>>>
> >>>>> Think we will be well served having more official bigtop repos
> >>> serving, I
> >>>>> setup a prototype off 0.8, serving from S3.., few things for us to
> >>> figure
> >>>>> out build/process wise to properly hook the CI into S3/repo
> publishing
> >>>>>
> >>>>> Evans created this jira for general tracking:
> >>>>> https://issues.apache.org/jira/browse/BIGTOP-1897
> >>>>>
> >>>>> Not all of which are required for new CI to be ready, for example
> newly
> >>>>> added one for optimizing build process to reuse artifacts can start
> >>> working
> >>>>> on in parallel to new CI and 1.0 release but get completed after both
> >>> are
> >>>>> taken care of
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: Zeng, Tom [mailto:tomzeng@amazon.com]
> >>>>> Sent: Thursday, June 25, 2015 3:36 PM
> >>>>> To: Evans Ye; Konstantin Boudnik
> >>>>> Cc: dev@bigtop.apache.org
> >>>>> Subject: Re: Brand new Bigtop AWS account
> >>>>>
> >>>>> Hi Evans and Cos, what's the status of this, do we have the build CI
> >>>>> environment set up yet?
> >>>>>
> >>>>> Thanks,
> >>>>> Tom
> >>>>>
> >>>>> From: Evans Ye <ev...@apache.org>>
> >>>>> Date: Sunday, April 19, 2015 at 11:12 AM
> >>>>> To: "dev@bigtop.apache.org<ma...@bigtop.apache.org>"
> >>>>> <de...@bigtop.apache.org>>, "Zeng, Tom"
> >>>>> <to...@amazon.com>>
> >>>>> Subject: Re: Brand new Bigtop AWS account
> >>>>>
> >>>>> Hi guys,
> >>>>>
> >>>>> I've copied security group settings from the old account to the new
> one
> >>>>> provided by EMR team (thanks Tom!).
> >>>>> I've also checked our existing jenkins and there are some useful jobs
> >>>>> already defined. It seems that I only need to have some slightly
> >>>>> modification on them to get it start working.
> >>>>> My plan is to create a m3.xlarge instance and directly plug it on our
> >>>>> existing jenkins as a slave so that I can start to make some
> progress.
> >>>>> I'll keep you posted. Suggestions are welcome as well. :)
> >>>>>
> >>>>> Thanks,
> >>>>> Evans
> >>>>>
> >>>>>
> >>>>> 2015-04-09 13:52 GMT+08:00 Konstantin Boudnik
> >>>>> <co...@apache.org>>:
> >>>>> Cycling back: today we got a credit coupon from Amazon EMR team so we
> >>> have
> >>>>> everything to start building new CI infra. Please contact me directly
> >>> if
> >>>>> you
> >>>>> plan to help with setting it up, so I can create an account with
> proper
> >>>>> permissions for you.
> >>>>>
> >>>>> Special thanks are going to Tom Zeng and his team for making it
> happen!
> >>>>> Thank you very much guys!
> >>>>>
> >>>>> Cos
> >>>>>
> >>>>>> On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
> >>>>>> Guys,
> >>>>>>
> >>>>>> I want to start a separate thread to track the CI preparations for
> >>> the
> >>>>>> release next month (fingers crossed). Clearly, we can make a release
> >>>>>> without CI, but it'd way easier to test and create binary artifacts
> >>> if
> >>>>>> we have a working environment for official validation. Roman has
> done
> >>>>>> a lot in this direction (many thanks!), but there are still a few
> >>>>>> rough edges, which might be easy to finish of.
> >>>>>>
> >>>>>> I want to figure out a couple of things:
> >>>>>> - what's the state of CI and how much still needs to be done (Rvs?
> >>> Could
> >>>>> you
> >>>>>> share any first hand feedback?)
> >>>>>> - who would be able to help with the CI completion? I can commit
> >>> some of
> >>>>> my
> >>>>>> cycles, but it'd be great to have few more hands on that. Clearly,
> >>>>> some
> >>>>>> Jenkins-foo and prior CI skills won't hurt ;)
> >>>>>>
> >>>>>> Please chime in if you can help. Thanks a lot!
> >>>>>> Cos
> >>>
>
Re: Brand new Bigtop AWS account
Posted by Jay Vyas <ja...@gmail.com>.
Agree with cos. Tests are utilities that we ship but not always perfect given the rapid pace of change
> On Jul 13, 2015, at 6:03 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> I don't think we ever had _all_ tests green considering how crappy Hive tests
> are, etc. I'd say we can punt on a few failing ones. I know I sound terrible
> ;(
>
> Cos
>
>> On Thu, Jul 09, 2015 at 10:49PM, Evans Ye wrote:
>> Sorry I didn't make it clear at the first place.
>> What I really want to ask is whether we need to have all the smoke
>> tests/integration tests to be tested and passed before we roll out 1.0.
>> For example, there might be 2 or 3 failed test in hadoop integration test.
>> But failed task doesn't mean that the component really has problems.
>> May I know what did we do in 0.8 release?
>>
>> 2015-07-01 2:43 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>>
>>>> On Mon, Jun 29, 2015 at 09:46PM, Evans Ye wrote:
>>>> I think Nate answered most of the questions, so thanks a lot!
>>>> To me the most critical task is to have 1.0 released, while CI overhaul
>>> is
>>>> part of the task.
>>>> From my point of view, there're two thinks holding us back from release.
>>>>
>>>> 1. The 1.0 branch is still at bleeding age to support all the claimed
>>> OSs.
>>>> The primary issues are in debian(should be fixed now) and oepnsuse
>>> working
>>>> as bigtop build slaves.
>>>
>>> Sorry guys - I was away for a while and still am catching up here. Evans,
>>> are
>>> you saying that there're issues that need to be fixed in the trunk and
>>> branch-1.0 as well? Or these are just CI related and do no require
>>> additional
>>> commits into the branch?
>>>
>>> The release criteria for us is pretty much what we define it to be ;) If we
>>> see 'good-enough' quality of the produced packages/tests/deployment
>>> mechanism
>>> - we can start the vote for the release any moment. Mind you - the packages
>>> aren't required for the release, as any ASF release is source code only.
>>> We do
>>> binaries as a extra service to our users.
>>>
>>> Cos
>>>
>>>> 2. Migrating the CI master to EC2 under EMR donated account. Since I
>>> can't
>>>> get in the old CI master, there's no way for me to retrieve current job
>>>> configs and restore them back to new CI master. Roman probably has
>>> backups
>>>> but he might be busying on other things.
>>>>
>>>> 3. I'll let cos add more since I'm not quite familiar with release
>>>> criteria, but I'd love to help.
>>>>
>>>> I guess we can't have 2 solved in short period, so I might start to work
>>> on
>>>> setting up repositories on old CI master directly.
>>>>
>>>> Thanks,
>>>> Evans
>>>>
>>>> 2015-06-27 3:46 GMT+08:00 <na...@reactor8.com>:
>>>>
>>>>> Still work in progress, will let Evans add some more details as he has
>>> been
>>>>> the one kickin but on CI tasks. Besides some more setup and planned
>>>>> enhancements focus is getting 1.0 out the door, then turning towards
>>> more
>>>>> foundational CI work for 1.x and beyond.
>>>>>
>>>>> Think we will be well served having more official bigtop repos
>>> serving, I
>>>>> setup a prototype off 0.8, serving from S3.., few things for us to
>>> figure
>>>>> out build/process wise to properly hook the CI into S3/repo publishing
>>>>>
>>>>> Evans created this jira for general tracking:
>>>>> https://issues.apache.org/jira/browse/BIGTOP-1897
>>>>>
>>>>> Not all of which are required for new CI to be ready, for example newly
>>>>> added one for optimizing build process to reuse artifacts can start
>>> working
>>>>> on in parallel to new CI and 1.0 release but get completed after both
>>> are
>>>>> taken care of
>>>>>
>>>>> -----Original Message-----
>>>>> From: Zeng, Tom [mailto:tomzeng@amazon.com]
>>>>> Sent: Thursday, June 25, 2015 3:36 PM
>>>>> To: Evans Ye; Konstantin Boudnik
>>>>> Cc: dev@bigtop.apache.org
>>>>> Subject: Re: Brand new Bigtop AWS account
>>>>>
>>>>> Hi Evans and Cos, what's the status of this, do we have the build CI
>>>>> environment set up yet?
>>>>>
>>>>> Thanks,
>>>>> Tom
>>>>>
>>>>> From: Evans Ye <ev...@apache.org>>
>>>>> Date: Sunday, April 19, 2015 at 11:12 AM
>>>>> To: "dev@bigtop.apache.org<ma...@bigtop.apache.org>"
>>>>> <de...@bigtop.apache.org>>, "Zeng, Tom"
>>>>> <to...@amazon.com>>
>>>>> Subject: Re: Brand new Bigtop AWS account
>>>>>
>>>>> Hi guys,
>>>>>
>>>>> I've copied security group settings from the old account to the new one
>>>>> provided by EMR team (thanks Tom!).
>>>>> I've also checked our existing jenkins and there are some useful jobs
>>>>> already defined. It seems that I only need to have some slightly
>>>>> modification on them to get it start working.
>>>>> My plan is to create a m3.xlarge instance and directly plug it on our
>>>>> existing jenkins as a slave so that I can start to make some progress.
>>>>> I'll keep you posted. Suggestions are welcome as well. :)
>>>>>
>>>>> Thanks,
>>>>> Evans
>>>>>
>>>>>
>>>>> 2015-04-09 13:52 GMT+08:00 Konstantin Boudnik
>>>>> <co...@apache.org>>:
>>>>> Cycling back: today we got a credit coupon from Amazon EMR team so we
>>> have
>>>>> everything to start building new CI infra. Please contact me directly
>>> if
>>>>> you
>>>>> plan to help with setting it up, so I can create an account with proper
>>>>> permissions for you.
>>>>>
>>>>> Special thanks are going to Tom Zeng and his team for making it happen!
>>>>> Thank you very much guys!
>>>>>
>>>>> Cos
>>>>>
>>>>>> On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
>>>>>> Guys,
>>>>>>
>>>>>> I want to start a separate thread to track the CI preparations for
>>> the
>>>>>> release next month (fingers crossed). Clearly, we can make a release
>>>>>> without CI, but it'd way easier to test and create binary artifacts
>>> if
>>>>>> we have a working environment for official validation. Roman has done
>>>>>> a lot in this direction (many thanks!), but there are still a few
>>>>>> rough edges, which might be easy to finish of.
>>>>>>
>>>>>> I want to figure out a couple of things:
>>>>>> - what's the state of CI and how much still needs to be done (Rvs?
>>> Could
>>>>> you
>>>>>> share any first hand feedback?)
>>>>>> - who would be able to help with the CI completion? I can commit
>>> some of
>>>>> my
>>>>>> cycles, but it'd be great to have few more hands on that. Clearly,
>>>>> some
>>>>>> Jenkins-foo and prior CI skills won't hurt ;)
>>>>>>
>>>>>> Please chime in if you can help. Thanks a lot!
>>>>>> Cos
>>>
Re: Brand new Bigtop AWS account
Posted by Konstantin Boudnik <co...@apache.org>.
I don't think we ever had _all_ tests green considering how crappy Hive tests
are, etc. I'd say we can punt on a few failing ones. I know I sound terrible
;(
Cos
On Thu, Jul 09, 2015 at 10:49PM, Evans Ye wrote:
> Sorry I didn't make it clear at the first place.
> What I really want to ask is whether we need to have all the smoke
> tests/integration tests to be tested and passed before we roll out 1.0.
> For example, there might be 2 or 3 failed test in hadoop integration test.
> But failed task doesn't mean that the component really has problems.
> May I know what did we do in 0.8 release?
>
> 2015-07-01 2:43 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>
> > On Mon, Jun 29, 2015 at 09:46PM, Evans Ye wrote:
> > > I think Nate answered most of the questions, so thanks a lot!
> > > To me the most critical task is to have 1.0 released, while CI overhaul
> > is
> > > part of the task.
> > > From my point of view, there're two thinks holding us back from release.
> > >
> > > 1. The 1.0 branch is still at bleeding age to support all the claimed
> > OSs.
> > > The primary issues are in debian(should be fixed now) and oepnsuse
> > working
> > > as bigtop build slaves.
> >
> > Sorry guys - I was away for a while and still am catching up here. Evans,
> > are
> > you saying that there're issues that need to be fixed in the trunk and
> > branch-1.0 as well? Or these are just CI related and do no require
> > additional
> > commits into the branch?
> >
> > The release criteria for us is pretty much what we define it to be ;) If we
> > see 'good-enough' quality of the produced packages/tests/deployment
> > mechanism
> > - we can start the vote for the release any moment. Mind you - the packages
> > aren't required for the release, as any ASF release is source code only.
> > We do
> > binaries as a extra service to our users.
> >
> > Cos
> >
> > > 2. Migrating the CI master to EC2 under EMR donated account. Since I
> > can't
> > > get in the old CI master, there's no way for me to retrieve current job
> > > configs and restore them back to new CI master. Roman probably has
> > backups
> > > but he might be busying on other things.
> > >
> > > 3. I'll let cos add more since I'm not quite familiar with release
> > > criteria, but I'd love to help.
> > >
> > > I guess we can't have 2 solved in short period, so I might start to work
> > on
> > > setting up repositories on old CI master directly.
> > >
> > > Thanks,
> > > Evans
> > >
> > > 2015-06-27 3:46 GMT+08:00 <na...@reactor8.com>:
> > >
> > > > Still work in progress, will let Evans add some more details as he has
> > been
> > > > the one kickin but on CI tasks. Besides some more setup and planned
> > > > enhancements focus is getting 1.0 out the door, then turning towards
> > more
> > > > foundational CI work for 1.x and beyond.
> > > >
> > > > Think we will be well served having more official bigtop repos
> > serving, I
> > > > setup a prototype off 0.8, serving from S3.., few things for us to
> > figure
> > > > out build/process wise to properly hook the CI into S3/repo publishing
> > > >
> > > > Evans created this jira for general tracking:
> > > > https://issues.apache.org/jira/browse/BIGTOP-1897
> > > >
> > > > Not all of which are required for new CI to be ready, for example newly
> > > > added one for optimizing build process to reuse artifacts can start
> > working
> > > > on in parallel to new CI and 1.0 release but get completed after both
> > are
> > > > taken care of
> > > >
> > > > -----Original Message-----
> > > > From: Zeng, Tom [mailto:tomzeng@amazon.com]
> > > > Sent: Thursday, June 25, 2015 3:36 PM
> > > > To: Evans Ye; Konstantin Boudnik
> > > > Cc: dev@bigtop.apache.org
> > > > Subject: Re: Brand new Bigtop AWS account
> > > >
> > > > Hi Evans and Cos, what's the status of this, do we have the build CI
> > > > environment set up yet?
> > > >
> > > > Thanks,
> > > > Tom
> > > >
> > > > From: Evans Ye <ev...@apache.org>>
> > > > Date: Sunday, April 19, 2015 at 11:12 AM
> > > > To: "dev@bigtop.apache.org<ma...@bigtop.apache.org>"
> > > > <de...@bigtop.apache.org>>, "Zeng, Tom"
> > > > <to...@amazon.com>>
> > > > Subject: Re: Brand new Bigtop AWS account
> > > >
> > > > Hi guys,
> > > >
> > > > I've copied security group settings from the old account to the new one
> > > > provided by EMR team (thanks Tom!).
> > > > I've also checked our existing jenkins and there are some useful jobs
> > > > already defined. It seems that I only need to have some slightly
> > > > modification on them to get it start working.
> > > > My plan is to create a m3.xlarge instance and directly plug it on our
> > > > existing jenkins as a slave so that I can start to make some progress.
> > > > I'll keep you posted. Suggestions are welcome as well. :)
> > > >
> > > > Thanks,
> > > > Evans
> > > >
> > > >
> > > > 2015-04-09 13:52 GMT+08:00 Konstantin Boudnik
> > > > <co...@apache.org>>:
> > > > Cycling back: today we got a credit coupon from Amazon EMR team so we
> > have
> > > > everything to start building new CI infra. Please contact me directly
> > if
> > > > you
> > > > plan to help with setting it up, so I can create an account with proper
> > > > permissions for you.
> > > >
> > > > Special thanks are going to Tom Zeng and his team for making it happen!
> > > > Thank you very much guys!
> > > >
> > > > Cos
> > > >
> > > > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
> > > > > Guys,
> > > > >
> > > > > I want to start a separate thread to track the CI preparations for
> > the
> > > > > release next month (fingers crossed). Clearly, we can make a release
> > > > > without CI, but it'd way easier to test and create binary artifacts
> > if
> > > > > we have a working environment for official validation. Roman has done
> > > > > a lot in this direction (many thanks!), but there are still a few
> > > > > rough edges, which might be easy to finish of.
> > > > >
> > > > > I want to figure out a couple of things:
> > > > > - what's the state of CI and how much still needs to be done (Rvs?
> > Could
> > > > you
> > > > > share any first hand feedback?)
> > > > > - who would be able to help with the CI completion? I can commit
> > some of
> > > > my
> > > > > cycles, but it'd be great to have few more hands on that. Clearly,
> > > > some
> > > > > Jenkins-foo and prior CI skills won't hurt ;)
> > > > >
> > > > > Please chime in if you can help. Thanks a lot!
> > > > > Cos
> > > > >
> > > >
> > > >
> > > >
> >