You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Olaf Flebbe <of...@oflebbe.de> on 2015/07/11 10:12:47 UTC
Re: Brand new Bigtop AWS account
Jay,
What's the JIRA Issue number?
Olaf
> Am 10.07.2015 um 23:30 schrieb Jay Vyas <ja...@gmail.com>:
>
> My thought is no. Hive tests were broken in 0.8 and we released hive. We continue evolving on head so it's ok. But in pretty lax about this stuff
>
> Sent from my iPhone
>
>> On Jul 9, 2015, at 10:49 AM, Evans Ye <ev...@apache.org> wrote:
>>
>> Sorry I didn't make it clear at the first place.
>> What I really want to ask is whether we need to have all the smoke
>> tests/integration tests to be tested and passed before we roll out 1.0.
>> For example, there might be 2 or 3 failed test in hadoop integration test.
>> But failed task doesn't mean that the component really has problems.
>> May I know what did we do in 0.8 release?
>>
>> 2015-07-01 2:43 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>>
>>>> On Mon, Jun 29, 2015 at 09:46PM, Evans Ye wrote:
>>>> I think Nate answered most of the questions, so thanks a lot!
>>>> To me the most critical task is to have 1.0 released, while CI overhaul
>>> is
>>>> part of the task.
>>>> From my point of view, there're two thinks holding us back from release.
>>>>
>>>> 1. The 1.0 branch is still at bleeding age to support all the claimed
>>> OSs.
>>>> The primary issues are in debian(should be fixed now) and oepnsuse
>>> working
>>>> as bigtop build slaves.
>>>
>>> Sorry guys - I was away for a while and still am catching up here. Evans,
>>> are
>>> you saying that there're issues that need to be fixed in the trunk and
>>> branch-1.0 as well? Or these are just CI related and do no require
>>> additional
>>> commits into the branch?
>>>
>>> The release criteria for us is pretty much what we define it to be ;) If we
>>> see 'good-enough' quality of the produced packages/tests/deployment
>>> mechanism
>>> - we can start the vote for the release any moment. Mind you - the packages
>>> aren't required for the release, as any ASF release is source code only.
>>> We do
>>> binaries as a extra service to our users.
>>>
>>> Cos
>>>
>>>> 2. Migrating the CI master to EC2 under EMR donated account. Since I
>>> can't
>>>> get in the old CI master, there's no way for me to retrieve current job
>>>> configs and restore them back to new CI master. Roman probably has
>>> backups
>>>> but he might be busying on other things.
>>>>
>>>> 3. I'll let cos add more since I'm not quite familiar with release
>>>> criteria, but I'd love to help.
>>>>
>>>> I guess we can't have 2 solved in short period, so I might start to work
>>> on
>>>> setting up repositories on old CI master directly.
>>>>
>>>> Thanks,
>>>> Evans
>>>>
>>>> 2015-06-27 3:46 GMT+08:00 <na...@reactor8.com>:
>>>>
>>>>> Still work in progress, will let Evans add some more details as he has
>>> been
>>>>> the one kickin but on CI tasks. Besides some more setup and planned
>>>>> enhancements focus is getting 1.0 out the door, then turning towards
>>> more
>>>>> foundational CI work for 1.x and beyond.
>>>>>
>>>>> Think we will be well served having more official bigtop repos
>>> serving, I
>>>>> setup a prototype off 0.8, serving from S3.., few things for us to
>>> figure
>>>>> out build/process wise to properly hook the CI into S3/repo publishing
>>>>>
>>>>> Evans created this jira for general tracking:
>>>>> https://issues.apache.org/jira/browse/BIGTOP-1897
>>>>>
>>>>> Not all of which are required for new CI to be ready, for example newly
>>>>> added one for optimizing build process to reuse artifacts can start
>>> working
>>>>> on in parallel to new CI and 1.0 release but get completed after both
>>> are
>>>>> taken care of
>>>>>
>>>>> -----Original Message-----
>>>>> From: Zeng, Tom [mailto:tomzeng@amazon.com]
>>>>> Sent: Thursday, June 25, 2015 3:36 PM
>>>>> To: Evans Ye; Konstantin Boudnik
>>>>> Cc: dev@bigtop.apache.org
>>>>> Subject: Re: Brand new Bigtop AWS account
>>>>>
>>>>> Hi Evans and Cos, what's the status of this, do we have the build CI
>>>>> environment set up yet?
>>>>>
>>>>> Thanks,
>>>>> Tom
>>>>>
>>>>> From: Evans Ye <ev...@apache.org>>
>>>>> Date: Sunday, April 19, 2015 at 11:12 AM
>>>>> To: "dev@bigtop.apache.org<ma...@bigtop.apache.org>"
>>>>> <de...@bigtop.apache.org>>, "Zeng, Tom"
>>>>> <to...@amazon.com>>
>>>>> Subject: Re: Brand new Bigtop AWS account
>>>>>
>>>>> Hi guys,
>>>>>
>>>>> I've copied security group settings from the old account to the new one
>>>>> provided by EMR team (thanks Tom!).
>>>>> I've also checked our existing jenkins and there are some useful jobs
>>>>> already defined. It seems that I only need to have some slightly
>>>>> modification on them to get it start working.
>>>>> My plan is to create a m3.xlarge instance and directly plug it on our
>>>>> existing jenkins as a slave so that I can start to make some progress.
>>>>> I'll keep you posted. Suggestions are welcome as well. :)
>>>>>
>>>>> Thanks,
>>>>> Evans
>>>>>
>>>>>
>>>>> 2015-04-09 13:52 GMT+08:00 Konstantin Boudnik
>>>>> <co...@apache.org>>:
>>>>> Cycling back: today we got a credit coupon from Amazon EMR team so we
>>> have
>>>>> everything to start building new CI infra. Please contact me directly
>>> if
>>>>> you
>>>>> plan to help with setting it up, so I can create an account with proper
>>>>> permissions for you.
>>>>>
>>>>> Special thanks are going to Tom Zeng and his team for making it happen!
>>>>> Thank you very much guys!
>>>>>
>>>>> Cos
>>>>>
>>>>>> On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
>>>>>> Guys,
>>>>>>
>>>>>> I want to start a separate thread to track the CI preparations for
>>> the
>>>>>> release next month (fingers crossed). Clearly, we can make a release
>>>>>> without CI, but it'd way easier to test and create binary artifacts
>>> if
>>>>>> we have a working environment for official validation. Roman has done
>>>>>> a lot in this direction (many thanks!), but there are still a few
>>>>>> rough edges, which might be easy to finish of.
>>>>>>
>>>>>> I want to figure out a couple of things:
>>>>>> - what's the state of CI and how much still needs to be done (Rvs?
>>> Could
>>>>> you
>>>>>> share any first hand feedback?)
>>>>>> - who would be able to help with the CI completion? I can commit
>>> some of
>>>>> my
>>>>>> cycles, but it'd be great to have few more hands on that. Clearly,
>>>>> some
>>>>>> Jenkins-foo and prior CI skills won't hurt ;)
>>>>>>
>>>>>> Please chime in if you can help. Thanks a lot!
>>>>>> Cos
>>>
Re: Brand new Bigtop AWS account
Posted by Jay Vyas <ja...@gmail.com>.
I was just answering the question of "do all tests need to pass for release"... Imo not really :).
The issue number for hive tests? That was an issue in the past https://issues.apache.org/jira/plugins/servlet/mobile#issue/BIGTOP-1450 fixed it .
> On Jul 11, 2015, at 4:12 AM, Olaf Flebbe <of...@oflebbe.de> wrote:
>
> Jay,
>
> What's the JIRA Issue number?
>
> Olaf
>
>> Am 10.07.2015 um 23:30 schrieb Jay Vyas <ja...@gmail.com>:
>>
>> My thought is no. Hive tests were broken in 0.8 and we released hive. We continue evolving on head so it's ok. But in pretty lax about this stuff
>>
>> Sent from my iPhone
>>
>>> On Jul 9, 2015, at 10:49 AM, Evans Ye <ev...@apache.org> wrote:
>>>
>>> Sorry I didn't make it clear at the first place.
>>> What I really want to ask is whether we need to have all the smoke
>>> tests/integration tests to be tested and passed before we roll out 1.0.
>>> For example, there might be 2 or 3 failed test in hadoop integration test.
>>> But failed task doesn't mean that the component really has problems.
>>> May I know what did we do in 0.8 release?
>>>
>>> 2015-07-01 2:43 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>>>
>>>>> On Mon, Jun 29, 2015 at 09:46PM, Evans Ye wrote:
>>>>> I think Nate answered most of the questions, so thanks a lot!
>>>>> To me the most critical task is to have 1.0 released, while CI overhaul
>>>> is
>>>>> part of the task.
>>>>> From my point of view, there're two thinks holding us back from release.
>>>>>
>>>>> 1. The 1.0 branch is still at bleeding age to support all the claimed
>>>> OSs.
>>>>> The primary issues are in debian(should be fixed now) and oepnsuse
>>>> working
>>>>> as bigtop build slaves.
>>>>
>>>> Sorry guys - I was away for a while and still am catching up here. Evans,
>>>> are
>>>> you saying that there're issues that need to be fixed in the trunk and
>>>> branch-1.0 as well? Or these are just CI related and do no require
>>>> additional
>>>> commits into the branch?
>>>>
>>>> The release criteria for us is pretty much what we define it to be ;) If we
>>>> see 'good-enough' quality of the produced packages/tests/deployment
>>>> mechanism
>>>> - we can start the vote for the release any moment. Mind you - the packages
>>>> aren't required for the release, as any ASF release is source code only.
>>>> We do
>>>> binaries as a extra service to our users.
>>>>
>>>> Cos
>>>>
>>>>> 2. Migrating the CI master to EC2 under EMR donated account. Since I
>>>> can't
>>>>> get in the old CI master, there's no way for me to retrieve current job
>>>>> configs and restore them back to new CI master. Roman probably has
>>>> backups
>>>>> but he might be busying on other things.
>>>>>
>>>>> 3. I'll let cos add more since I'm not quite familiar with release
>>>>> criteria, but I'd love to help.
>>>>>
>>>>> I guess we can't have 2 solved in short period, so I might start to work
>>>> on
>>>>> setting up repositories on old CI master directly.
>>>>>
>>>>> Thanks,
>>>>> Evans
>>>>>
>>>>> 2015-06-27 3:46 GMT+08:00 <na...@reactor8.com>:
>>>>>
>>>>>> Still work in progress, will let Evans add some more details as he has
>>>> been
>>>>>> the one kickin but on CI tasks. Besides some more setup and planned
>>>>>> enhancements focus is getting 1.0 out the door, then turning towards
>>>> more
>>>>>> foundational CI work for 1.x and beyond.
>>>>>>
>>>>>> Think we will be well served having more official bigtop repos
>>>> serving, I
>>>>>> setup a prototype off 0.8, serving from S3.., few things for us to
>>>> figure
>>>>>> out build/process wise to properly hook the CI into S3/repo publishing
>>>>>>
>>>>>> Evans created this jira for general tracking:
>>>>>> https://issues.apache.org/jira/browse/BIGTOP-1897
>>>>>>
>>>>>> Not all of which are required for new CI to be ready, for example newly
>>>>>> added one for optimizing build process to reuse artifacts can start
>>>> working
>>>>>> on in parallel to new CI and 1.0 release but get completed after both
>>>> are
>>>>>> taken care of
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Zeng, Tom [mailto:tomzeng@amazon.com]
>>>>>> Sent: Thursday, June 25, 2015 3:36 PM
>>>>>> To: Evans Ye; Konstantin Boudnik
>>>>>> Cc: dev@bigtop.apache.org
>>>>>> Subject: Re: Brand new Bigtop AWS account
>>>>>>
>>>>>> Hi Evans and Cos, what's the status of this, do we have the build CI
>>>>>> environment set up yet?
>>>>>>
>>>>>> Thanks,
>>>>>> Tom
>>>>>>
>>>>>> From: Evans Ye <ev...@apache.org>>
>>>>>> Date: Sunday, April 19, 2015 at 11:12 AM
>>>>>> To: "dev@bigtop.apache.org<ma...@bigtop.apache.org>"
>>>>>> <de...@bigtop.apache.org>>, "Zeng, Tom"
>>>>>> <to...@amazon.com>>
>>>>>> Subject: Re: Brand new Bigtop AWS account
>>>>>>
>>>>>> Hi guys,
>>>>>>
>>>>>> I've copied security group settings from the old account to the new one
>>>>>> provided by EMR team (thanks Tom!).
>>>>>> I've also checked our existing jenkins and there are some useful jobs
>>>>>> already defined. It seems that I only need to have some slightly
>>>>>> modification on them to get it start working.
>>>>>> My plan is to create a m3.xlarge instance and directly plug it on our
>>>>>> existing jenkins as a slave so that I can start to make some progress.
>>>>>> I'll keep you posted. Suggestions are welcome as well. :)
>>>>>>
>>>>>> Thanks,
>>>>>> Evans
>>>>>>
>>>>>>
>>>>>> 2015-04-09 13:52 GMT+08:00 Konstantin Boudnik
>>>>>> <co...@apache.org>>:
>>>>>> Cycling back: today we got a credit coupon from Amazon EMR team so we
>>>> have
>>>>>> everything to start building new CI infra. Please contact me directly
>>>> if
>>>>>> you
>>>>>> plan to help with setting it up, so I can create an account with proper
>>>>>> permissions for you.
>>>>>>
>>>>>> Special thanks are going to Tom Zeng and his team for making it happen!
>>>>>> Thank you very much guys!
>>>>>>
>>>>>> Cos
>>>>>>
>>>>>>> On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
>>>>>>> Guys,
>>>>>>>
>>>>>>> I want to start a separate thread to track the CI preparations for
>>>> the
>>>>>>> release next month (fingers crossed). Clearly, we can make a release
>>>>>>> without CI, but it'd way easier to test and create binary artifacts
>>>> if
>>>>>>> we have a working environment for official validation. Roman has done
>>>>>>> a lot in this direction (many thanks!), but there are still a few
>>>>>>> rough edges, which might be easy to finish of.
>>>>>>>
>>>>>>> I want to figure out a couple of things:
>>>>>>> - what's the state of CI and how much still needs to be done (Rvs?
>>>> Could
>>>>>> you
>>>>>>> share any first hand feedback?)
>>>>>>> - who would be able to help with the CI completion? I can commit
>>>> some of
>>>>>> my
>>>>>>> cycles, but it'd be great to have few more hands on that. Clearly,
>>>>>> some
>>>>>>> Jenkins-foo and prior CI skills won't hurt ;)
>>>>>>>
>>>>>>> Please chime in if you can help. Thanks a lot!
>>>>>>> Cos
>