You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Evans Ye <ev...@apache.org> on 2017/06/05 04:32:14 UTC
Bigtop Puppet current status
Hi Bigtopers,
I've shared this on my Apache Big Data talk, but haven't got a chance to
share with you yet.
This is a Jenkins matrix job to show the healthy status of our component
deployment code.
Each run deploys hdfs + Y on X, where X is the OS and Y is the component
shown on the matrix:
https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-trunk-deployments/45/
Generally speaking, the status of our deployment code needs a lot of effort
to polish.
So, for 1.2.1 release, I think we should make some changes:
1). Remove unmaintained failing components
2). Get maintained components fixed
1) might be easier to achieve, while 2) takes time.
In order to maintain a good code quality, I suggest proceed with 1).
Any different thoughts?
Re: Bigtop Puppet current status
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi Evans,
thanks for your post and contribution!
Digging into some of the problems is showing me that the problem is not only related to puppet rather to the packages itself.
It is not absolutly clear to me what you meant by "removing" component: The puppet code related to that part of bigtop (for instance "hue" or remove both packaging and puppet deployment code ?
I am ok to remove packages (both package and puppet) with no maintainer attached.
We may need to have maintainers for platforms as well.
To be more specific:
For instance the breakage of zeppelin is due to a IMHO broken architecture of zeppelin itself. It runs maven and tries to download stuff from the internet a service start time. Sorry, I can only to propose to remove zeppelin altogether, since we cannot fix it.
It looks like hue-packaging has gone havoc. Since I seem to be the only one who was interested in this in the past I am ok to remove "hue" altogether. If someone still likes to have it, I can look into this issue, it may not be that complicated to fix.
Puppet Problems:
kerberos on centos-7. I may look into this if I find time.
zookeeper at ubuntu, but it seems you (Evans) are already working on it.
Best,
Olaf
> Am 05.06.2017 um 06:32 schrieb Evans Ye <ev...@apache.org>:
>
> Hi Bigtopers,
>
> I've shared this on my Apache Big Data talk, but haven't got a chance to
> share with you yet.
> This is a Jenkins matrix job to show the healthy status of our component
> deployment code.
> Each run deploys hdfs + Y on X, where X is the OS and Y is the component
> shown on the matrix:
>
> https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-trunk-deployments/45/
>
> Generally speaking, the status of our deployment code needs a lot of effort
> to polish.
> So, for 1.2.1 release, I think we should make some changes:
>
> 1). Remove unmaintained failing components
> 2). Get maintained components fixed
>
> 1) might be easier to achieve, while 2) takes time.
> In order to maintain a good code quality, I suggest proceed with 1).
>
> Any different thoughts?
Re: Bigtop Puppet current status
Posted by Konstantin Boudnik <co...@apache.org>.
I have looked into the provisioner job and I think I found the root
cause of why we aren't testing the latest builds. Please see my
comment on BIGTOP-2295 and please reply there with your comments!
Thanks!
--
Take care,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622
Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.
On Mon, Jun 12, 2017 at 12:38 PM, Konstantin Boudnik <co...@apache.org> wrote:
> Thanks Olaf for reminding about this. I think it is important for us to get
> fixed ASAP. I left some comments there to hopefully restart the discussion.
>
> Cos
>
> On Mon, Jun 12, 2017 at 09:12PM, Olaf Flebbe wrote:
>> 1. Hi Cos,
>> 1. BIGTOP-2295
>> Olaf
>
>
Re: Bigtop Puppet current status
Posted by Konstantin Boudnik <co...@apache.org>.
Thanks Olaf for reminding about this. I think it is important for us to get
fixed ASAP. I left some comments there to hopefully restart the discussion.
Cos
On Mon, Jun 12, 2017 at 09:12PM, Olaf Flebbe wrote:
> 1. Hi Cos,
> 1. BIGTOP-2295
> Olaf
Re: Bigtop Puppet current status
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi Cos,
BIGTOP-2295 <https://issues.apache.org/jira/browse/BIGTOP-2295>
Olaf
Re: Bigtop Puppet current status
Posted by Konstantin Boudnik <co...@apache.org>.
On Mon, Jun 12, 2017 at 08:47PM, Olaf Flebbe wrote:
> Hi Cos,
>
> The deployment CI job of bigtop-trunk use the repo and packages of the last
> bigtop-release. So any change in package (i.e. fixes) will not be reflected
> in the provisioner jobs. I am really tired in arguing against this.
Ah, I am not crazy then! Thanks for confirming my sanity, Olaf!
So, the next question is how we can combine the night package build with
Evans' deployment job. Yes, I know - the provisioning matrix server a
different purpose to polish our Puppet code. But, and that's a big but -
while polishing Puppet we might find out that it is a package that is
problematic (BIGTOP-2165 is a case in point). In wish case we need to test the
latest packages.
I wonder if we can collect all packages together and build a repo on a nightly
basis? Or perhaps we can directly use Jenkins' artifacts? I did the former for
another project a while back and it is a-ok if you have just a handful of
packages. In our case we are talking about Gigabytes of stuff being copied
between the slaves, and I don't think it would be tht efficient. May be we can
use a shared location to store the binary artifacts?
Thoughts?
Cos
> > Am 12.06.2017 um 20:40 schrieb Konstantin Boudnik <co...@apache.org>:
> >
> > So, I got BIGTOP-2165 in and it has addressed the ignite service startup issue
> > in my local testing, but I still see failing build in our CI [1]
> >
> > Evans, looks like I need your help to sort it our. Here's how I deploy the
> > cluster:
> > - first I provision hdfs/yarn/mapreduce components from our 1.2 repo
> > - then I change the repo URL to my local one, where I have a freshly built
> > ignite packages, and run 'puppet apply' manually
> > - the service comes up just fine and I see the process running. I also can
> > stop it using 'service ignite-hadoop stop'
> >
> > I was looking into the deployment matrix's config, but had a bit of a hard
> > time deciphering where the packages are coming from. Is there a chance of a
> > race-condition where the deployment might be using the packages from the last
> > build or something or a sort? I guess the easiest answer to this would be to
> > wait until the next provisioner cycle and see if the situation with CentOS'
> > Ignite deployment has improved.
> >
> > This bug was driving me nuts for the past two years, and now that I thought I
> > have fixed it there are two different outcomes in different environments. So
> > please accept my apologies if I am barking at the wrong tree ;)
> >
> > Thanks for your help!
> >
> > [1] https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-trunk-deployments/BUILD_SLAVE=docker-slave-02,COMPONENTS=ignite_hadoop,OS=centos6/54/console
> >
> > Cos
> >
> > On Thu, Jun 08, 2017 at 10:03AM, Konstantin Boudnik wrote:
> >> Cool! I am trying to get some Ignite's dev@ help with BIGTOP-2108 -
> >> hopefully, it won't be long before it is fixed. Thanks!
> >> --
> >> Take care,
> >> Konstantin (Cos) Boudnik
> >> 2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622
> >>
> >> Disclaimer: Opinions expressed in this email are those of the author,
> >> and do not necessarily represent the views of any company the author
> >> might be affiliated with at the moment of writing.
> >>
> >>
> >> On Thu, Jun 8, 2017 at 8:55 AM, Evans Ye <ev...@apache.org> wrote:
> >>> OK. I agreed with you.
> >>> Considered that most of us are making contributions in free time.
> >>> Let's make the release less stressed but more solid.
> >>>
> >>> I plan to make 1.2.1 release at the end of the June.
> >>> That should make plenty of time for blockers to be removed.
> >>>
> >>> Finally, I'd like to thank you all for taking actions to our deployment
> >>> issues.
> >>>
> >>>
> >>> 2017-06-08 1:11 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> >>>
> >>>> On Thu, Jun 08, 2017 at 12:57AM, Evans Ye wrote:
> >>>>> Cos,
> >>>>> Thanks a lot!
> >>>>>
> >>>>> Olaf,
> >>>>> My original thought was to remove those puppet code which no one has
> >>>>> interest to fix it.
> >>>>> But you're right. Some of them are broken at package level...
> >>>>>
> >>>>> I've dig a bit in and found some of them are failing due to Java not
> >>>>> presented before daemons starting up.
> >>>>> I've uploaded a patch in BIGTOP-2775, which should fix the problem.
> >>>>> But we still got many others failing for various reasons.
> >>>>>
> >>>>> Roman,
> >>>>> Good idea. We can track them via JIRAs.
> >>>>>
> >>>>> All,
> >>>>> I still hope that 1.2.1 can be released before DataWorks Summit.
> >>>>> However, at this point I'd like to get everybody's thoughts that whether
> >>>> we
> >>>>> should release it and mark these stuffs as known issues, or we get them
> >>>>> fixed and then release a fully functional release.
> >>>>> Which one you're in favor?
> >>>>
> >>>> Eitherway is good for me. Making the release before the mid of next week is
> >>>> only possible if we have everything in place right now and only need to do
> >>>> the
> >>>> release itself and vote on it. Otherwise, the timeframe is too tight...
> >>>>
> >>>> Cos
> >>>>
> >>>>> 2017-06-07 6:15 GMT+08:00 Roman Shaposhnik <ro...@shaposhnik.org>:
> >>>>>
> >>>>>> On Sun, Jun 4, 2017 at 9:32 PM, Evans Ye <ev...@apache.org> wrote:
> >>>>>>> Hi Bigtopers,
> >>>>>>>
> >>>>>>> I've shared this on my Apache Big Data talk, but haven't got a
> >>>> chance to
> >>>>>>> share with you yet.
> >>>>>>> This is a Jenkins matrix job to show the healthy status of our
> >>>> component
> >>>>>>> deployment code.
> >>>>>>> Each run deploys hdfs + Y on X, where X is the OS and Y is the
> >>>> component
> >>>>>>> shown on the matrix:
> >>>>>>>
> >>>>>>> https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-
> >>>>>> trunk-deployments/45/
> >>>>>>
> >>>>>> This is super awesome! Thanks for spending time on putting it together!
> >>>>>>
> >>>>>>> Generally speaking, the status of our deployment code needs a lot of
> >>>>>> effort
> >>>>>>> to polish.
> >>>>>>> So, for 1.2.1 release, I think we should make some changes:
> >>>>>>>
> >>>>>>> 1). Remove unmaintained failing components
> >>>>>>> 2). Get maintained components fixed
> >>>>>>>
> >>>>>>> 1) might be easier to achieve, while 2) takes time.
> >>>>>>> In order to maintain a good code quality, I suggest proceed with 1).
> >>>>>>>
> >>>>>>> Any different thoughts?
> >>>>>>
> >>>>>> Shall we at least start with filing JIRAs for everything that's
> >>>>>> failing -- we can discuss
> >>>>>> the particulars on the JIRAs then.
> >>>>>>
> >>>>>> I can try filing some tomorrow if nobody beats me to it (please do ;-))
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Roman.
> >>>>>>
> >>>>
>
Re: Bigtop Puppet current status
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi Cos,
The deployment CI job of bigtop-trunk use the repo and packages of the last bigtop-release. So any change in package (i.e. fixes) will not be reflected in the provisioner jobs. I am really tired in arguing against this.
Olaf
> Am 12.06.2017 um 20:40 schrieb Konstantin Boudnik <co...@apache.org>:
>
> So, I got BIGTOP-2165 in and it has addressed the ignite service startup issue
> in my local testing, but I still see failing build in our CI [1]
>
> Evans, looks like I need your help to sort it our. Here's how I deploy the
> cluster:
> - first I provision hdfs/yarn/mapreduce components from our 1.2 repo
> - then I change the repo URL to my local one, where I have a freshly built
> ignite packages, and run 'puppet apply' manually
> - the service comes up just fine and I see the process running. I also can
> stop it using 'service ignite-hadoop stop'
>
> I was looking into the deployment matrix's config, but had a bit of a hard
> time deciphering where the packages are coming from. Is there a chance of a
> race-condition where the deployment might be using the packages from the last
> build or something or a sort? I guess the easiest answer to this would be to
> wait until the next provisioner cycle and see if the situation with CentOS'
> Ignite deployment has improved.
>
> This bug was driving me nuts for the past two years, and now that I thought I
> have fixed it there are two different outcomes in different environments. So
> please accept my apologies if I am barking at the wrong tree ;)
>
> Thanks for your help!
>
> [1] https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-trunk-deployments/BUILD_SLAVE=docker-slave-02,COMPONENTS=ignite_hadoop,OS=centos6/54/console
>
> Cos
>
> On Thu, Jun 08, 2017 at 10:03AM, Konstantin Boudnik wrote:
>> Cool! I am trying to get some Ignite's dev@ help with BIGTOP-2108 -
>> hopefully, it won't be long before it is fixed. Thanks!
>> --
>> Take care,
>> Konstantin (Cos) Boudnik
>> 2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622
>>
>> Disclaimer: Opinions expressed in this email are those of the author,
>> and do not necessarily represent the views of any company the author
>> might be affiliated with at the moment of writing.
>>
>>
>> On Thu, Jun 8, 2017 at 8:55 AM, Evans Ye <ev...@apache.org> wrote:
>>> OK. I agreed with you.
>>> Considered that most of us are making contributions in free time.
>>> Let's make the release less stressed but more solid.
>>>
>>> I plan to make 1.2.1 release at the end of the June.
>>> That should make plenty of time for blockers to be removed.
>>>
>>> Finally, I'd like to thank you all for taking actions to our deployment
>>> issues.
>>>
>>>
>>> 2017-06-08 1:11 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>>>
>>>> On Thu, Jun 08, 2017 at 12:57AM, Evans Ye wrote:
>>>>> Cos,
>>>>> Thanks a lot!
>>>>>
>>>>> Olaf,
>>>>> My original thought was to remove those puppet code which no one has
>>>>> interest to fix it.
>>>>> But you're right. Some of them are broken at package level...
>>>>>
>>>>> I've dig a bit in and found some of them are failing due to Java not
>>>>> presented before daemons starting up.
>>>>> I've uploaded a patch in BIGTOP-2775, which should fix the problem.
>>>>> But we still got many others failing for various reasons.
>>>>>
>>>>> Roman,
>>>>> Good idea. We can track them via JIRAs.
>>>>>
>>>>> All,
>>>>> I still hope that 1.2.1 can be released before DataWorks Summit.
>>>>> However, at this point I'd like to get everybody's thoughts that whether
>>>> we
>>>>> should release it and mark these stuffs as known issues, or we get them
>>>>> fixed and then release a fully functional release.
>>>>> Which one you're in favor?
>>>>
>>>> Eitherway is good for me. Making the release before the mid of next week is
>>>> only possible if we have everything in place right now and only need to do
>>>> the
>>>> release itself and vote on it. Otherwise, the timeframe is too tight...
>>>>
>>>> Cos
>>>>
>>>>> 2017-06-07 6:15 GMT+08:00 Roman Shaposhnik <ro...@shaposhnik.org>:
>>>>>
>>>>>> On Sun, Jun 4, 2017 at 9:32 PM, Evans Ye <ev...@apache.org> wrote:
>>>>>>> Hi Bigtopers,
>>>>>>>
>>>>>>> I've shared this on my Apache Big Data talk, but haven't got a
>>>> chance to
>>>>>>> share with you yet.
>>>>>>> This is a Jenkins matrix job to show the healthy status of our
>>>> component
>>>>>>> deployment code.
>>>>>>> Each run deploys hdfs + Y on X, where X is the OS and Y is the
>>>> component
>>>>>>> shown on the matrix:
>>>>>>>
>>>>>>> https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-
>>>>>> trunk-deployments/45/
>>>>>>
>>>>>> This is super awesome! Thanks for spending time on putting it together!
>>>>>>
>>>>>>> Generally speaking, the status of our deployment code needs a lot of
>>>>>> effort
>>>>>>> to polish.
>>>>>>> So, for 1.2.1 release, I think we should make some changes:
>>>>>>>
>>>>>>> 1). Remove unmaintained failing components
>>>>>>> 2). Get maintained components fixed
>>>>>>>
>>>>>>> 1) might be easier to achieve, while 2) takes time.
>>>>>>> In order to maintain a good code quality, I suggest proceed with 1).
>>>>>>>
>>>>>>> Any different thoughts?
>>>>>>
>>>>>> Shall we at least start with filing JIRAs for everything that's
>>>>>> failing -- we can discuss
>>>>>> the particulars on the JIRAs then.
>>>>>>
>>>>>> I can try filing some tomorrow if nobody beats me to it (please do ;-))
>>>>>>
>>>>>> Thanks,
>>>>>> Roman.
>>>>>>
>>>>
Re: Bigtop Puppet current status
Posted by Konstantin Boudnik <co...@apache.org>.
So, I got BIGTOP-2165 in and it has addressed the ignite service startup issue
in my local testing, but I still see failing build in our CI [1]
Evans, looks like I need your help to sort it our. Here's how I deploy the
cluster:
- first I provision hdfs/yarn/mapreduce components from our 1.2 repo
- then I change the repo URL to my local one, where I have a freshly built
ignite packages, and run 'puppet apply' manually
- the service comes up just fine and I see the process running. I also can
stop it using 'service ignite-hadoop stop'
I was looking into the deployment matrix's config, but had a bit of a hard
time deciphering where the packages are coming from. Is there a chance of a
race-condition where the deployment might be using the packages from the last
build or something or a sort? I guess the easiest answer to this would be to
wait until the next provisioner cycle and see if the situation with CentOS'
Ignite deployment has improved.
This bug was driving me nuts for the past two years, and now that I thought I
have fixed it there are two different outcomes in different environments. So
please accept my apologies if I am barking at the wrong tree ;)
Thanks for your help!
[1] https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-trunk-deployments/BUILD_SLAVE=docker-slave-02,COMPONENTS=ignite_hadoop,OS=centos6/54/console
Cos
On Thu, Jun 08, 2017 at 10:03AM, Konstantin Boudnik wrote:
> Cool! I am trying to get some Ignite's dev@ help with BIGTOP-2108 -
> hopefully, it won't be long before it is fixed. Thanks!
> --
> Take care,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622
>
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
>
>
> On Thu, Jun 8, 2017 at 8:55 AM, Evans Ye <ev...@apache.org> wrote:
> > OK. I agreed with you.
> > Considered that most of us are making contributions in free time.
> > Let's make the release less stressed but more solid.
> >
> > I plan to make 1.2.1 release at the end of the June.
> > That should make plenty of time for blockers to be removed.
> >
> > Finally, I'd like to thank you all for taking actions to our deployment
> > issues.
> >
> >
> > 2017-06-08 1:11 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> >
> >> On Thu, Jun 08, 2017 at 12:57AM, Evans Ye wrote:
> >> > Cos,
> >> > Thanks a lot!
> >> >
> >> > Olaf,
> >> > My original thought was to remove those puppet code which no one has
> >> > interest to fix it.
> >> > But you're right. Some of them are broken at package level...
> >> >
> >> > I've dig a bit in and found some of them are failing due to Java not
> >> > presented before daemons starting up.
> >> > I've uploaded a patch in BIGTOP-2775, which should fix the problem.
> >> > But we still got many others failing for various reasons.
> >> >
> >> > Roman,
> >> > Good idea. We can track them via JIRAs.
> >> >
> >> > All,
> >> > I still hope that 1.2.1 can be released before DataWorks Summit.
> >> > However, at this point I'd like to get everybody's thoughts that whether
> >> we
> >> > should release it and mark these stuffs as known issues, or we get them
> >> > fixed and then release a fully functional release.
> >> > Which one you're in favor?
> >>
> >> Eitherway is good for me. Making the release before the mid of next week is
> >> only possible if we have everything in place right now and only need to do
> >> the
> >> release itself and vote on it. Otherwise, the timeframe is too tight...
> >>
> >> Cos
> >>
> >> > 2017-06-07 6:15 GMT+08:00 Roman Shaposhnik <ro...@shaposhnik.org>:
> >> >
> >> > > On Sun, Jun 4, 2017 at 9:32 PM, Evans Ye <ev...@apache.org> wrote:
> >> > > > Hi Bigtopers,
> >> > > >
> >> > > > I've shared this on my Apache Big Data talk, but haven't got a
> >> chance to
> >> > > > share with you yet.
> >> > > > This is a Jenkins matrix job to show the healthy status of our
> >> component
> >> > > > deployment code.
> >> > > > Each run deploys hdfs + Y on X, where X is the OS and Y is the
> >> component
> >> > > > shown on the matrix:
> >> > > >
> >> > > > https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-
> >> > > trunk-deployments/45/
> >> > >
> >> > > This is super awesome! Thanks for spending time on putting it together!
> >> > >
> >> > > > Generally speaking, the status of our deployment code needs a lot of
> >> > > effort
> >> > > > to polish.
> >> > > > So, for 1.2.1 release, I think we should make some changes:
> >> > > >
> >> > > > 1). Remove unmaintained failing components
> >> > > > 2). Get maintained components fixed
> >> > > >
> >> > > > 1) might be easier to achieve, while 2) takes time.
> >> > > > In order to maintain a good code quality, I suggest proceed with 1).
> >> > > >
> >> > > > Any different thoughts?
> >> > >
> >> > > Shall we at least start with filing JIRAs for everything that's
> >> > > failing -- we can discuss
> >> > > the particulars on the JIRAs then.
> >> > >
> >> > > I can try filing some tomorrow if nobody beats me to it (please do ;-))
> >> > >
> >> > > Thanks,
> >> > > Roman.
> >> > >
> >>
Re: Bigtop Puppet current status
Posted by Konstantin Boudnik <co...@apache.org>.
Cool! I am trying to get some Ignite's dev@ help with BIGTOP-2108 -
hopefully, it won't be long before it is fixed. Thanks!
--
Take care,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622
Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.
On Thu, Jun 8, 2017 at 8:55 AM, Evans Ye <ev...@apache.org> wrote:
> OK. I agreed with you.
> Considered that most of us are making contributions in free time.
> Let's make the release less stressed but more solid.
>
> I plan to make 1.2.1 release at the end of the June.
> That should make plenty of time for blockers to be removed.
>
> Finally, I'd like to thank you all for taking actions to our deployment
> issues.
>
>
> 2017-06-08 1:11 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>
>> On Thu, Jun 08, 2017 at 12:57AM, Evans Ye wrote:
>> > Cos,
>> > Thanks a lot!
>> >
>> > Olaf,
>> > My original thought was to remove those puppet code which no one has
>> > interest to fix it.
>> > But you're right. Some of them are broken at package level...
>> >
>> > I've dig a bit in and found some of them are failing due to Java not
>> > presented before daemons starting up.
>> > I've uploaded a patch in BIGTOP-2775, which should fix the problem.
>> > But we still got many others failing for various reasons.
>> >
>> > Roman,
>> > Good idea. We can track them via JIRAs.
>> >
>> > All,
>> > I still hope that 1.2.1 can be released before DataWorks Summit.
>> > However, at this point I'd like to get everybody's thoughts that whether
>> we
>> > should release it and mark these stuffs as known issues, or we get them
>> > fixed and then release a fully functional release.
>> > Which one you're in favor?
>>
>> Eitherway is good for me. Making the release before the mid of next week is
>> only possible if we have everything in place right now and only need to do
>> the
>> release itself and vote on it. Otherwise, the timeframe is too tight...
>>
>> Cos
>>
>> > 2017-06-07 6:15 GMT+08:00 Roman Shaposhnik <ro...@shaposhnik.org>:
>> >
>> > > On Sun, Jun 4, 2017 at 9:32 PM, Evans Ye <ev...@apache.org> wrote:
>> > > > Hi Bigtopers,
>> > > >
>> > > > I've shared this on my Apache Big Data talk, but haven't got a
>> chance to
>> > > > share with you yet.
>> > > > This is a Jenkins matrix job to show the healthy status of our
>> component
>> > > > deployment code.
>> > > > Each run deploys hdfs + Y on X, where X is the OS and Y is the
>> component
>> > > > shown on the matrix:
>> > > >
>> > > > https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-
>> > > trunk-deployments/45/
>> > >
>> > > This is super awesome! Thanks for spending time on putting it together!
>> > >
>> > > > Generally speaking, the status of our deployment code needs a lot of
>> > > effort
>> > > > to polish.
>> > > > So, for 1.2.1 release, I think we should make some changes:
>> > > >
>> > > > 1). Remove unmaintained failing components
>> > > > 2). Get maintained components fixed
>> > > >
>> > > > 1) might be easier to achieve, while 2) takes time.
>> > > > In order to maintain a good code quality, I suggest proceed with 1).
>> > > >
>> > > > Any different thoughts?
>> > >
>> > > Shall we at least start with filing JIRAs for everything that's
>> > > failing -- we can discuss
>> > > the particulars on the JIRAs then.
>> > >
>> > > I can try filing some tomorrow if nobody beats me to it (please do ;-))
>> > >
>> > > Thanks,
>> > > Roman.
>> > >
>>
Re: Bigtop Puppet current status
Posted by Evans Ye <ev...@apache.org>.
OK. I agreed with you.
Considered that most of us are making contributions in free time.
Let's make the release less stressed but more solid.
I plan to make 1.2.1 release at the end of the June.
That should make plenty of time for blockers to be removed.
Finally, I'd like to thank you all for taking actions to our deployment
issues.
2017-06-08 1:11 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> On Thu, Jun 08, 2017 at 12:57AM, Evans Ye wrote:
> > Cos,
> > Thanks a lot!
> >
> > Olaf,
> > My original thought was to remove those puppet code which no one has
> > interest to fix it.
> > But you're right. Some of them are broken at package level...
> >
> > I've dig a bit in and found some of them are failing due to Java not
> > presented before daemons starting up.
> > I've uploaded a patch in BIGTOP-2775, which should fix the problem.
> > But we still got many others failing for various reasons.
> >
> > Roman,
> > Good idea. We can track them via JIRAs.
> >
> > All,
> > I still hope that 1.2.1 can be released before DataWorks Summit.
> > However, at this point I'd like to get everybody's thoughts that whether
> we
> > should release it and mark these stuffs as known issues, or we get them
> > fixed and then release a fully functional release.
> > Which one you're in favor?
>
> Eitherway is good for me. Making the release before the mid of next week is
> only possible if we have everything in place right now and only need to do
> the
> release itself and vote on it. Otherwise, the timeframe is too tight...
>
> Cos
>
> > 2017-06-07 6:15 GMT+08:00 Roman Shaposhnik <ro...@shaposhnik.org>:
> >
> > > On Sun, Jun 4, 2017 at 9:32 PM, Evans Ye <ev...@apache.org> wrote:
> > > > Hi Bigtopers,
> > > >
> > > > I've shared this on my Apache Big Data talk, but haven't got a
> chance to
> > > > share with you yet.
> > > > This is a Jenkins matrix job to show the healthy status of our
> component
> > > > deployment code.
> > > > Each run deploys hdfs + Y on X, where X is the OS and Y is the
> component
> > > > shown on the matrix:
> > > >
> > > > https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-
> > > trunk-deployments/45/
> > >
> > > This is super awesome! Thanks for spending time on putting it together!
> > >
> > > > Generally speaking, the status of our deployment code needs a lot of
> > > effort
> > > > to polish.
> > > > So, for 1.2.1 release, I think we should make some changes:
> > > >
> > > > 1). Remove unmaintained failing components
> > > > 2). Get maintained components fixed
> > > >
> > > > 1) might be easier to achieve, while 2) takes time.
> > > > In order to maintain a good code quality, I suggest proceed with 1).
> > > >
> > > > Any different thoughts?
> > >
> > > Shall we at least start with filing JIRAs for everything that's
> > > failing -- we can discuss
> > > the particulars on the JIRAs then.
> > >
> > > I can try filing some tomorrow if nobody beats me to it (please do ;-))
> > >
> > > Thanks,
> > > Roman.
> > >
>
Re: Bigtop Puppet current status
Posted by Konstantin Boudnik <co...@apache.org>.
On Thu, Jun 08, 2017 at 12:57AM, Evans Ye wrote:
> Cos,
> Thanks a lot!
>
> Olaf,
> My original thought was to remove those puppet code which no one has
> interest to fix it.
> But you're right. Some of them are broken at package level...
>
> I've dig a bit in and found some of them are failing due to Java not
> presented before daemons starting up.
> I've uploaded a patch in BIGTOP-2775, which should fix the problem.
> But we still got many others failing for various reasons.
>
> Roman,
> Good idea. We can track them via JIRAs.
>
> All,
> I still hope that 1.2.1 can be released before DataWorks Summit.
> However, at this point I'd like to get everybody's thoughts that whether we
> should release it and mark these stuffs as known issues, or we get them
> fixed and then release a fully functional release.
> Which one you're in favor?
Eitherway is good for me. Making the release before the mid of next week is
only possible if we have everything in place right now and only need to do the
release itself and vote on it. Otherwise, the timeframe is too tight...
Cos
> 2017-06-07 6:15 GMT+08:00 Roman Shaposhnik <ro...@shaposhnik.org>:
>
> > On Sun, Jun 4, 2017 at 9:32 PM, Evans Ye <ev...@apache.org> wrote:
> > > Hi Bigtopers,
> > >
> > > I've shared this on my Apache Big Data talk, but haven't got a chance to
> > > share with you yet.
> > > This is a Jenkins matrix job to show the healthy status of our component
> > > deployment code.
> > > Each run deploys hdfs + Y on X, where X is the OS and Y is the component
> > > shown on the matrix:
> > >
> > > https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-
> > trunk-deployments/45/
> >
> > This is super awesome! Thanks for spending time on putting it together!
> >
> > > Generally speaking, the status of our deployment code needs a lot of
> > effort
> > > to polish.
> > > So, for 1.2.1 release, I think we should make some changes:
> > >
> > > 1). Remove unmaintained failing components
> > > 2). Get maintained components fixed
> > >
> > > 1) might be easier to achieve, while 2) takes time.
> > > In order to maintain a good code quality, I suggest proceed with 1).
> > >
> > > Any different thoughts?
> >
> > Shall we at least start with filing JIRAs for everything that's
> > failing -- we can discuss
> > the particulars on the JIRAs then.
> >
> > I can try filing some tomorrow if nobody beats me to it (please do ;-))
> >
> > Thanks,
> > Roman.
> >
Re: Bigtop Puppet current status
Posted by Evans Ye <ev...@apache.org>.
Cos,
Thanks a lot!
Olaf,
My original thought was to remove those puppet code which no one has
interest to fix it.
But you're right. Some of them are broken at package level...
I've dig a bit in and found some of them are failing due to Java not
presented before daemons starting up.
I've uploaded a patch in BIGTOP-2775, which should fix the problem.
But we still got many others failing for various reasons.
Roman,
Good idea. We can track them via JIRAs.
All,
I still hope that 1.2.1 can be released before DataWorks Summit.
However, at this point I'd like to get everybody's thoughts that whether we
should release it and mark these stuffs as known issues, or we get them
fixed and then release a fully functional release.
Which one you're in favor?
2017-06-07 6:15 GMT+08:00 Roman Shaposhnik <ro...@shaposhnik.org>:
> On Sun, Jun 4, 2017 at 9:32 PM, Evans Ye <ev...@apache.org> wrote:
> > Hi Bigtopers,
> >
> > I've shared this on my Apache Big Data talk, but haven't got a chance to
> > share with you yet.
> > This is a Jenkins matrix job to show the healthy status of our component
> > deployment code.
> > Each run deploys hdfs + Y on X, where X is the OS and Y is the component
> > shown on the matrix:
> >
> > https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-
> trunk-deployments/45/
>
> This is super awesome! Thanks for spending time on putting it together!
>
> > Generally speaking, the status of our deployment code needs a lot of
> effort
> > to polish.
> > So, for 1.2.1 release, I think we should make some changes:
> >
> > 1). Remove unmaintained failing components
> > 2). Get maintained components fixed
> >
> > 1) might be easier to achieve, while 2) takes time.
> > In order to maintain a good code quality, I suggest proceed with 1).
> >
> > Any different thoughts?
>
> Shall we at least start with filing JIRAs for everything that's
> failing -- we can discuss
> the particulars on the JIRAs then.
>
> I can try filing some tomorrow if nobody beats me to it (please do ;-))
>
> Thanks,
> Roman.
>
Re: Bigtop Puppet current status
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Sun, Jun 4, 2017 at 9:32 PM, Evans Ye <ev...@apache.org> wrote:
> Hi Bigtopers,
>
> I've shared this on my Apache Big Data talk, but haven't got a chance to
> share with you yet.
> This is a Jenkins matrix job to show the healthy status of our component
> deployment code.
> Each run deploys hdfs + Y on X, where X is the OS and Y is the component
> shown on the matrix:
>
> https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-trunk-deployments/45/
This is super awesome! Thanks for spending time on putting it together!
> Generally speaking, the status of our deployment code needs a lot of effort
> to polish.
> So, for 1.2.1 release, I think we should make some changes:
>
> 1). Remove unmaintained failing components
> 2). Get maintained components fixed
>
> 1) might be easier to achieve, while 2) takes time.
> In order to maintain a good code quality, I suggest proceed with 1).
>
> Any different thoughts?
Shall we at least start with filing JIRAs for everything that's
failing -- we can discuss
the particulars on the JIRAs then.
I can try filing some tomorrow if nobody beats me to it (please do ;-))
Thanks,
Roman.
Re: Bigtop Puppet current status
Posted by Konstantin Boudnik <co...@apache.org>.
+1
I will fix Ignite - I know about this problem since 1.2.0 and was
meant to work on, but other things took a priority.
In general I completely agree that we don't have to keep struggling
with components without active maintenance. We've been through this
conversation before, which resulted in the introduction of MAINTAINERS
file. However, it hasn't fully fixed the issue.
Thanks,
Cos
--
Take care,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622
Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.
On Sun, Jun 4, 2017 at 9:32 PM, Evans Ye <ev...@apache.org> wrote:
> Hi Bigtopers,
>
> I've shared this on my Apache Big Data talk, but haven't got a chance to
> share with you yet.
> This is a Jenkins matrix job to show the healthy status of our component
> deployment code.
> Each run deploys hdfs + Y on X, where X is the OS and Y is the component
> shown on the matrix:
>
> https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-trunk-deployments/45/
>
> Generally speaking, the status of our deployment code needs a lot of effort
> to polish.
> So, for 1.2.1 release, I think we should make some changes:
>
> 1). Remove unmaintained failing components
> 2). Get maintained components fixed
>
> 1) might be easier to achieve, while 2) takes time.
> In order to maintain a good code quality, I suggest proceed with 1).
>
> Any different thoughts?