You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Evans Ye <ev...@apache.org> on 2015/09/21 10:19:25 UTC
Nightly CI build is failing
Hi all !
I spent some time looking into this and found that the massive job failing
is telling us that the build dependency feature is now working! Which is
because of every build that has dependency setting start from building
upstream packages locally. That fills up the disk size on every build
slaves.
As I mentioned earlier, our CI build env is a sub-optimal design because of
limited disk size on instances. I have no choice to split components
equally on 4 slaves manually to ease the disk consumption.
Maybe I should start trying nexus server to put built jars there?
Or I can specify an option to tell gradle to fetch jars directly from
public maven instead of building upstream components locally?
Thanks,
Evans
Re: Nightly CI build is failing
Posted by Andrew Purtell <an...@gmail.com>.
> Hbase is failing weirdly and don't know what to make out of it 08:57:25 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on project hbase: Error during page generation: Error rendering Maven report: Failed during checkstyle execution: Unable to find suppressions file at location: hbase/checkstyle-suppressions.xml: Could not find resource 'hbase/checkstyle-suppressions.xml'. -> [Help 1]
Sorry I'm just getting to this mail now. We don't have this issue upstream, at least that I've seen when making releases, but let me try a Bigtop build tomorrow and see if I can reproduce it there.
> On Oct 8, 2015, at 9:32 AM, Konstantin Boudnik <co...@apache.org> wrote:
>
> Looks like we are pretty much all-green on the CI side. Two issues left:
> - Mahout expects Maven >=3.3.3 (which I think is coming with the new
> containers, right?)
> - Hbase is failing weirdly and don't know what to make out of it
> 08:57:25 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on project
> hbase: Error during page generation: Error rendering Maven report: Failed
> during checkstyle execution: Unable to find suppressions file at location:
> hbase/checkstyle-suppressions.xml: Could not find resource
> 'hbase/checkstyle-suppressions.xml'. -> [Help 1]
>
> It seems like an upstream issue, isn't it? Andrew, any suggestions?
>
> Thanks,
> Cos
>
>> On Mon, Sep 28, 2015 at 10:23PM, Evans Ye wrote:
>> Fix tez build, which was failing because of some docker daemon issue on the
>> build slave. :)
>>
>> 2015-09-25 19:06 GMT+02:00 Evans Ye <ev...@apache.org>:
>>
>>> With nodeps setting added now our CI is almost back to normal.
>>> There're still 5 components failing but the other's are good.
>>> I need to update our build slaves so that mahout can resume to normal.
>>> I don't know why the others still failing.
>>> Will look into them when back from budapest ;)
>>>
>>> 2015-09-22 2:50 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>>>
>>>> I think for now it would be reasonable to just turn off the dependency
>>>> graph
>>>> building. To do that we need to add
>>>> -Dbuildnodeps=true
>>>> to the gradle command line.
>>>>
>>>> Cos
>>>>
>>>>> On Tue, Sep 22, 2015 at 01:40AM, Evans Ye wrote:
>>>>> Yup, and I think we've discussed that before in BIGTOP-1906.
>>>>>
>>>>> Right now I don't have any clue how to build that server up, but that's
>>>> a
>>>>> good way to go since our users would somehow need to set there internal
>>>>> jars repository for hadoop apps developer to download dependencies.
>>>>>
>>>>> At this point, I'm thinking that maybe Cos can give me a hint of code
>>>>> snippet how to turn the dependency build off. If there's no such
>>>> feature I
>>>>> can try to add it so that we can resume the CI status.
>>>>>
>>>>>
>>>>> 2015-09-21 20:36 GMT+08:00 Jay Vyas <ja...@gmail.com>:
>>>>>
>>>>>> Thanks Evans. I like your idea of separate Bigtop artifacts stored
>>>>>> somewhere...
>>>>>>
>>>>>> If Bigtop can publish its own jars to its own nexus (or else to
>>>> upstream
>>>>>> apaches maven server) that is ideal I guess!
>>>>>>
>>>>>>> On Sep 21, 2015, at 4:19 AM, Evans Ye <ev...@apache.org> wrote:
>>>>>>>
>>>>>>> Hi all !
>>>>>>>
>>>>>>> I spent some time looking into this and found that the massive job
>>>>>> failing
>>>>>>> is telling us that the build dependency feature is now working!
>>>> Which is
>>>>>>> because of every build that has dependency setting start from
>>>> building
>>>>>>> upstream packages locally. That fills up the disk size on every
>>>> build
>>>>>>> slaves.
>>>>>>> As I mentioned earlier, our CI build env is a sub-optimal design
>>>> because
>>>>>> of
>>>>>>> limited disk size on instances. I have no choice to split components
>>>>>>> equally on 4 slaves manually to ease the disk consumption.
>>>>>>>
>>>>>>> Maybe I should start trying nexus server to put built jars there?
>>>>>>> Or I can specify an option to tell gradle to fetch jars directly
>>>> from
>>>>>>> public maven instead of building upstream components locally?
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Evans
>>>
>>>
Re: Nightly CI build is failing
Posted by Konstantin Boudnik <co...@apache.org>.
On Fri, Oct 09, 2015 at 02:55AM, Evans Ye wrote:
> Yes. We'll do that on new CI.
> Changing every job setting is a big headache to me. ;)
Could be done with sed right on the config files, instead of doing this via
webUI. The latter is a pain in the butt, of course.
Cos
> 2015-10-09 2:52 GMT+08:00 Evans Ye <ev...@apache.org>:
>
> > The CI currently running on cloudera master is still using old images w/o
> > trunk prefix, hence it's still not maven 3.3.3 yet.
> > I'd prefer to switch to new images on our new CI master. In new CI env w/
> > sufficient disk we don't need to have bunch of jobs for each package
> > separately. Using a multi-parameter build job would be much easier to
> > maintain.
> >
> > My observation on HBase build is it is flipping. Sometimes it's good,
> > sometimes not. I guess it's the upstream issue...
> >
> > BTW, I noticed that BIGTOP-2059
> > <https://issues.apache.org/jira/browse/BIGTOP-2059>, BIGTOP-2063
> > <https://issues.apache.org/jira/browse/BIGTOP-2063> have been pushed w/o
> > +1.
> > Are you doing CTR, Cos?
> > If so that's fine but we should have a short announcement to officially
> > start the trial. :)
> >
> >
> > 2015-10-09 1:51 GMT+08:00 Olaf Flebbe <of...@oflebbe.de>:
> >
> >> hi,
> >>
> >> The build dockers are updated. Be sure to use
> >> bigtop/slaves:trunk-centos-7 for instance. The trunk-... containers denote
> >> the current git trunk built by docker/build-slaves/... recipies.
> >>
> >> -olaf
> >>
> >>
> >> > Am 08.10.2015 um 19:32 schrieb Konstantin Boudnik <co...@apache.org>:
> >> >
> >> > Indeed, but I don't think we have upgraded the build dockers just yet.
> >> > Otherwise, the build would be just fine, I think.
> >> >
> >> > Cos
> >> >
> >> > On Thu, Oct 08, 2015 at 10:11AM, Jonathan Kelly wrote:
> >> >> Maven should already be 3.3.3 as of
> >> >> https://issues.apache.org/jira/browse/BIGTOP-1953, right?
> >> >>
> >> >> On Thu, Oct 8, 2015 at 9:32 AM, Konstantin Boudnik <co...@apache.org>
> >> wrote:
> >> >>
> >> >>> Looks like we are pretty much all-green on the CI side. Two issues
> >> left:
> >> >>> - Mahout expects Maven >=3.3.3 (which I think is coming with the new
> >> >>> containers, right?)
> >> >>> - Hbase is failing weirdly and don't know what to make out of it
> >> >>> 08:57:25 [ERROR] Failed to execute goal
> >> >>> org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on
> >> >>> project
> >> >>> hbase: Error during page generation: Error rendering Maven report:
> >> Failed
> >> >>> during checkstyle execution: Unable to find suppressions file at
> >> location:
> >> >>> hbase/checkstyle-suppressions.xml: Could not find resource
> >> >>> 'hbase/checkstyle-suppressions.xml'. -> [Help 1]
> >> >>>
> >> >>> It seems like an upstream issue, isn't it? Andrew, any suggestions?
> >> >>>
> >> >>> Thanks,
> >> >>> Cos
> >> >>>
> >> >>> On Mon, Sep 28, 2015 at 10:23PM, Evans Ye wrote:
> >> >>>> Fix tez build, which was failing because of some docker daemon issue
> >> on
> >> >>> the
> >> >>>> build slave. :)
> >> >>>>
> >> >>>> 2015-09-25 19:06 GMT+02:00 Evans Ye <ev...@apache.org>:
> >> >>>>
> >> >>>>> With nodeps setting added now our CI is almost back to normal.
> >> >>>>> There're still 5 components failing but the other's are good.
> >> >>>>> I need to update our build slaves so that mahout can resume to
> >> normal.
> >> >>>>> I don't know why the others still failing.
> >> >>>>> Will look into them when back from budapest ;)
> >> >>>>>
> >> >>>>> 2015-09-22 2:50 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> >> >>>>>
> >> >>>>>> I think for now it would be reasonable to just turn off the
> >> dependency
> >> >>>>>> graph
> >> >>>>>> building. To do that we need to add
> >> >>>>>> -Dbuildnodeps=true
> >> >>>>>> to the gradle command line.
> >> >>>>>>
> >> >>>>>> Cos
> >> >>>>>>
> >> >>>>>> On Tue, Sep 22, 2015 at 01:40AM, Evans Ye wrote:
> >> >>>>>>> Yup, and I think we've discussed that before in BIGTOP-1906.
> >> >>>>>>>
> >> >>>>>>> Right now I don't have any clue how to build that server up, but
> >> >>> that's
> >> >>>>>> a
> >> >>>>>>> good way to go since our users would somehow need to set there
> >> >>> internal
> >> >>>>>>> jars repository for hadoop apps developer to download
> >> dependencies.
> >> >>>>>>>
> >> >>>>>>> At this point, I'm thinking that maybe Cos can give me a hint of
> >> >>> code
> >> >>>>>>> snippet how to turn the dependency build off. If there's no such
> >> >>>>>> feature I
> >> >>>>>>> can try to add it so that we can resume the CI status.
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> 2015-09-21 20:36 GMT+08:00 Jay Vyas <jayunit100.apache@gmail.com
> >> >:
> >> >>>>>>>
> >> >>>>>>>> Thanks Evans. I like your idea of separate Bigtop artifacts
> >> >>> stored
> >> >>>>>>>> somewhere...
> >> >>>>>>>>
> >> >>>>>>>> If Bigtop can publish its own jars to its own nexus (or else to
> >> >>>>>> upstream
> >> >>>>>>>> apaches maven server) that is ideal I guess!
> >> >>>>>>>>
> >> >>>>>>>>> On Sep 21, 2015, at 4:19 AM, Evans Ye <ev...@apache.org>
> >> >>> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>> Hi all !
> >> >>>>>>>>>
> >> >>>>>>>>> I spent some time looking into this and found that the massive
> >> >>> job
> >> >>>>>>>> failing
> >> >>>>>>>>> is telling us that the build dependency feature is now working!
> >> >>>>>> Which is
> >> >>>>>>>>> because of every build that has dependency setting start from
> >> >>>>>> building
> >> >>>>>>>>> upstream packages locally. That fills up the disk size on every
> >> >>>>>> build
> >> >>>>>>>>> slaves.
> >> >>>>>>>>> As I mentioned earlier, our CI build env is a sub-optimal design
> >> >>>>>> because
> >> >>>>>>>> of
> >> >>>>>>>>> limited disk size on instances. I have no choice to split
> >> >>> components
> >> >>>>>>>>> equally on 4 slaves manually to ease the disk consumption.
> >> >>>>>>>>>
> >> >>>>>>>>> Maybe I should start trying nexus server to put built jars
> >> >>> there?
> >> >>>>>>>>> Or I can specify an option to tell gradle to fetch jars directly
> >> >>>>>> from
> >> >>>>>>>>> public maven instead of building upstream components locally?
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>> Thanks,
> >> >>>>>>>>> Evans
> >> >>>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>>
> >> >>>
> >>
> >>
> >
Re: Nightly CI build is failing
Posted by Konstantin Boudnik <co...@apache.org>.
Looks like that's exactly where the problem is coming from. Currently the
master job does this:
docker run --rm -v `pwd`:/ws -v /root/.m2:/root/.m2 bigtop/slaves:$BUILD_ENVIRONMENTS
where the environments are from [centos-6 centos-7 fedora-20 ubuntu-14.04 debian-8].
Looks like that like needs to be changed for something like
docker run --rm -v `pwd`:/ws -v /root/.m2:/root/.m2 bigtop/slaves:trunk-$BUILD_ENVIRONMENTS
Evans, shall we wait until the transition to the new server is over?
Cos
On Thu, Oct 08, 2015 at 07:51PM, Olaf Flebbe wrote:
> hi,
>
> The build dockers are updated. Be sure to use bigtop/slaves:trunk-centos-7
> for instance. The trunk-... containers denote the current git trunk built by
> docker/build-slaves/... recipies.
>
> -olaf
>
>
> > Am 08.10.2015 um 19:32 schrieb Konstantin Boudnik <co...@apache.org>:
> >
> > Indeed, but I don't think we have upgraded the build dockers just yet.
> > Otherwise, the build would be just fine, I think.
> >
> > Cos
> >
> > On Thu, Oct 08, 2015 at 10:11AM, Jonathan Kelly wrote:
> >> Maven should already be 3.3.3 as of
> >> https://issues.apache.org/jira/browse/BIGTOP-1953, right?
> >>
> >> On Thu, Oct 8, 2015 at 9:32 AM, Konstantin Boudnik <co...@apache.org> wrote:
> >>
> >>> Looks like we are pretty much all-green on the CI side. Two issues left:
> >>> - Mahout expects Maven >=3.3.3 (which I think is coming with the new
> >>> containers, right?)
> >>> - Hbase is failing weirdly and don't know what to make out of it
> >>> 08:57:25 [ERROR] Failed to execute goal
> >>> org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on
> >>> project
> >>> hbase: Error during page generation: Error rendering Maven report: Failed
> >>> during checkstyle execution: Unable to find suppressions file at location:
> >>> hbase/checkstyle-suppressions.xml: Could not find resource
> >>> 'hbase/checkstyle-suppressions.xml'. -> [Help 1]
> >>>
> >>> It seems like an upstream issue, isn't it? Andrew, any suggestions?
> >>>
> >>> Thanks,
> >>> Cos
> >>>
> >>> On Mon, Sep 28, 2015 at 10:23PM, Evans Ye wrote:
> >>>> Fix tez build, which was failing because of some docker daemon issue on
> >>> the
> >>>> build slave. :)
> >>>>
> >>>> 2015-09-25 19:06 GMT+02:00 Evans Ye <ev...@apache.org>:
> >>>>
> >>>>> With nodeps setting added now our CI is almost back to normal.
> >>>>> There're still 5 components failing but the other's are good.
> >>>>> I need to update our build slaves so that mahout can resume to normal.
> >>>>> I don't know why the others still failing.
> >>>>> Will look into them when back from budapest ;)
> >>>>>
> >>>>> 2015-09-22 2:50 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> >>>>>
> >>>>>> I think for now it would be reasonable to just turn off the dependency
> >>>>>> graph
> >>>>>> building. To do that we need to add
> >>>>>> -Dbuildnodeps=true
> >>>>>> to the gradle command line.
> >>>>>>
> >>>>>> Cos
> >>>>>>
> >>>>>> On Tue, Sep 22, 2015 at 01:40AM, Evans Ye wrote:
> >>>>>>> Yup, and I think we've discussed that before in BIGTOP-1906.
> >>>>>>>
> >>>>>>> Right now I don't have any clue how to build that server up, but
> >>> that's
> >>>>>> a
> >>>>>>> good way to go since our users would somehow need to set there
> >>> internal
> >>>>>>> jars repository for hadoop apps developer to download dependencies.
> >>>>>>>
> >>>>>>> At this point, I'm thinking that maybe Cos can give me a hint of
> >>> code
> >>>>>>> snippet how to turn the dependency build off. If there's no such
> >>>>>> feature I
> >>>>>>> can try to add it so that we can resume the CI status.
> >>>>>>>
> >>>>>>>
> >>>>>>> 2015-09-21 20:36 GMT+08:00 Jay Vyas <ja...@gmail.com>:
> >>>>>>>
> >>>>>>>> Thanks Evans. I like your idea of separate Bigtop artifacts
> >>> stored
> >>>>>>>> somewhere...
> >>>>>>>>
> >>>>>>>> If Bigtop can publish its own jars to its own nexus (or else to
> >>>>>> upstream
> >>>>>>>> apaches maven server) that is ideal I guess!
> >>>>>>>>
> >>>>>>>>> On Sep 21, 2015, at 4:19 AM, Evans Ye <ev...@apache.org>
> >>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi all !
> >>>>>>>>>
> >>>>>>>>> I spent some time looking into this and found that the massive
> >>> job
> >>>>>>>> failing
> >>>>>>>>> is telling us that the build dependency feature is now working!
> >>>>>> Which is
> >>>>>>>>> because of every build that has dependency setting start from
> >>>>>> building
> >>>>>>>>> upstream packages locally. That fills up the disk size on every
> >>>>>> build
> >>>>>>>>> slaves.
> >>>>>>>>> As I mentioned earlier, our CI build env is a sub-optimal design
> >>>>>> because
> >>>>>>>> of
> >>>>>>>>> limited disk size on instances. I have no choice to split
> >>> components
> >>>>>>>>> equally on 4 slaves manually to ease the disk consumption.
> >>>>>>>>>
> >>>>>>>>> Maybe I should start trying nexus server to put built jars
> >>> there?
> >>>>>>>>> Or I can specify an option to tell gradle to fetch jars directly
> >>>>>> from
> >>>>>>>>> public maven instead of building upstream components locally?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Evans
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>
>
Re: Nightly CI build is failing
Posted by Evans Ye <ev...@apache.org>.
Yes. We'll do that on new CI.
Changing every job setting is a big headache to me. ;)
2015-10-09 2:52 GMT+08:00 Evans Ye <ev...@apache.org>:
> The CI currently running on cloudera master is still using old images w/o
> trunk prefix, hence it's still not maven 3.3.3 yet.
> I'd prefer to switch to new images on our new CI master. In new CI env w/
> sufficient disk we don't need to have bunch of jobs for each package
> separately. Using a multi-parameter build job would be much easier to
> maintain.
>
> My observation on HBase build is it is flipping. Sometimes it's good,
> sometimes not. I guess it's the upstream issue...
>
> BTW, I noticed that BIGTOP-2059
> <https://issues.apache.org/jira/browse/BIGTOP-2059>, BIGTOP-2063
> <https://issues.apache.org/jira/browse/BIGTOP-2063> have been pushed w/o
> +1.
> Are you doing CTR, Cos?
> If so that's fine but we should have a short announcement to officially
> start the trial. :)
>
>
> 2015-10-09 1:51 GMT+08:00 Olaf Flebbe <of...@oflebbe.de>:
>
>> hi,
>>
>> The build dockers are updated. Be sure to use
>> bigtop/slaves:trunk-centos-7 for instance. The trunk-... containers denote
>> the current git trunk built by docker/build-slaves/... recipies.
>>
>> -olaf
>>
>>
>> > Am 08.10.2015 um 19:32 schrieb Konstantin Boudnik <co...@apache.org>:
>> >
>> > Indeed, but I don't think we have upgraded the build dockers just yet.
>> > Otherwise, the build would be just fine, I think.
>> >
>> > Cos
>> >
>> > On Thu, Oct 08, 2015 at 10:11AM, Jonathan Kelly wrote:
>> >> Maven should already be 3.3.3 as of
>> >> https://issues.apache.org/jira/browse/BIGTOP-1953, right?
>> >>
>> >> On Thu, Oct 8, 2015 at 9:32 AM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>> >>
>> >>> Looks like we are pretty much all-green on the CI side. Two issues
>> left:
>> >>> - Mahout expects Maven >=3.3.3 (which I think is coming with the new
>> >>> containers, right?)
>> >>> - Hbase is failing weirdly and don't know what to make out of it
>> >>> 08:57:25 [ERROR] Failed to execute goal
>> >>> org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on
>> >>> project
>> >>> hbase: Error during page generation: Error rendering Maven report:
>> Failed
>> >>> during checkstyle execution: Unable to find suppressions file at
>> location:
>> >>> hbase/checkstyle-suppressions.xml: Could not find resource
>> >>> 'hbase/checkstyle-suppressions.xml'. -> [Help 1]
>> >>>
>> >>> It seems like an upstream issue, isn't it? Andrew, any suggestions?
>> >>>
>> >>> Thanks,
>> >>> Cos
>> >>>
>> >>> On Mon, Sep 28, 2015 at 10:23PM, Evans Ye wrote:
>> >>>> Fix tez build, which was failing because of some docker daemon issue
>> on
>> >>> the
>> >>>> build slave. :)
>> >>>>
>> >>>> 2015-09-25 19:06 GMT+02:00 Evans Ye <ev...@apache.org>:
>> >>>>
>> >>>>> With nodeps setting added now our CI is almost back to normal.
>> >>>>> There're still 5 components failing but the other's are good.
>> >>>>> I need to update our build slaves so that mahout can resume to
>> normal.
>> >>>>> I don't know why the others still failing.
>> >>>>> Will look into them when back from budapest ;)
>> >>>>>
>> >>>>> 2015-09-22 2:50 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>> >>>>>
>> >>>>>> I think for now it would be reasonable to just turn off the
>> dependency
>> >>>>>> graph
>> >>>>>> building. To do that we need to add
>> >>>>>> -Dbuildnodeps=true
>> >>>>>> to the gradle command line.
>> >>>>>>
>> >>>>>> Cos
>> >>>>>>
>> >>>>>> On Tue, Sep 22, 2015 at 01:40AM, Evans Ye wrote:
>> >>>>>>> Yup, and I think we've discussed that before in BIGTOP-1906.
>> >>>>>>>
>> >>>>>>> Right now I don't have any clue how to build that server up, but
>> >>> that's
>> >>>>>> a
>> >>>>>>> good way to go since our users would somehow need to set there
>> >>> internal
>> >>>>>>> jars repository for hadoop apps developer to download
>> dependencies.
>> >>>>>>>
>> >>>>>>> At this point, I'm thinking that maybe Cos can give me a hint of
>> >>> code
>> >>>>>>> snippet how to turn the dependency build off. If there's no such
>> >>>>>> feature I
>> >>>>>>> can try to add it so that we can resume the CI status.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> 2015-09-21 20:36 GMT+08:00 Jay Vyas <jayunit100.apache@gmail.com
>> >:
>> >>>>>>>
>> >>>>>>>> Thanks Evans. I like your idea of separate Bigtop artifacts
>> >>> stored
>> >>>>>>>> somewhere...
>> >>>>>>>>
>> >>>>>>>> If Bigtop can publish its own jars to its own nexus (or else to
>> >>>>>> upstream
>> >>>>>>>> apaches maven server) that is ideal I guess!
>> >>>>>>>>
>> >>>>>>>>> On Sep 21, 2015, at 4:19 AM, Evans Ye <ev...@apache.org>
>> >>> wrote:
>> >>>>>>>>>
>> >>>>>>>>> Hi all !
>> >>>>>>>>>
>> >>>>>>>>> I spent some time looking into this and found that the massive
>> >>> job
>> >>>>>>>> failing
>> >>>>>>>>> is telling us that the build dependency feature is now working!
>> >>>>>> Which is
>> >>>>>>>>> because of every build that has dependency setting start from
>> >>>>>> building
>> >>>>>>>>> upstream packages locally. That fills up the disk size on every
>> >>>>>> build
>> >>>>>>>>> slaves.
>> >>>>>>>>> As I mentioned earlier, our CI build env is a sub-optimal design
>> >>>>>> because
>> >>>>>>>> of
>> >>>>>>>>> limited disk size on instances. I have no choice to split
>> >>> components
>> >>>>>>>>> equally on 4 slaves manually to ease the disk consumption.
>> >>>>>>>>>
>> >>>>>>>>> Maybe I should start trying nexus server to put built jars
>> >>> there?
>> >>>>>>>>> Or I can specify an option to tell gradle to fetch jars directly
>> >>>>>> from
>> >>>>>>>>> public maven instead of building upstream components locally?
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Thanks,
>> >>>>>>>>> Evans
>> >>>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>
>>
>>
>
Re: Nightly CI build is failing
Posted by Evans Ye <ev...@apache.org>.
The CI currently running on cloudera master is still using old images w/o
trunk prefix, hence it's still not maven 3.3.3 yet.
I'd prefer to switch to new images on our new CI master. In new CI env w/
sufficient disk we don't need to have bunch of jobs for each package
separately. Using a multi-parameter build job would be much easier to
maintain.
My observation on HBase build is it is flipping. Sometimes it's good,
sometimes not. I guess it's the upstream issue...
BTW, I noticed that BIGTOP-2059
<https://issues.apache.org/jira/browse/BIGTOP-2059>, BIGTOP-2063
<https://issues.apache.org/jira/browse/BIGTOP-2063> have been pushed w/o +1.
Are you doing CTR, Cos?
If so that's fine but we should have a short announcement to officially
start the trial. :)
2015-10-09 1:51 GMT+08:00 Olaf Flebbe <of...@oflebbe.de>:
> hi,
>
> The build dockers are updated. Be sure to use bigtop/slaves:trunk-centos-7
> for instance. The trunk-... containers denote the current git trunk built
> by docker/build-slaves/... recipies.
>
> -olaf
>
>
> > Am 08.10.2015 um 19:32 schrieb Konstantin Boudnik <co...@apache.org>:
> >
> > Indeed, but I don't think we have upgraded the build dockers just yet.
> > Otherwise, the build would be just fine, I think.
> >
> > Cos
> >
> > On Thu, Oct 08, 2015 at 10:11AM, Jonathan Kelly wrote:
> >> Maven should already be 3.3.3 as of
> >> https://issues.apache.org/jira/browse/BIGTOP-1953, right?
> >>
> >> On Thu, Oct 8, 2015 at 9:32 AM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >>
> >>> Looks like we are pretty much all-green on the CI side. Two issues
> left:
> >>> - Mahout expects Maven >=3.3.3 (which I think is coming with the new
> >>> containers, right?)
> >>> - Hbase is failing weirdly and don't know what to make out of it
> >>> 08:57:25 [ERROR] Failed to execute goal
> >>> org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on
> >>> project
> >>> hbase: Error during page generation: Error rendering Maven report:
> Failed
> >>> during checkstyle execution: Unable to find suppressions file at
> location:
> >>> hbase/checkstyle-suppressions.xml: Could not find resource
> >>> 'hbase/checkstyle-suppressions.xml'. -> [Help 1]
> >>>
> >>> It seems like an upstream issue, isn't it? Andrew, any suggestions?
> >>>
> >>> Thanks,
> >>> Cos
> >>>
> >>> On Mon, Sep 28, 2015 at 10:23PM, Evans Ye wrote:
> >>>> Fix tez build, which was failing because of some docker daemon issue
> on
> >>> the
> >>>> build slave. :)
> >>>>
> >>>> 2015-09-25 19:06 GMT+02:00 Evans Ye <ev...@apache.org>:
> >>>>
> >>>>> With nodeps setting added now our CI is almost back to normal.
> >>>>> There're still 5 components failing but the other's are good.
> >>>>> I need to update our build slaves so that mahout can resume to
> normal.
> >>>>> I don't know why the others still failing.
> >>>>> Will look into them when back from budapest ;)
> >>>>>
> >>>>> 2015-09-22 2:50 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> >>>>>
> >>>>>> I think for now it would be reasonable to just turn off the
> dependency
> >>>>>> graph
> >>>>>> building. To do that we need to add
> >>>>>> -Dbuildnodeps=true
> >>>>>> to the gradle command line.
> >>>>>>
> >>>>>> Cos
> >>>>>>
> >>>>>> On Tue, Sep 22, 2015 at 01:40AM, Evans Ye wrote:
> >>>>>>> Yup, and I think we've discussed that before in BIGTOP-1906.
> >>>>>>>
> >>>>>>> Right now I don't have any clue how to build that server up, but
> >>> that's
> >>>>>> a
> >>>>>>> good way to go since our users would somehow need to set there
> >>> internal
> >>>>>>> jars repository for hadoop apps developer to download dependencies.
> >>>>>>>
> >>>>>>> At this point, I'm thinking that maybe Cos can give me a hint of
> >>> code
> >>>>>>> snippet how to turn the dependency build off. If there's no such
> >>>>>> feature I
> >>>>>>> can try to add it so that we can resume the CI status.
> >>>>>>>
> >>>>>>>
> >>>>>>> 2015-09-21 20:36 GMT+08:00 Jay Vyas <ja...@gmail.com>:
> >>>>>>>
> >>>>>>>> Thanks Evans. I like your idea of separate Bigtop artifacts
> >>> stored
> >>>>>>>> somewhere...
> >>>>>>>>
> >>>>>>>> If Bigtop can publish its own jars to its own nexus (or else to
> >>>>>> upstream
> >>>>>>>> apaches maven server) that is ideal I guess!
> >>>>>>>>
> >>>>>>>>> On Sep 21, 2015, at 4:19 AM, Evans Ye <ev...@apache.org>
> >>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi all !
> >>>>>>>>>
> >>>>>>>>> I spent some time looking into this and found that the massive
> >>> job
> >>>>>>>> failing
> >>>>>>>>> is telling us that the build dependency feature is now working!
> >>>>>> Which is
> >>>>>>>>> because of every build that has dependency setting start from
> >>>>>> building
> >>>>>>>>> upstream packages locally. That fills up the disk size on every
> >>>>>> build
> >>>>>>>>> slaves.
> >>>>>>>>> As I mentioned earlier, our CI build env is a sub-optimal design
> >>>>>> because
> >>>>>>>> of
> >>>>>>>>> limited disk size on instances. I have no choice to split
> >>> components
> >>>>>>>>> equally on 4 slaves manually to ease the disk consumption.
> >>>>>>>>>
> >>>>>>>>> Maybe I should start trying nexus server to put built jars
> >>> there?
> >>>>>>>>> Or I can specify an option to tell gradle to fetch jars directly
> >>>>>> from
> >>>>>>>>> public maven instead of building upstream components locally?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Evans
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>
>
>
Re: Nightly CI build is failing
Posted by Olaf Flebbe <of...@oflebbe.de>.
hi,
The build dockers are updated. Be sure to use bigtop/slaves:trunk-centos-7 for instance. The trunk-... containers denote the current git trunk built by docker/build-slaves/... recipies.
-olaf
> Am 08.10.2015 um 19:32 schrieb Konstantin Boudnik <co...@apache.org>:
>
> Indeed, but I don't think we have upgraded the build dockers just yet.
> Otherwise, the build would be just fine, I think.
>
> Cos
>
> On Thu, Oct 08, 2015 at 10:11AM, Jonathan Kelly wrote:
>> Maven should already be 3.3.3 as of
>> https://issues.apache.org/jira/browse/BIGTOP-1953, right?
>>
>> On Thu, Oct 8, 2015 at 9:32 AM, Konstantin Boudnik <co...@apache.org> wrote:
>>
>>> Looks like we are pretty much all-green on the CI side. Two issues left:
>>> - Mahout expects Maven >=3.3.3 (which I think is coming with the new
>>> containers, right?)
>>> - Hbase is failing weirdly and don't know what to make out of it
>>> 08:57:25 [ERROR] Failed to execute goal
>>> org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on
>>> project
>>> hbase: Error during page generation: Error rendering Maven report: Failed
>>> during checkstyle execution: Unable to find suppressions file at location:
>>> hbase/checkstyle-suppressions.xml: Could not find resource
>>> 'hbase/checkstyle-suppressions.xml'. -> [Help 1]
>>>
>>> It seems like an upstream issue, isn't it? Andrew, any suggestions?
>>>
>>> Thanks,
>>> Cos
>>>
>>> On Mon, Sep 28, 2015 at 10:23PM, Evans Ye wrote:
>>>> Fix tez build, which was failing because of some docker daemon issue on
>>> the
>>>> build slave. :)
>>>>
>>>> 2015-09-25 19:06 GMT+02:00 Evans Ye <ev...@apache.org>:
>>>>
>>>>> With nodeps setting added now our CI is almost back to normal.
>>>>> There're still 5 components failing but the other's are good.
>>>>> I need to update our build slaves so that mahout can resume to normal.
>>>>> I don't know why the others still failing.
>>>>> Will look into them when back from budapest ;)
>>>>>
>>>>> 2015-09-22 2:50 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>>>>>
>>>>>> I think for now it would be reasonable to just turn off the dependency
>>>>>> graph
>>>>>> building. To do that we need to add
>>>>>> -Dbuildnodeps=true
>>>>>> to the gradle command line.
>>>>>>
>>>>>> Cos
>>>>>>
>>>>>> On Tue, Sep 22, 2015 at 01:40AM, Evans Ye wrote:
>>>>>>> Yup, and I think we've discussed that before in BIGTOP-1906.
>>>>>>>
>>>>>>> Right now I don't have any clue how to build that server up, but
>>> that's
>>>>>> a
>>>>>>> good way to go since our users would somehow need to set there
>>> internal
>>>>>>> jars repository for hadoop apps developer to download dependencies.
>>>>>>>
>>>>>>> At this point, I'm thinking that maybe Cos can give me a hint of
>>> code
>>>>>>> snippet how to turn the dependency build off. If there's no such
>>>>>> feature I
>>>>>>> can try to add it so that we can resume the CI status.
>>>>>>>
>>>>>>>
>>>>>>> 2015-09-21 20:36 GMT+08:00 Jay Vyas <ja...@gmail.com>:
>>>>>>>
>>>>>>>> Thanks Evans. I like your idea of separate Bigtop artifacts
>>> stored
>>>>>>>> somewhere...
>>>>>>>>
>>>>>>>> If Bigtop can publish its own jars to its own nexus (or else to
>>>>>> upstream
>>>>>>>> apaches maven server) that is ideal I guess!
>>>>>>>>
>>>>>>>>> On Sep 21, 2015, at 4:19 AM, Evans Ye <ev...@apache.org>
>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi all !
>>>>>>>>>
>>>>>>>>> I spent some time looking into this and found that the massive
>>> job
>>>>>>>> failing
>>>>>>>>> is telling us that the build dependency feature is now working!
>>>>>> Which is
>>>>>>>>> because of every build that has dependency setting start from
>>>>>> building
>>>>>>>>> upstream packages locally. That fills up the disk size on every
>>>>>> build
>>>>>>>>> slaves.
>>>>>>>>> As I mentioned earlier, our CI build env is a sub-optimal design
>>>>>> because
>>>>>>>> of
>>>>>>>>> limited disk size on instances. I have no choice to split
>>> components
>>>>>>>>> equally on 4 slaves manually to ease the disk consumption.
>>>>>>>>>
>>>>>>>>> Maybe I should start trying nexus server to put built jars
>>> there?
>>>>>>>>> Or I can specify an option to tell gradle to fetch jars directly
>>>>>> from
>>>>>>>>> public maven instead of building upstream components locally?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Evans
>>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>
Re: Nightly CI build is failing
Posted by Konstantin Boudnik <co...@apache.org>.
Indeed, but I don't think we have upgraded the build dockers just yet.
Otherwise, the build would be just fine, I think.
Cos
On Thu, Oct 08, 2015 at 10:11AM, Jonathan Kelly wrote:
> Maven should already be 3.3.3 as of
> https://issues.apache.org/jira/browse/BIGTOP-1953, right?
>
> On Thu, Oct 8, 2015 at 9:32 AM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > Looks like we are pretty much all-green on the CI side. Two issues left:
> > - Mahout expects Maven >=3.3.3 (which I think is coming with the new
> > containers, right?)
> > - Hbase is failing weirdly and don't know what to make out of it
> > 08:57:25 [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on
> > project
> > hbase: Error during page generation: Error rendering Maven report: Failed
> > during checkstyle execution: Unable to find suppressions file at location:
> > hbase/checkstyle-suppressions.xml: Could not find resource
> > 'hbase/checkstyle-suppressions.xml'. -> [Help 1]
> >
> > It seems like an upstream issue, isn't it? Andrew, any suggestions?
> >
> > Thanks,
> > Cos
> >
> > On Mon, Sep 28, 2015 at 10:23PM, Evans Ye wrote:
> > > Fix tez build, which was failing because of some docker daemon issue on
> > the
> > > build slave. :)
> > >
> > > 2015-09-25 19:06 GMT+02:00 Evans Ye <ev...@apache.org>:
> > >
> > > > With nodeps setting added now our CI is almost back to normal.
> > > > There're still 5 components failing but the other's are good.
> > > > I need to update our build slaves so that mahout can resume to normal.
> > > > I don't know why the others still failing.
> > > > Will look into them when back from budapest ;)
> > > >
> > > > 2015-09-22 2:50 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > > >
> > > >> I think for now it would be reasonable to just turn off the dependency
> > > >> graph
> > > >> building. To do that we need to add
> > > >> -Dbuildnodeps=true
> > > >> to the gradle command line.
> > > >>
> > > >> Cos
> > > >>
> > > >> On Tue, Sep 22, 2015 at 01:40AM, Evans Ye wrote:
> > > >> > Yup, and I think we've discussed that before in BIGTOP-1906.
> > > >> >
> > > >> > Right now I don't have any clue how to build that server up, but
> > that's
> > > >> a
> > > >> > good way to go since our users would somehow need to set there
> > internal
> > > >> > jars repository for hadoop apps developer to download dependencies.
> > > >> >
> > > >> > At this point, I'm thinking that maybe Cos can give me a hint of
> > code
> > > >> > snippet how to turn the dependency build off. If there's no such
> > > >> feature I
> > > >> > can try to add it so that we can resume the CI status.
> > > >> >
> > > >> >
> > > >> > 2015-09-21 20:36 GMT+08:00 Jay Vyas <ja...@gmail.com>:
> > > >> >
> > > >> > > Thanks Evans. I like your idea of separate Bigtop artifacts
> > stored
> > > >> > > somewhere...
> > > >> > >
> > > >> > > If Bigtop can publish its own jars to its own nexus (or else to
> > > >> upstream
> > > >> > > apaches maven server) that is ideal I guess!
> > > >> > >
> > > >> > > > On Sep 21, 2015, at 4:19 AM, Evans Ye <ev...@apache.org>
> > wrote:
> > > >> > > >
> > > >> > > > Hi all !
> > > >> > > >
> > > >> > > > I spent some time looking into this and found that the massive
> > job
> > > >> > > failing
> > > >> > > > is telling us that the build dependency feature is now working!
> > > >> Which is
> > > >> > > > because of every build that has dependency setting start from
> > > >> building
> > > >> > > > upstream packages locally. That fills up the disk size on every
> > > >> build
> > > >> > > > slaves.
> > > >> > > > As I mentioned earlier, our CI build env is a sub-optimal design
> > > >> because
> > > >> > > of
> > > >> > > > limited disk size on instances. I have no choice to split
> > components
> > > >> > > > equally on 4 slaves manually to ease the disk consumption.
> > > >> > > >
> > > >> > > > Maybe I should start trying nexus server to put built jars
> > there?
> > > >> > > > Or I can specify an option to tell gradle to fetch jars directly
> > > >> from
> > > >> > > > public maven instead of building upstream components locally?
> > > >> > > >
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Evans
> > > >> > >
> > > >>
> > > >
> > > >
> >
Re: Nightly CI build is failing
Posted by Jonathan Kelly <jo...@gmail.com>.
Maven should already be 3.3.3 as of
https://issues.apache.org/jira/browse/BIGTOP-1953, right?
On Thu, Oct 8, 2015 at 9:32 AM, Konstantin Boudnik <co...@apache.org> wrote:
> Looks like we are pretty much all-green on the CI side. Two issues left:
> - Mahout expects Maven >=3.3.3 (which I think is coming with the new
> containers, right?)
> - Hbase is failing weirdly and don't know what to make out of it
> 08:57:25 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on
> project
> hbase: Error during page generation: Error rendering Maven report: Failed
> during checkstyle execution: Unable to find suppressions file at location:
> hbase/checkstyle-suppressions.xml: Could not find resource
> 'hbase/checkstyle-suppressions.xml'. -> [Help 1]
>
> It seems like an upstream issue, isn't it? Andrew, any suggestions?
>
> Thanks,
> Cos
>
> On Mon, Sep 28, 2015 at 10:23PM, Evans Ye wrote:
> > Fix tez build, which was failing because of some docker daemon issue on
> the
> > build slave. :)
> >
> > 2015-09-25 19:06 GMT+02:00 Evans Ye <ev...@apache.org>:
> >
> > > With nodeps setting added now our CI is almost back to normal.
> > > There're still 5 components failing but the other's are good.
> > > I need to update our build slaves so that mahout can resume to normal.
> > > I don't know why the others still failing.
> > > Will look into them when back from budapest ;)
> > >
> > > 2015-09-22 2:50 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > >
> > >> I think for now it would be reasonable to just turn off the dependency
> > >> graph
> > >> building. To do that we need to add
> > >> -Dbuildnodeps=true
> > >> to the gradle command line.
> > >>
> > >> Cos
> > >>
> > >> On Tue, Sep 22, 2015 at 01:40AM, Evans Ye wrote:
> > >> > Yup, and I think we've discussed that before in BIGTOP-1906.
> > >> >
> > >> > Right now I don't have any clue how to build that server up, but
> that's
> > >> a
> > >> > good way to go since our users would somehow need to set there
> internal
> > >> > jars repository for hadoop apps developer to download dependencies.
> > >> >
> > >> > At this point, I'm thinking that maybe Cos can give me a hint of
> code
> > >> > snippet how to turn the dependency build off. If there's no such
> > >> feature I
> > >> > can try to add it so that we can resume the CI status.
> > >> >
> > >> >
> > >> > 2015-09-21 20:36 GMT+08:00 Jay Vyas <ja...@gmail.com>:
> > >> >
> > >> > > Thanks Evans. I like your idea of separate Bigtop artifacts
> stored
> > >> > > somewhere...
> > >> > >
> > >> > > If Bigtop can publish its own jars to its own nexus (or else to
> > >> upstream
> > >> > > apaches maven server) that is ideal I guess!
> > >> > >
> > >> > > > On Sep 21, 2015, at 4:19 AM, Evans Ye <ev...@apache.org>
> wrote:
> > >> > > >
> > >> > > > Hi all !
> > >> > > >
> > >> > > > I spent some time looking into this and found that the massive
> job
> > >> > > failing
> > >> > > > is telling us that the build dependency feature is now working!
> > >> Which is
> > >> > > > because of every build that has dependency setting start from
> > >> building
> > >> > > > upstream packages locally. That fills up the disk size on every
> > >> build
> > >> > > > slaves.
> > >> > > > As I mentioned earlier, our CI build env is a sub-optimal design
> > >> because
> > >> > > of
> > >> > > > limited disk size on instances. I have no choice to split
> components
> > >> > > > equally on 4 slaves manually to ease the disk consumption.
> > >> > > >
> > >> > > > Maybe I should start trying nexus server to put built jars
> there?
> > >> > > > Or I can specify an option to tell gradle to fetch jars directly
> > >> from
> > >> > > > public maven instead of building upstream components locally?
> > >> > > >
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Evans
> > >> > >
> > >>
> > >
> > >
>
Re: Nightly CI build is failing
Posted by Konstantin Boudnik <co...@apache.org>.
Looks like we are pretty much all-green on the CI side. Two issues left:
- Mahout expects Maven >=3.3.3 (which I think is coming with the new
containers, right?)
- Hbase is failing weirdly and don't know what to make out of it
08:57:25 [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on project
hbase: Error during page generation: Error rendering Maven report: Failed
during checkstyle execution: Unable to find suppressions file at location:
hbase/checkstyle-suppressions.xml: Could not find resource
'hbase/checkstyle-suppressions.xml'. -> [Help 1]
It seems like an upstream issue, isn't it? Andrew, any suggestions?
Thanks,
Cos
On Mon, Sep 28, 2015 at 10:23PM, Evans Ye wrote:
> Fix tez build, which was failing because of some docker daemon issue on the
> build slave. :)
>
> 2015-09-25 19:06 GMT+02:00 Evans Ye <ev...@apache.org>:
>
> > With nodeps setting added now our CI is almost back to normal.
> > There're still 5 components failing but the other's are good.
> > I need to update our build slaves so that mahout can resume to normal.
> > I don't know why the others still failing.
> > Will look into them when back from budapest ;)
> >
> > 2015-09-22 2:50 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> >
> >> I think for now it would be reasonable to just turn off the dependency
> >> graph
> >> building. To do that we need to add
> >> -Dbuildnodeps=true
> >> to the gradle command line.
> >>
> >> Cos
> >>
> >> On Tue, Sep 22, 2015 at 01:40AM, Evans Ye wrote:
> >> > Yup, and I think we've discussed that before in BIGTOP-1906.
> >> >
> >> > Right now I don't have any clue how to build that server up, but that's
> >> a
> >> > good way to go since our users would somehow need to set there internal
> >> > jars repository for hadoop apps developer to download dependencies.
> >> >
> >> > At this point, I'm thinking that maybe Cos can give me a hint of code
> >> > snippet how to turn the dependency build off. If there's no such
> >> feature I
> >> > can try to add it so that we can resume the CI status.
> >> >
> >> >
> >> > 2015-09-21 20:36 GMT+08:00 Jay Vyas <ja...@gmail.com>:
> >> >
> >> > > Thanks Evans. I like your idea of separate Bigtop artifacts stored
> >> > > somewhere...
> >> > >
> >> > > If Bigtop can publish its own jars to its own nexus (or else to
> >> upstream
> >> > > apaches maven server) that is ideal I guess!
> >> > >
> >> > > > On Sep 21, 2015, at 4:19 AM, Evans Ye <ev...@apache.org> wrote:
> >> > > >
> >> > > > Hi all !
> >> > > >
> >> > > > I spent some time looking into this and found that the massive job
> >> > > failing
> >> > > > is telling us that the build dependency feature is now working!
> >> Which is
> >> > > > because of every build that has dependency setting start from
> >> building
> >> > > > upstream packages locally. That fills up the disk size on every
> >> build
> >> > > > slaves.
> >> > > > As I mentioned earlier, our CI build env is a sub-optimal design
> >> because
> >> > > of
> >> > > > limited disk size on instances. I have no choice to split components
> >> > > > equally on 4 slaves manually to ease the disk consumption.
> >> > > >
> >> > > > Maybe I should start trying nexus server to put built jars there?
> >> > > > Or I can specify an option to tell gradle to fetch jars directly
> >> from
> >> > > > public maven instead of building upstream components locally?
> >> > > >
> >> > > >
> >> > > > Thanks,
> >> > > > Evans
> >> > >
> >>
> >
> >
Re: Nightly CI build is failing
Posted by Evans Ye <ev...@apache.org>.
Fix tez build, which was failing because of some docker daemon issue on the
build slave. :)
2015-09-25 19:06 GMT+02:00 Evans Ye <ev...@apache.org>:
> With nodeps setting added now our CI is almost back to normal.
> There're still 5 components failing but the other's are good.
> I need to update our build slaves so that mahout can resume to normal.
> I don't know why the others still failing.
> Will look into them when back from budapest ;)
>
> 2015-09-22 2:50 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>
>> I think for now it would be reasonable to just turn off the dependency
>> graph
>> building. To do that we need to add
>> -Dbuildnodeps=true
>> to the gradle command line.
>>
>> Cos
>>
>> On Tue, Sep 22, 2015 at 01:40AM, Evans Ye wrote:
>> > Yup, and I think we've discussed that before in BIGTOP-1906.
>> >
>> > Right now I don't have any clue how to build that server up, but that's
>> a
>> > good way to go since our users would somehow need to set there internal
>> > jars repository for hadoop apps developer to download dependencies.
>> >
>> > At this point, I'm thinking that maybe Cos can give me a hint of code
>> > snippet how to turn the dependency build off. If there's no such
>> feature I
>> > can try to add it so that we can resume the CI status.
>> >
>> >
>> > 2015-09-21 20:36 GMT+08:00 Jay Vyas <ja...@gmail.com>:
>> >
>> > > Thanks Evans. I like your idea of separate Bigtop artifacts stored
>> > > somewhere...
>> > >
>> > > If Bigtop can publish its own jars to its own nexus (or else to
>> upstream
>> > > apaches maven server) that is ideal I guess!
>> > >
>> > > > On Sep 21, 2015, at 4:19 AM, Evans Ye <ev...@apache.org> wrote:
>> > > >
>> > > > Hi all !
>> > > >
>> > > > I spent some time looking into this and found that the massive job
>> > > failing
>> > > > is telling us that the build dependency feature is now working!
>> Which is
>> > > > because of every build that has dependency setting start from
>> building
>> > > > upstream packages locally. That fills up the disk size on every
>> build
>> > > > slaves.
>> > > > As I mentioned earlier, our CI build env is a sub-optimal design
>> because
>> > > of
>> > > > limited disk size on instances. I have no choice to split components
>> > > > equally on 4 slaves manually to ease the disk consumption.
>> > > >
>> > > > Maybe I should start trying nexus server to put built jars there?
>> > > > Or I can specify an option to tell gradle to fetch jars directly
>> from
>> > > > public maven instead of building upstream components locally?
>> > > >
>> > > >
>> > > > Thanks,
>> > > > Evans
>> > >
>>
>
>
Re: Nightly CI build is failing
Posted by Evans Ye <ev...@apache.org>.
With nodeps setting added now our CI is almost back to normal.
There're still 5 components failing but the other's are good.
I need to update our build slaves so that mahout can resume to normal.
I don't know why the others still failing.
Will look into them when back from budapest ;)
2015-09-22 2:50 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> I think for now it would be reasonable to just turn off the dependency
> graph
> building. To do that we need to add
> -Dbuildnodeps=true
> to the gradle command line.
>
> Cos
>
> On Tue, Sep 22, 2015 at 01:40AM, Evans Ye wrote:
> > Yup, and I think we've discussed that before in BIGTOP-1906.
> >
> > Right now I don't have any clue how to build that server up, but that's a
> > good way to go since our users would somehow need to set there internal
> > jars repository for hadoop apps developer to download dependencies.
> >
> > At this point, I'm thinking that maybe Cos can give me a hint of code
> > snippet how to turn the dependency build off. If there's no such feature
> I
> > can try to add it so that we can resume the CI status.
> >
> >
> > 2015-09-21 20:36 GMT+08:00 Jay Vyas <ja...@gmail.com>:
> >
> > > Thanks Evans. I like your idea of separate Bigtop artifacts stored
> > > somewhere...
> > >
> > > If Bigtop can publish its own jars to its own nexus (or else to
> upstream
> > > apaches maven server) that is ideal I guess!
> > >
> > > > On Sep 21, 2015, at 4:19 AM, Evans Ye <ev...@apache.org> wrote:
> > > >
> > > > Hi all !
> > > >
> > > > I spent some time looking into this and found that the massive job
> > > failing
> > > > is telling us that the build dependency feature is now working!
> Which is
> > > > because of every build that has dependency setting start from
> building
> > > > upstream packages locally. That fills up the disk size on every build
> > > > slaves.
> > > > As I mentioned earlier, our CI build env is a sub-optimal design
> because
> > > of
> > > > limited disk size on instances. I have no choice to split components
> > > > equally on 4 slaves manually to ease the disk consumption.
> > > >
> > > > Maybe I should start trying nexus server to put built jars there?
> > > > Or I can specify an option to tell gradle to fetch jars directly from
> > > > public maven instead of building upstream components locally?
> > > >
> > > >
> > > > Thanks,
> > > > Evans
> > >
>
Re: Nightly CI build is failing
Posted by Konstantin Boudnik <co...@apache.org>.
I think for now it would be reasonable to just turn off the dependency graph
building. To do that we need to add
-Dbuildnodeps=true
to the gradle command line.
Cos
On Tue, Sep 22, 2015 at 01:40AM, Evans Ye wrote:
> Yup, and I think we've discussed that before in BIGTOP-1906.
>
> Right now I don't have any clue how to build that server up, but that's a
> good way to go since our users would somehow need to set there internal
> jars repository for hadoop apps developer to download dependencies.
>
> At this point, I'm thinking that maybe Cos can give me a hint of code
> snippet how to turn the dependency build off. If there's no such feature I
> can try to add it so that we can resume the CI status.
>
>
> 2015-09-21 20:36 GMT+08:00 Jay Vyas <ja...@gmail.com>:
>
> > Thanks Evans. I like your idea of separate Bigtop artifacts stored
> > somewhere...
> >
> > If Bigtop can publish its own jars to its own nexus (or else to upstream
> > apaches maven server) that is ideal I guess!
> >
> > > On Sep 21, 2015, at 4:19 AM, Evans Ye <ev...@apache.org> wrote:
> > >
> > > Hi all !
> > >
> > > I spent some time looking into this and found that the massive job
> > failing
> > > is telling us that the build dependency feature is now working! Which is
> > > because of every build that has dependency setting start from building
> > > upstream packages locally. That fills up the disk size on every build
> > > slaves.
> > > As I mentioned earlier, our CI build env is a sub-optimal design because
> > of
> > > limited disk size on instances. I have no choice to split components
> > > equally on 4 slaves manually to ease the disk consumption.
> > >
> > > Maybe I should start trying nexus server to put built jars there?
> > > Or I can specify an option to tell gradle to fetch jars directly from
> > > public maven instead of building upstream components locally?
> > >
> > >
> > > Thanks,
> > > Evans
> >
Re: Nightly CI build is failing
Posted by Evans Ye <ev...@apache.org>.
Yup, and I think we've discussed that before in BIGTOP-1906.
Right now I don't have any clue how to build that server up, but that's a
good way to go since our users would somehow need to set there internal
jars repository for hadoop apps developer to download dependencies.
At this point, I'm thinking that maybe Cos can give me a hint of code
snippet how to turn the dependency build off. If there's no such feature I
can try to add it so that we can resume the CI status.
2015-09-21 20:36 GMT+08:00 Jay Vyas <ja...@gmail.com>:
> Thanks Evans. I like your idea of separate Bigtop artifacts stored
> somewhere...
>
> If Bigtop can publish its own jars to its own nexus (or else to upstream
> apaches maven server) that is ideal I guess!
>
> > On Sep 21, 2015, at 4:19 AM, Evans Ye <ev...@apache.org> wrote:
> >
> > Hi all !
> >
> > I spent some time looking into this and found that the massive job
> failing
> > is telling us that the build dependency feature is now working! Which is
> > because of every build that has dependency setting start from building
> > upstream packages locally. That fills up the disk size on every build
> > slaves.
> > As I mentioned earlier, our CI build env is a sub-optimal design because
> of
> > limited disk size on instances. I have no choice to split components
> > equally on 4 slaves manually to ease the disk consumption.
> >
> > Maybe I should start trying nexus server to put built jars there?
> > Or I can specify an option to tell gradle to fetch jars directly from
> > public maven instead of building upstream components locally?
> >
> >
> > Thanks,
> > Evans
>
Re: Nightly CI build is failing
Posted by Jay Vyas <ja...@gmail.com>.
Thanks Evans. I like your idea of separate Bigtop artifacts stored somewhere...
If Bigtop can publish its own jars to its own nexus (or else to upstream apaches maven server) that is ideal I guess!
> On Sep 21, 2015, at 4:19 AM, Evans Ye <ev...@apache.org> wrote:
>
> Hi all !
>
> I spent some time looking into this and found that the massive job failing
> is telling us that the build dependency feature is now working! Which is
> because of every build that has dependency setting start from building
> upstream packages locally. That fills up the disk size on every build
> slaves.
> As I mentioned earlier, our CI build env is a sub-optimal design because of
> limited disk size on instances. I have no choice to split components
> equally on 4 slaves manually to ease the disk consumption.
>
> Maybe I should start trying nexus server to put built jars there?
> Or I can specify an option to tell gradle to fetch jars directly from
> public maven instead of building upstream components locally?
>
>
> Thanks,
> Evans