You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Dagang Wei <fu...@gmail.com> on 2018/07/22 17:33:34 UTC
Re: Speed up Bigtop build through parallelism?
Hi Cos,
Is there a guide or could you point me to the source code of "splitting the
build by components"?
The problem I am experience now is that, sometimes component build (e.g.,
hbase) is flaky (due to network issue, etc), then I need to retry the whole
build. I am thinking about splitting package build with apt repository
build, apt repository build depends on debs, so that instead of retry the
whole build for a single component failure, only retry the failed component
build.
Thanks!
On Sun, Jun 17, 2018 at 10:51 AM functicons <fu...@gmail.com> wrote:
> Thanks Cos!
>
> On Sat, Jun 16, 2018 at 12:58 AM Konstantin Boudnik <co...@apache.org>
> wrote:
>
>> What we do is splitting the build by components (remember that some of
>> them are dependent on each other: HBase-Hadoop, etc.) and then execute
>> component builds at different agents.
>> That's what we do with ci.bigtop.apache.org
>>
>> There is a tricky step of gathering all the artifacts together, but it is
>> relatively easy to accomplish.
>>
>> Hope it helps.
>> --
>> Regards,
>> Cos
>>
>> On June 16, 2018 7:16:21 AM GMT+03:00, functicons <fu...@gmail.com>
>> wrote:
>> >Hi folks,
>> >
>> >We are building packages from our fork of Bigtop, but the build takes
>> >really long. So I'm wondering is it possible to build in parallel? All
>> >suggestions are appreciated. Thanks!
>> >
>> >Dagang
>>
>
Re: Speed up Bigtop build through parallelism?
Posted by Sergey Soldatov <se...@gmail.com>.
OK
On Jul 22, 2018 12:28 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
Check this out
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
--
With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622
Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.
On Sun, Jul 22, 2018 at 8:33 PM, Dagang Wei <fu...@gmail.com> wrote:
> Hi Cos,
>
> Is there a guide or could you point me to the source code of "splitting
the
> build by components"?
>
> The problem I am experience now is that, sometimes component build (e.g.,
> hbase) is flaky (due to network issue, etc), then I need to retry the
whole
> build. I am thinking about splitting package build with apt repository
> build, apt repository build depends on debs, so that instead of retry the
> whole build for a single component failure, only retry the failed
component
> build.
>
> Thanks!
>
> On Sun, Jun 17, 2018 at 10:51 AM functicons <fu...@gmail.com> wrote:
>>
>> Thanks Cos!
>>
>> On Sat, Jun 16, 2018 at 12:58 AM Konstantin Boudnik <co...@apache.org>
>> wrote:
>>>
>>> What we do is splitting the build by components (remember that some of
>>> them are dependent on each other: HBase-Hadoop, etc.) and then execute
>>> component builds at different agents.
>>> That's what we do with ci.bigtop.apache.org
>>>
>>> There is a tricky step of gathering all the artifacts together, but it
is
>>> relatively easy to accomplish.
>>>
>>> Hope it helps.
>>> --
>>> Regards,
>>> Cos
>>>
>>> On June 16, 2018 7:16:21 AM GMT+03:00, functicons <fu...@gmail.com>
>>> wrote:
>>> >Hi folks,
>>> >
>>> >We are building packages from our fork of Bigtop, but the build takes
>>> >really long. So I'm wondering is it possible to build in parallel? All
>>> >suggestions are appreciated. Thanks!
>>> >
>>> >Dagang
Re: Speed up Bigtop build through parallelism?
Posted by Konstantin Boudnik <co...@apache.org>.
Check this out
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
--
With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622
Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.
On Sun, Jul 22, 2018 at 8:33 PM, Dagang Wei <fu...@gmail.com> wrote:
> Hi Cos,
>
> Is there a guide or could you point me to the source code of "splitting the
> build by components"?
>
> The problem I am experience now is that, sometimes component build (e.g.,
> hbase) is flaky (due to network issue, etc), then I need to retry the whole
> build. I am thinking about splitting package build with apt repository
> build, apt repository build depends on debs, so that instead of retry the
> whole build for a single component failure, only retry the failed component
> build.
>
> Thanks!
>
> On Sun, Jun 17, 2018 at 10:51 AM functicons <fu...@gmail.com> wrote:
>>
>> Thanks Cos!
>>
>> On Sat, Jun 16, 2018 at 12:58 AM Konstantin Boudnik <co...@apache.org>
>> wrote:
>>>
>>> What we do is splitting the build by components (remember that some of
>>> them are dependent on each other: HBase-Hadoop, etc.) and then execute
>>> component builds at different agents.
>>> That's what we do with ci.bigtop.apache.org
>>>
>>> There is a tricky step of gathering all the artifacts together, but it is
>>> relatively easy to accomplish.
>>>
>>> Hope it helps.
>>> --
>>> Regards,
>>> Cos
>>>
>>> On June 16, 2018 7:16:21 AM GMT+03:00, functicons <fu...@gmail.com>
>>> wrote:
>>> >Hi folks,
>>> >
>>> >We are building packages from our fork of Bigtop, but the build takes
>>> >really long. So I'm wondering is it possible to build in parallel? All
>>> >suggestions are appreciated. Thanks!
>>> >
>>> >Dagang