You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Zhenyu Zheng <zh...@gmail.com> on 2019/07/08 07:42:12 UTC
[Bigtop] Some question about build Hadoop on AArch64
Hi Bigtop:
I'm trying to build Hadoop on ARM64 platform and I've meet some problem
related to protobuf 2.5.0 seems it does not build on ARM64, and there is an
issue in Hadoop about this:
https://issues.apache.org/jira/browse/HADOOP-13363 which has not yet been
solved. But I noticed that Bigtop has successfuly built Hadoop 2.8.5 on
ARM64, eg.
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=ubuntu-16.04-aarch64/
so
I'm quite curious how bigtop solved this problem?
Thanks in advance
Kevin Zheng
Re: [Bigtop] Some question about build Hadoop on AArch64
Posted by Jay Vyas <ja...@gmail.com>.
Just making sure, so my next question then is what the story for arm64 container builds are.... and wether or not we need different base images (I assume we do).
That said I think you *can* build arm images on an x86 based computer as long as you set up the docket image correctly... and the original poster wasn’t able to build on Arm, so must be a subtle missing piece in our container builds.?
> On Jul 8, 2019, at 4:05 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
>
> Hi Jay,
>
> no not a stretch. We already have it.
>
> yes we have containerized builds for long on all platforms (ppc64le, aarch64 and amd64). Local builders are deprecated since bigtop-1.2.
>
> Documented in confluence
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk
>
> What specifically are you missing?
>
> Olaf
>
>
>
>> On 08.07.19 21:27, Jay Vyas wrote:
>> Hi olaf- we do have containeirzed builders now, right? I might be wrong about this, as I haven’t built full packages for bigtop in the last 12 months or so, but I used to use docker... Am thinking we should update docs to reflect that to build bigtop it’s easier to use our docker containers (Evans gave a good presentation on this https://events.static.linuxfound.org/sites/events/files/slides/How%20Bigtop%20Leveraged%20Docker%20for%20Build%20Automation%20and%20%20One-click%20Hadoop%20Provisioning%20ApacheBigData2015EU%20pub.pdf). ... and really just move away from using local builders of any other sort.... is that too much of a stretch?
>>> On Jul 8, 2019, at 3:11 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
>>>
>>> Hi
>>>
>>> Right. This is automated compilation of protobuf is done while preparing the docker images.
>>>
>>> bigtop_toolchain/manifests/protobuf.pp
>>>
>>> I like to note we use this approach for all platforms for consistency.
>>>
>>> Best,
>>> Olaf
>>>
>>>
>>>> On 08.07.19 10:18, Zhenyu Zheng wrote:
>>>> Hi Yuqi,
>>>> Thanks for the reply, so can I understand that what bigtop did is similar
>>>> to what was suggested in:
>>>> https://groups.google.com/forum/#!topic/protobuf/fwLF5_t3q3U ? I mean,
>>>> backport some patches
>>>> to 2.5.0 and build a protobuf that can work on ARM64?
>>>> BR,
>>>> Kevin Zheng
>>>>> On Mon, Jul 8, 2019 at 4:14 PM Yuqi Gu <yu...@linaro.org> wrote:
>>>>> Hi Zhenyu Zheng,
>>>>>
>>>>> Pls try to build Hadoop in docker:
>>>>> docker run -v `pwd`:/ws bigtop/slaves:1.4.0-ubuntu-16.04-aarch64 bash -l -c
>>>>> 'cd /ws ; hadoop-pkg'
>>>>>
>>>>> The protobuf patches for Hadoop are list: Here
>>>>> <
>>>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/do-component-build#L45-L105
>>>>>>
>>>>>
>>>>> BRs,
>>>>> Yuqi
>>>>>
>>>>> On Mon, 8 Jul 2019 at 15:42, Zhenyu Zheng <zh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Bigtop:
>>>>>>
>>>>>> I'm trying to build Hadoop on ARM64 platform and I've meet some problem
>>>>>> related to protobuf 2.5.0 seems it does not build on ARM64, and there is
>>>>> an
>>>>>> issue in Hadoop about this:
>>>>>> https://issues.apache.org/jira/browse/HADOOP-13363 which has not yet
>>>>> been
>>>>>> solved. But I noticed that Bigtop has successfuly built Hadoop 2.8.5 on
>>>>>> ARM64, eg.
>>>>>>
>>>>>>
>>>>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=ubuntu-16.04-aarch64/
>>>>>> so
>>>>>> I'm quite curious how bigtop solved this problem?
>>>>>>
>>>>>> Thanks in advance
>>>>>>
>>>>>> Kevin Zheng
>>>>>>
>>>>>
Re: [Bigtop] Some question about build Hadoop on AArch64
Posted by Evans Ye <ev...@apache.org>.
Since bigtop 1.4 we have all things integrated at gradlew and its easier to
build/test on docker. See:
https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
Olaf Flebbe <of...@oflebbe.de> 於 2019年7月9日 週二 04:05 寫道：
> Hi Jay,
>
> no not a stretch. We already have it.
>
> yes we have containerized builds for long on all platforms (ppc64le,
> aarch64 and amd64). Local builders are deprecated since bigtop-1.2.
>
> Documented in confluence
>
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk
>
> What specifically are you missing?
>
> Olaf
>
>
>
> On 08.07.19 21:27, Jay Vyas wrote:
> > Hi olaf- we do have containeirzed builders now, right? I might be wrong
> about this, as I haven’t built full packages for bigtop in the last 12
> months or so, but I used to use docker... Am thinking we should update docs
> to reflect that to build bigtop it’s easier to use our docker containers
> (Evans gave a good presentation on this
> https://events.static.linuxfound.org/sites/events/files/slides/How%20Bigtop%20Leveraged%20Docker%20for%20Build%20Automation%20and%20%20One-click%20Hadoop%20Provisioning%20ApacheBigData2015EU%20pub.pdf).
> ... and really just move away from using local builders of any other
> sort.... is that too much of a stretch?
> >
> >> On Jul 8, 2019, at 3:11 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
> >>
> >> Hi
> >>
> >> Right. This is automated compilation of protobuf is done while
> preparing the docker images.
> >>
> >> bigtop_toolchain/manifests/protobuf.pp
> >>
> >> I like to note we use this approach for all platforms for consistency.
> >>
> >> Best,
> >> Olaf
> >>
> >>
> >>> On 08.07.19 10:18, Zhenyu Zheng wrote:
> >>> Hi Yuqi,
> >>> Thanks for the reply, so can I understand that what bigtop did is
> similar
> >>> to what was suggested in:
> >>> https://groups.google.com/forum/#!topic/protobuf/fwLF5_t3q3U ? I
> mean,
> >>> backport some patches
> >>> to 2.5.0 and build a protobuf that can work on ARM64?
> >>> BR,
> >>> Kevin Zheng
> >>>> On Mon, Jul 8, 2019 at 4:14 PM Yuqi Gu <yu...@linaro.org> wrote:
> >>>> Hi Zhenyu Zheng,
> >>>>
> >>>> Pls try to build Hadoop in docker:
> >>>> docker run -v `pwd`:/ws bigtop/slaves:1.4.0-ubuntu-16.04-aarch64 bash
> -l -c
> >>>> 'cd /ws ; hadoop-pkg'
> >>>>
> >>>> The protobuf patches for Hadoop are list: Here
> >>>> <
> >>>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/do-component-build#L45-L105
> >>>>>
> >>>>
> >>>> BRs,
> >>>> Yuqi
> >>>>
> >>>> On Mon, 8 Jul 2019 at 15:42, Zhenyu Zheng <zh...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Bigtop:
> >>>>>
> >>>>> I'm trying to build Hadoop on ARM64 platform and I've meet some
> problem
> >>>>> related to protobuf 2.5.0 seems it does not build on ARM64, and
> there is
> >>>> an
> >>>>> issue in Hadoop about this:
> >>>>> https://issues.apache.org/jira/browse/HADOOP-13363 which has not yet
> >>>> been
> >>>>> solved. But I noticed that Bigtop has successfuly built Hadoop 2.8.5
> on
> >>>>> ARM64, eg.
> >>>>>
> >>>>>
> >>>>
> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=ubuntu-16.04-aarch64/
> >>>>> so
> >>>>> I'm quite curious how bigtop solved this problem?
> >>>>>
> >>>>> Thanks in advance
> >>>>>
> >>>>> Kevin Zheng
> >>>>>
> >>>>
> >
>
Re: [Bigtop] Some question about build Hadoop on AArch64
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi Jay,
no not a stretch. We already have it.
yes we have containerized builds for long on all platforms (ppc64le,
aarch64 and amd64). Local builders are deprecated since bigtop-1.2.
Documented in confluence
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk
What specifically are you missing?
Olaf
On 08.07.19 21:27, Jay Vyas wrote:
> Hi olaf- we do have containeirzed builders now, right? I might be wrong about this, as I haven’t built full packages for bigtop in the last 12 months or so, but I used to use docker... Am thinking we should update docs to reflect that to build bigtop it’s easier to use our docker containers (Evans gave a good presentation on this https://events.static.linuxfound.org/sites/events/files/slides/How%20Bigtop%20Leveraged%20Docker%20for%20Build%20Automation%20and%20%20One-click%20Hadoop%20Provisioning%20ApacheBigData2015EU%20pub.pdf). ... and really just move away from using local builders of any other sort.... is that too much of a stretch?
>
>> On Jul 8, 2019, at 3:11 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
>>
>> Hi
>>
>> Right. This is automated compilation of protobuf is done while preparing the docker images.
>>
>> bigtop_toolchain/manifests/protobuf.pp
>>
>> I like to note we use this approach for all platforms for consistency.
>>
>> Best,
>> Olaf
>>
>>
>>> On 08.07.19 10:18, Zhenyu Zheng wrote:
>>> Hi Yuqi,
>>> Thanks for the reply, so can I understand that what bigtop did is similar
>>> to what was suggested in:
>>> https://groups.google.com/forum/#!topic/protobuf/fwLF5_t3q3U ? I mean,
>>> backport some patches
>>> to 2.5.0 and build a protobuf that can work on ARM64?
>>> BR,
>>> Kevin Zheng
>>>> On Mon, Jul 8, 2019 at 4:14 PM Yuqi Gu <yu...@linaro.org> wrote:
>>>> Hi Zhenyu Zheng,
>>>>
>>>> Pls try to build Hadoop in docker:
>>>> docker run -v `pwd`:/ws bigtop/slaves:1.4.0-ubuntu-16.04-aarch64 bash -l -c
>>>> 'cd /ws ; hadoop-pkg'
>>>>
>>>> The protobuf patches for Hadoop are list: Here
>>>> <
>>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/do-component-build#L45-L105
>>>>>
>>>>
>>>> BRs,
>>>> Yuqi
>>>>
>>>> On Mon, 8 Jul 2019 at 15:42, Zhenyu Zheng <zh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Bigtop:
>>>>>
>>>>> I'm trying to build Hadoop on ARM64 platform and I've meet some problem
>>>>> related to protobuf 2.5.0 seems it does not build on ARM64, and there is
>>>> an
>>>>> issue in Hadoop about this:
>>>>> https://issues.apache.org/jira/browse/HADOOP-13363 which has not yet
>>>> been
>>>>> solved. But I noticed that Bigtop has successfuly built Hadoop 2.8.5 on
>>>>> ARM64, eg.
>>>>>
>>>>>
>>>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=ubuntu-16.04-aarch64/
>>>>> so
>>>>> I'm quite curious how bigtop solved this problem?
>>>>>
>>>>> Thanks in advance
>>>>>
>>>>> Kevin Zheng
>>>>>
>>>>
>
Re: [Bigtop] Some question about build Hadoop on AArch64
Posted by Jay Vyas <ja...@gmail.com>.
Hi olaf- we do have containeirzed builders now, right? I might be wrong about this, as I haven’t built full packages for bigtop in the last 12 months or so, but I used to use docker... Am thinking we should update docs to reflect that to build bigtop it’s easier to use our docker containers (Evans gave a good presentation on this https://events.static.linuxfound.org/sites/events/files/slides/How%20Bigtop%20Leveraged%20Docker%20for%20Build%20Automation%20and%20%20One-click%20Hadoop%20Provisioning%20ApacheBigData2015EU%20pub.pdf). ... and really just move away from using local builders of any other sort.... is that too much of a stretch?
> On Jul 8, 2019, at 3:11 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
>
> Hi
>
> Right. This is automated compilation of protobuf is done while preparing the docker images.
>
> bigtop_toolchain/manifests/protobuf.pp
>
> I like to note we use this approach for all platforms for consistency.
>
> Best,
> Olaf
>
>
>> On 08.07.19 10:18, Zhenyu Zheng wrote:
>> Hi Yuqi,
>> Thanks for the reply, so can I understand that what bigtop did is similar
>> to what was suggested in:
>> https://groups.google.com/forum/#!topic/protobuf/fwLF5_t3q3U ? I mean,
>> backport some patches
>> to 2.5.0 and build a protobuf that can work on ARM64?
>> BR,
>> Kevin Zheng
>>> On Mon, Jul 8, 2019 at 4:14 PM Yuqi Gu <yu...@linaro.org> wrote:
>>> Hi Zhenyu Zheng,
>>>
>>> Pls try to build Hadoop in docker:
>>> docker run -v `pwd`:/ws bigtop/slaves:1.4.0-ubuntu-16.04-aarch64 bash -l -c
>>> 'cd /ws ; hadoop-pkg'
>>>
>>> The protobuf patches for Hadoop are list: Here
>>> <
>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/do-component-build#L45-L105
>>>>
>>>
>>> BRs,
>>> Yuqi
>>>
>>> On Mon, 8 Jul 2019 at 15:42, Zhenyu Zheng <zh...@gmail.com>
>>> wrote:
>>>
>>>> Hi Bigtop:
>>>>
>>>> I'm trying to build Hadoop on ARM64 platform and I've meet some problem
>>>> related to protobuf 2.5.0 seems it does not build on ARM64, and there is
>>> an
>>>> issue in Hadoop about this:
>>>> https://issues.apache.org/jira/browse/HADOOP-13363 which has not yet
>>> been
>>>> solved. But I noticed that Bigtop has successfuly built Hadoop 2.8.5 on
>>>> ARM64, eg.
>>>>
>>>>
>>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=ubuntu-16.04-aarch64/
>>>> so
>>>> I'm quite curious how bigtop solved this problem?
>>>>
>>>> Thanks in advance
>>>>
>>>> Kevin Zheng
>>>>
>>>
Re: [Bigtop] Some question about build Hadoop on AArch64
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi
Right. This is automated compilation of protobuf is done while preparing
the docker images.
bigtop_toolchain/manifests/protobuf.pp
I like to note we use this approach for all platforms for consistency.
Best,
Olaf
On 08.07.19 10:18, Zhenyu Zheng wrote:
> Hi Yuqi,
>
> Thanks for the reply, so can I understand that what bigtop did is similar
> to what was suggested in:
> https://groups.google.com/forum/#!topic/protobuf/fwLF5_t3q3U ? I mean,
> backport some patches
> to 2.5.0 and build a protobuf that can work on ARM64?
>
> BR,
>
> Kevin Zheng
>
> On Mon, Jul 8, 2019 at 4:14 PM Yuqi Gu <yu...@linaro.org> wrote:
>
>> Hi Zhenyu Zheng,
>>
>> Pls try to build Hadoop in docker:
>> docker run -v `pwd`:/ws bigtop/slaves:1.4.0-ubuntu-16.04-aarch64 bash -l -c
>> 'cd /ws ; hadoop-pkg'
>>
>> The protobuf patches for Hadoop are list: Here
>> <
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/do-component-build#L45-L105
>>>
>>
>> BRs,
>> Yuqi
>>
>> On Mon, 8 Jul 2019 at 15:42, Zhenyu Zheng <zh...@gmail.com>
>> wrote:
>>
>>> Hi Bigtop:
>>>
>>> I'm trying to build Hadoop on ARM64 platform and I've meet some problem
>>> related to protobuf 2.5.0 seems it does not build on ARM64, and there is
>> an
>>> issue in Hadoop about this:
>>> https://issues.apache.org/jira/browse/HADOOP-13363 which has not yet
>> been
>>> solved. But I noticed that Bigtop has successfuly built Hadoop 2.8.5 on
>>> ARM64, eg.
>>>
>>>
>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=ubuntu-16.04-aarch64/
>>> so
>>> I'm quite curious how bigtop solved this problem?
>>>
>>> Thanks in advance
>>>
>>> Kevin Zheng
>>>
>>
>
Re: [Bigtop] Some question about build Hadoop on AArch64
Posted by Yuqi Gu <yu...@linaro.org>.
Hi Kevin,
Protobuf-2.5 will be built in bigtop_toolchain.
See : BIGTOP-2698 <https://issues.apache.org/jira/browse/BIGTOP-2698>
Bigtop Hadoop debian package also depends on utils, groovy and jsvc:
docker run -v `pwd`:/ws bigtop/slaves:1.4.0-ubuntu-16.04-aarch64 bash -l -c
'cd /ws ; gradle bigtop-utils-deb bigtop-groovy-deb bigtop-jsvc-deb
hadoop-deb'
BRs,
Yuqi
On Mon, 8 Jul 2019 at 16:19, Zhenyu Zheng <zh...@gmail.com> wrote:
> Hi Yuqi,
>
> Thanks for the reply, so can I understand that what bigtop did is similar
> to what was suggested in:
> https://groups.google.com/forum/#!topic/protobuf/fwLF5_t3q3U ? I mean,
> backport some patches
> to 2.5.0 and build a protobuf that can work on ARM64?
>
> BR,
>
> Kevin Zheng
>
> On Mon, Jul 8, 2019 at 4:14 PM Yuqi Gu <yu...@linaro.org> wrote:
>
> > Hi Zhenyu Zheng,
> >
> > Pls try to build Hadoop in docker:
> > docker run -v `pwd`:/ws bigtop/slaves:1.4.0-ubuntu-16.04-aarch64 bash -l
> -c
> > 'cd /ws ; hadoop-pkg'
> >
> > The protobuf patches for Hadoop are list: Here
> > <
> >
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/do-component-build#L45-L105
> > >
> >
> > BRs,
> > Yuqi
> >
> > On Mon, 8 Jul 2019 at 15:42, Zhenyu Zheng <zh...@gmail.com>
> > wrote:
> >
> > > Hi Bigtop:
> > >
> > > I'm trying to build Hadoop on ARM64 platform and I've meet some problem
> > > related to protobuf 2.5.0 seems it does not build on ARM64, and there
> is
> > an
> > > issue in Hadoop about this:
> > > https://issues.apache.org/jira/browse/HADOOP-13363 which has not yet
> > been
> > > solved. But I noticed that Bigtop has successfuly built Hadoop 2.8.5 on
> > > ARM64, eg.
> > >
> > >
> >
> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=ubuntu-16.04-aarch64/
> > > so
> > > I'm quite curious how bigtop solved this problem?
> > >
> > > Thanks in advance
> > >
> > > Kevin Zheng
> > >
> >
>
Re: [Bigtop] Some question about build Hadoop on AArch64
Posted by Zhenyu Zheng <zh...@gmail.com>.
Hi Yuqi,
Thanks for the reply, so can I understand that what bigtop did is similar
to what was suggested in:
https://groups.google.com/forum/#!topic/protobuf/fwLF5_t3q3U ? I mean,
backport some patches
to 2.5.0 and build a protobuf that can work on ARM64?
BR,
Kevin Zheng
On Mon, Jul 8, 2019 at 4:14 PM Yuqi Gu <yu...@linaro.org> wrote:
> Hi Zhenyu Zheng,
>
> Pls try to build Hadoop in docker:
> docker run -v `pwd`:/ws bigtop/slaves:1.4.0-ubuntu-16.04-aarch64 bash -l -c
> 'cd /ws ; hadoop-pkg'
>
> The protobuf patches for Hadoop are list: Here
> <
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/do-component-build#L45-L105
> >
>
> BRs,
> Yuqi
>
> On Mon, 8 Jul 2019 at 15:42, Zhenyu Zheng <zh...@gmail.com>
> wrote:
>
> > Hi Bigtop:
> >
> > I'm trying to build Hadoop on ARM64 platform and I've meet some problem
> > related to protobuf 2.5.0 seems it does not build on ARM64, and there is
> an
> > issue in Hadoop about this:
> > https://issues.apache.org/jira/browse/HADOOP-13363 which has not yet
> been
> > solved. But I noticed that Bigtop has successfuly built Hadoop 2.8.5 on
> > ARM64, eg.
> >
> >
> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=ubuntu-16.04-aarch64/
> > so
> > I'm quite curious how bigtop solved this problem?
> >
> > Thanks in advance
> >
> > Kevin Zheng
> >
>
Re: [Bigtop] Some question about build Hadoop on AArch64
Posted by Yuqi Gu <yu...@linaro.org>.
Hi Zhenyu Zheng,
Pls try to build Hadoop in docker:
docker run -v `pwd`:/ws bigtop/slaves:1.4.0-ubuntu-16.04-aarch64 bash -l -c
'cd /ws ; hadoop-pkg'
The protobuf patches for Hadoop are list: Here
<https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/do-component-build#L45-L105>
BRs,
Yuqi
On Mon, 8 Jul 2019 at 15:42, Zhenyu Zheng <zh...@gmail.com> wrote:
> Hi Bigtop:
>
> I'm trying to build Hadoop on ARM64 platform and I've meet some problem
> related to protobuf 2.5.0 seems it does not build on ARM64, and there is an
> issue in Hadoop about this:
> https://issues.apache.org/jira/browse/HADOOP-13363 which has not yet been
> solved. But I noticed that Bigtop has successfuly built Hadoop 2.8.5 on
> ARM64, eg.
>
> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=ubuntu-16.04-aarch64/
> so
> I'm quite curious how bigtop solved this problem?
>
> Thanks in advance
>
> Kevin Zheng
>