[DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B <vi...@apache.org>]

Hi Folks,

ARM is becoming famous lately in its processing capability and has got the
potential to run Bigdata workloads.
Many users have been moving to ARM machines due to its low cost.

In the past there were attempts to compile Hadoop on ARM (Rasberry PI) for
experimental purposes. Today ARM architecture is taking some of the
serverside processing as well. So there will be/is a real need of Hadoop to
support ARM architecture as well.

There are bunch of users who are trying out building Hadoop on ARM, trying
to add ARM CI to hadoop and facing issues[1]. Also some

As of today, Hadoop does not compile on ARM due to below issues, found from
testing done in openlab in [2].

1. Protobuf :
-------------------
     Hadoop project (also some downstream projects) stuck to protobuf 2.5.0
version, due to backward compatibility reasons. Protobuf-2.5.0 is not being
maintained in the community. While protobuf 3.x is being actively adopted
widely, still protobuf 3.x provides wire compatibility for proto2 messages.
Due to some compilation issues in the generated java code, which can induce
problems in downstream. Due to this reason protobuf upgrade from 2.5.0 was
not taken up.
In 3.0.0 onwards, hadoop supports shading of libraries to avoid classpath
problem in downstream projects.
    There are patches available to fix compilation in Hadoop. But need to
find a way to upgrade protobuf to latest version and still maintain the
downstream's classpath using shading feature of Hadoop build.

     There is a Jira for protobuf upgrade[3] created even before shade
support was added to Hadoop. Now need to revisit the Jira and continue
explore possibilities.

2. leveldbjni:
---------------
    Current leveldbjni used in YARN doesnot support ARM architecture, need
to check whether any of the future versions support ARM and can hadoop
upgrade to that version.


3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
-------------------------
'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by default in
the maven repository. Workaround is to build it locally and keep in local
maven repository.
Need to check whether any future versions of 'protoc-gen-grpc-java' is
having ARM executable and whether hadoop-yarn-csi can upgrade it?


Once the compilation issues are solved, then there might be many native
code related issues due to different architectures.
So to explore everything, need to join hands together and proceed.


Let us discuss and check, whether any body else out there who also need the
support of Hadoop on ARM architectures and ready to lend their hands and
time in this work.


[1] https://issues.apache.org/jira/browse/HADOOP-16358
[2]
https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
[3] https://issues.apache.org/jira/browse/HADOOP-13363

-Vinay

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Zhenyu Zheng <zh...@gmail.com>]

Hi All,

Thanks Vinaya bring the whole thing up and everyone else for discussing,
providing thoughts and doing actuall coding.

I noticed that protobuf upgrading has been the hottest discussion point
about the whole thread and  Vinayakumar, Duo Zhang and Akira Ajisaka have
done alot about protocbuf and the progress seems well:
https://issues.apache.org/jira/browse/HADOOP-13363

During the same time, we have keep trying to build and test hadoop
components, we have been debugging for all the problems occurred, and
providing possible solutions like:
https://issues.apache.org/jira/browse/HADOOP-16614
we would be really apappreciated if some one could check on the probosed
solution and provide feedbacks and thoughts.

Currently, we mainly focused on YARN module and we have finishing testing
and debugging, we can now passing all tests with fixes, we also found root
causes and possible sulotions to some already known problems like:
https://issues.apache.org/jira/browse/YARN-9511

I think on some level this could prove that we are willing to maintain and
improve the stuff in long term.

*Depending on this, I would like to propose adding an ARM CI to Hadoop
community, we can first start as a periodic job, or maybe for only one
component(YARN for example). We will donate machines to the current CI
system and we will maintain it and keep solve problems. By adding a CI, it
will also allow others that are interested in this to help review and debug
problems occoured. We are also willing to provide machines for contibutors
that are intrested or want to debuging for ARM related issues.*

And another thing I want to add on is, I can see from the thread and
previous discussions in ML and Jira, not much people seems interested in
things like make Hadoop working on Aarch64 platform. Here I want to provide
some ideas that might make this more attractive. When speaking of ARM or
Aarch64, devices like raspberry pi may be the first thing that pop up to
our head, we might think 'oh, it could be running on raspberry pi,
interesting' and treated it as a toy or test and then we go back to our
normal life. But actually, we can do much more. As we all know, over 90
percent of the mobile device uses ARM architecture, and there are more and
more ARM base datacenter servers are begaining to show up at the market.
With the increasing computing capability of mobile devices and increasing
speed of wireless connection like 5G, we could imagine what kind of
possiblity could we have if we can have Hadoop running on both datacenter
and mobile devices at edge. I'm not a Hadoop expert but I can imagine that
there is a ton of possiblilities and opportunities for new bussiness models
using Hadoop in the future. Probably some of them will change our life.
Just my 2 cents.

I have also updated in the issue that I've originally submitted, please
feel free to leave comments:
https://issues.apache.org/jira/browse/HADOOP-16358

BR,


On Fri, Sep 6, 2019 at 6:11 AM Vinayakumar B <vi...@apache.org>
wrote:

> Thanks @Anu
>
> I understand the concern. I took it in different manner.
>
> Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
> changes as early as possible, its better to do it trunk itself.
>
> I was able to come to successfull attempt of upgrading protobuf as per
> suggestion of stack in Jira
> https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
>  .
>
> Have created the PR. Please review.  Changes looks huge, because all
> references of "com.google.protobuf" relocated to
> "o.a.h.shaded.com.google.protobuf".
> Otherwise changes are reasonable.
>
> This change is with still keeping the current 2.5.0 dependency, for
> downstream builds. So essentially nothing should be changed for downstreams.
>
> -Vinay
>
>
> On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer <ae...@cloudera.com>
> wrote:
>
>> Yes, I think that is what Sunil and I are trying to suggest; the complex
>> dependencies like Protobuf, if you do it in the trunk you have a better
>> change of getting it done. Otherwise, at merge point random downstream
>> applications which you have never heard of will object, and Hadoop
>> compatibility rules are very clear so you cannot fix it.
>>
>> With that said, even doing this in the trunk is complex; It might be
>> good for you to host a meeting and get some feedback. I have openly said it
>> is a great idea like "belling the cat", but the effort is in getting the
>> community to agree and align. Solve that, most of your technical problems
>> will be easier to solve.
>>
>> If you go into a branch, it might be that the community might forget
>> about your work; and when you come in to merge you will see issues which
>> you did not think about.
>>
>> So, Here is what would be great if you can make this happen; for ARM
>> work, get a list of dependencies that needed to be upgraded; see if you can
>> get the community aligned with this goal; since ARM might not be in the
>> target for many users. You need to convince users that even without ARM,
>> this is a great idea.
>>
>> If you like we can get together during one of the HDFS meetups hosted by
>> Wei-chiu.
>>
>> Thanks
>> Anu
>>
>>
>>
>> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
>> wrote:
>>
>>> Hi all,
>>>
>>> Thanks for the response.
>>> As I see, protobuf upgrade is long pending and most awaited one.
>>>
>>> @Sunil
>>> Protobuf upgrade looks to be a non-trivial task.
>>> Thanks @Duo Zhang for the suggestion of
>>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>>> problem
>>> of dependency on build environment.
>>> However more problem lies in upgrade protobuf without breaking the
>>> downstream builds.
>>> Reason is many downstream projects directly refers server specific jars
>>> and
>>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>>> dependency.
>>>
>>> There are some historical discussions and suggestions on
>>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>>> upgrade.
>>> Recommended path for solution is, try to upgrade protobuf using shading
>>> of
>>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>>> dependencies for downstreams.
>>> I am trying out ideas on this and if it gets completed within time, may
>>> be
>>> we can target trunk itself for the protobuf upgrade.
>>>
>>> separate branch idea is suggested for the overall ARM support including
>>> protobuf upgrade and other problems mentioned in the discussion above.
>>>
>>> I dont expect separate branch to have a huge changes, apart from bug
>>> fixes,
>>> since there are no separate features specific to ARM is being planned.
>>> So timely rebase of separate branch would reduce the overhead on branch
>>> review/merge task.
>>>
>>> Still, if the solution to protobuf upgrade winds up early, without any
>>> side
>>> effects, I am more than happy to land it in trunk itself.
>>>
>>> Thanks,
>>> Vinay
>>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:
>>>
>>> > Thanks Vinay for starting the thread.
>>> >
>>> > I agree to Anu's view point related to protobuf. And with the
>>> suggestion
>>> > pointed out by Duo Zhang, if we can make use
>>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>>> 3.0.0
>>> > of protobuf will also be more easier.
>>> >
>>> > However i think its better to do this effort in trunk itself.
>>> > In offline talks, few members were interested to start 3.3.0 release.
>>> And
>>> > given that happens soon, I feel its better
>>> > we do this task in trunk itself as branch diverge is very much
>>> possible.
>>> > And to bring to call a merge on such a big branch will be even more
>>> tough
>>> > task.
>>> >
>>> > my 2 cents.
>>> >
>>> > Thanks
>>> > Sunil
>>> >
>>> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
>>> > wrote:
>>> >
>>> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>>> >> generate
>>> >> the protobuf code. It will download the protoc binary from the maven
>>> >> central so we do not need to install protoc on the build machine any
>>> more.
>>> >>
>>> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>>> >>
>>> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been
>>> failling
>>> >> for
>>> >> > over 2 month related to the Protobuf problem .
>>> >> > According to the latest successful build log:
>>> >> >
>>> >>
>>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>>> >> > the
>>> >> > os version was ubuntu 14.04 and for the jobs that are failling now
>>> such
>>> >> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console
>>> ,
>>> >> > the os version is 18.04. I'm not very familiar with the version
>>> changing
>>> >> > for the jobs but I did a little search, according to:
>>> >> >
>>> >> >
>>> >>
>>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>>> >> > &
>>> >> >
>>> >> >
>>> >>
>>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>>> >> > it both said that the version of libprotc-dev and protobuf-compiler
>>> >> > available for ubuntu 18.04 is 3.0.0
>>> >> >
>>> >> >
>>> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
>>> wrote:
>>> >> >
>>> >> >> Thanx Vinay for the initiative, Makes sense to add support for
>>> >> different
>>> >> >> architectures.
>>> >> >>
>>> >> >> +1, for the branch idea.
>>> >> >> Good Luck!!!
>>> >> >>
>>> >> >> -Ayush
>>> >> >>
>>> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <palomino219@gmail.com
>>> >
>>> >> >> wrote:
>>> >> >> >
>>> >> >> > For HBase, we purged all the protobuf related things from the
>>> public
>>> >> >> API,
>>> >> >> > and then upgraded to a shaded and relocated version of protobuf.
>>> We
>>> >> have
>>> >> >> > created a repo for this:
>>> >> >> >
>>> >> >> > https://github.com/apache/hbase-thirdparty
>>> >> >> >
>>> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>>> >> jars,
>>> >> >> our
>>> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
>>> >> >> discuss
>>> >> >> > on how to deal with the upgrading of coprocessor. Glad to see
>>> that
>>> >> the
>>> >> >> > hadoop community is also willing to solve the problem.
>>> >> >> >
>>> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
>>> 上午1:23写道：
>>> >> >> >
>>> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is
>>> proving
>>> >> that
>>> >> >> >> Hadoop and the downstream projects work correctly after you
>>> upgrade
>>> >> >> core
>>> >> >> >> components like Protobuf.
>>> >> >> >> So while branching and working on a branch is easy, merging back
>>> >> after
>>> >> >> you
>>> >> >> >> upgrade some of these core components is insanely hard. You
>>> might
>>> >> want
>>> >> >> to
>>> >> >> >> make sure that community buys into upgrading these components
>>> in the
>>> >> >> trunk.
>>> >> >> >> That way we will get testing and downstream components will
>>> notice
>>> >> when
>>> >> >> >> things break.
>>> >> >> >>
>>> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a
>>> really
>>> >> long
>>> >> >> >> time; I have argued that 2.5 is out of support and we cannot
>>> stay on
>>> >> >> that
>>> >> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>>> >> code
>>> >> >> base.
>>> >> >> >> It has been rightly pointed to me that while all the arguments I
>>> >> make
>>> >> >> is
>>> >> >> >> correct; it is a very complicated task to upgrade Protobuf, and
>>> the
>>> >> >> worst
>>> >> >> >> part is we will not even know what breaks until downstream
>>> projects
>>> >> >> pick up
>>> >> >> >> these changes and work against us.
>>> >> >> >>
>>> >> >> >> If we work off the Hadoop version 3 — and assume that we have
>>> >> >> "shading" in
>>> >> >> >> place for all deployments; it might be possible to get there;
>>> still
>>> >> a
>>> >> >> >> daunting task.
>>> >> >> >>
>>> >> >> >> So best of luck with the branch approach — But please remember,
>>> >> Merging
>>> >> >> >> back will be hard, Just my 2 cents.
>>> >> >> >>
>>> >> >> >> — Anu
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>>> >> zhengzhenyulixi@gmail.com
>>> >> >> >
>>> >> >> >> wrote:
>>> >> >> >>
>>> >> >> >>> Hi,
>>> >> >> >>>
>>> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea.
>>> A
>>> >> >> separate
>>> >> >> >>> branch with it's own ARM CI seems a really good idea.
>>> >> >> >>> By doing this we won't break any of the undergoing development
>>> in
>>> >> >> trunk
>>> >> >> >> and
>>> >> >> >>> a CI can be a very good way to show what are the
>>> >> >> >>> current problems and what have been fixed, it will also
>>> provide a
>>> >> very
>>> >> >> >> good
>>> >> >> >>> view for contributors that are intrested to working on
>>> >> >> >>> this. We can finally merge back the branch to trunk until the
>>> >> >> community
>>> >> >> >>> thinks it is good enough and stable enough. We can donate
>>> >> >> >>> ARM machines to the existing CI system for the job.
>>> >> >> >>>
>>> >> >> >>> I wonder if this approch possible?
>>> >> >> >>>
>>> >> >> >>> BR,
>>> >> >> >>>
>>> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>>> >> liusheng2048@gmail.com>
>>> >> >> >>> wrote:
>>> >> >> >>>
>>> >> >> >>>> Hi,
>>> >> >> >>>>
>>> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>>> >> community
>>> >> >> >>>> mentioned by Vinay. I am working on building and
>>> >> >> >>>> testing Hadoop components on aarch64 server these days,
>>> besides
>>> >> the
>>> >> >> >>> missing
>>> >> >> >>>> dependices of ARM platform issues #1 #2 #3
>>> >> >> >>>> mentioned by Vinay, other similar issue has also be found,
>>> such as
>>> >> >> the
>>> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>>> >> >> >>>>
>>> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>>> >> hoped to
>>> >> >> >> add
>>> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
>>> >> >> >>>> sure about if there is any potential effect or confilict on
>>> the
>>> >> trunk
>>> >> >> >>>> branch, so maybe creating a ARM specific branch for doing
>>> these
>>> >> stuff
>>> >> >> >>>> is a better choice, what do you think?
>>> >> >> >>>>
>>> >> >> >>>> Hope to hear thoughts from you :)
>>> >> >> >>>>
>>> >> >> >>>> BR,
>>> >> >> >>>> Liu sheng
>>> >> >> >>>>
>>> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二
>>> 上午5:34写道：
>>> >> >> >>>>
>>> >> >> >>>>> Hi Folks,
>>> >> >> >>>>>
>>> >> >> >>>>> ARM is becoming famous lately in its processing capability
>>> and
>>> >> has
>>> >> >> >> got
>>> >> >> >>>> the
>>> >> >> >>>>> potential to run Bigdata workloads.
>>> >> >> >>>>> Many users have been moving to ARM machines due to its low
>>> cost.
>>> >> >> >>>>>
>>> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>>> >> (Rasberry
>>> >> >> >> PI)
>>> >> >> >>>> for
>>> >> >> >>>>> experimental purposes. Today ARM architecture is taking some
>>> of
>>> >> the
>>> >> >> >>>>> serverside processing as well. So there will be/is a real
>>> need of
>>> >> >> >>> Hadoop
>>> >> >> >>>> to
>>> >> >> >>>>> support ARM architecture as well.
>>> >> >> >>>>>
>>> >> >> >>>>> There are bunch of users who are trying out building Hadoop
>>> on
>>> >> ARM,
>>> >> >> >>>> trying
>>> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>>> >> >> >>>>>
>>> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
>>> issues,
>>> >> >> >> found
>>> >> >> >>>> from
>>> >> >> >>>>> testing done in openlab in [2].
>>> >> >> >>>>>
>>> >> >> >>>>> 1. Protobuf :
>>> >> >> >>>>> -------------------
>>> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>>> >> protobuf
>>> >> >> >>>> 2.5.0
>>> >> >> >>>>> version, due to backward compatibility reasons.
>>> Protobuf-2.5.0 is
>>> >> >> not
>>> >> >> >>>> being
>>> >> >> >>>>> maintained in the community. While protobuf 3.x is being
>>> actively
>>> >> >> >>> adopted
>>> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
>>> proto2
>>> >> >> >>>> messages.
>>> >> >> >>>>> Due to some compilation issues in the generated java code,
>>> which
>>> >> can
>>> >> >> >>>> induce
>>> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
>>> from
>>> >> >> >> 2.5.0
>>> >> >> >>>> was
>>> >> >> >>>>> not taken up.
>>> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to
>>> avoid
>>> >> >> >>> classpath
>>> >> >> >>>>> problem in downstream projects.
>>> >> >> >>>>>    There are patches available to fix compilation in Hadoop.
>>> But
>>> >> >> >> need
>>> >> >> >>> to
>>> >> >> >>>>> find a way to upgrade protobuf to latest version and still
>>> >> maintain
>>> >> >> >> the
>>> >> >> >>>>> downstream's classpath using shading feature of Hadoop build.
>>> >> >> >>>>>
>>> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even
>>> before
>>> >> >> >> shade
>>> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>>> >> >> >> continue
>>> >> >> >>>>> explore possibilities.
>>> >> >> >>>>>
>>> >> >> >>>>> 2. leveldbjni:
>>> >> >> >>>>> ---------------
>>> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>>> >> architecture,
>>> >> >> >>>> need
>>> >> >> >>>>> to check whether any of the future versions support ARM and
>>> can
>>> >> >> >> hadoop
>>> >> >> >>>>> upgrade to that version.
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>>> >> >> >>>>> -------------------------
>>> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM
>>> executable by
>>> >> >> >>> default
>>> >> >> >>>> in
>>> >> >> >>>>> the maven repository. Workaround is to build it locally and
>>> keep
>>> >> in
>>> >> >> >>> local
>>> >> >> >>>>> maven repository.
>>> >> >> >>>>> Need to check whether any future versions of
>>> >> 'protoc-gen-grpc-java'
>>> >> >> >> is
>>> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can
>>> upgrade it?
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> Once the compilation issues are solved, then there might be
>>> many
>>> >> >> >> native
>>> >> >> >>>>> code related issues due to different architectures.
>>> >> >> >>>>> So to explore everything, need to join hands together and
>>> >> proceed.
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> Let us discuss and check, whether any body else out there who
>>> >> also
>>> >> >> >> need
>>> >> >> >>>> the
>>> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend
>>> their
>>> >> hands
>>> >> >> >>> and
>>> >> >> >>>>> time in this work.
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>>> >> >> >>>>> [2]
>>> >> >> >>
>>> >> >>
>>> >>
>>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>>> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>>> >> >> >>>>>
>>> >> >> >>>>> -Vinay
>>> >> >> >>
>>> >> >>
>>> >> >
>>> >>
>>> >
>>>
>>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B <vi...@apache.org>]

Thanks Zhenyu Zheng for the offer to donate machines for ARM CI.

We will definitely make use of it. Let me check how we can set up Jenkins
job for ARM.

-Thanks
Vinay


On Tue, 22 Oct 2019, 1:04 pm Zhenyu Zheng, <zh...@gmail.com>
wrote:

> Hi All,
>
> Some updates for ARM CI, our team has succesfully donated ARM resources
> and setup an ARM CI for Apache Spark:
> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/
> it will set to periodic job and then PR trigger when we think it is stable
> enough.
>
> I really hope we can do the same for Hadoop.
>
> BR,
>
> On Fri, Sep 6, 2019 at 6:11 AM Vinayakumar B <vi...@apache.org>
> wrote:
>
>> Thanks @Anu
>>
>> I understand the concern. I took it in different manner.
>>
>> Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
>> changes as early as possible, its better to do it trunk itself.
>>
>> I was able to come to successfull attempt of upgrading protobuf as per
>> suggestion of stack in Jira
>> https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
>>  .
>>
>> Have created the PR. Please review.  Changes looks huge, because all
>> references of "com.google.protobuf" relocated to
>> "o.a.h.shaded.com.google.protobuf".
>> Otherwise changes are reasonable.
>>
>> This change is with still keeping the current 2.5.0 dependency, for
>> downstream builds. So essentially nothing should be changed for downstreams.
>>
>> -Vinay
>>
>>
>> On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer <ae...@cloudera.com>
>> wrote:
>>
>>> Yes, I think that is what Sunil and I are trying to suggest; the complex
>>> dependencies like Protobuf, if you do it in the trunk you have a better
>>> change of getting it done. Otherwise, at merge point random downstream
>>> applications which you have never heard of will object, and Hadoop
>>> compatibility rules are very clear so you cannot fix it.
>>>
>>> With that said, even doing this in the trunk is complex; It might be
>>> good for you to host a meeting and get some feedback. I have openly said it
>>> is a great idea like "belling the cat", but the effort is in getting the
>>> community to agree and align. Solve that, most of your technical problems
>>> will be easier to solve.
>>>
>>> If you go into a branch, it might be that the community might forget
>>> about your work; and when you come in to merge you will see issues which
>>> you did not think about.
>>>
>>> So, Here is what would be great if you can make this happen; for ARM
>>> work, get a list of dependencies that needed to be upgraded; see if you can
>>> get the community aligned with this goal; since ARM might not be in the
>>> target for many users. You need to convince users that even without ARM,
>>> this is a great idea.
>>>
>>> If you like we can get together during one of the HDFS meetups hosted by
>>> Wei-chiu.
>>>
>>> Thanks
>>> Anu
>>>
>>>
>>>
>>> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Thanks for the response.
>>>> As I see, protobuf upgrade is long pending and most awaited one.
>>>>
>>>> @Sunil
>>>> Protobuf upgrade looks to be a non-trivial task.
>>>> Thanks @Duo Zhang for the suggestion of
>>>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>>>> problem
>>>> of dependency on build environment.
>>>> However more problem lies in upgrade protobuf without breaking the
>>>> downstream builds.
>>>> Reason is many downstream projects directly refers server specific jars
>>>> and
>>>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>>>> dependency.
>>>>
>>>> There are some historical discussions and suggestions on
>>>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>>>> upgrade.
>>>> Recommended path for solution is, try to upgrade protobuf using shading
>>>> of
>>>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>>>> dependencies for downstreams.
>>>> I am trying out ideas on this and if it gets completed within time, may
>>>> be
>>>> we can target trunk itself for the protobuf upgrade.
>>>>
>>>> separate branch idea is suggested for the overall ARM support including
>>>> protobuf upgrade and other problems mentioned in the discussion above.
>>>>
>>>> I dont expect separate branch to have a huge changes, apart from bug
>>>> fixes,
>>>> since there are no separate features specific to ARM is being planned.
>>>> So timely rebase of separate branch would reduce the overhead on branch
>>>> review/merge task.
>>>>
>>>> Still, if the solution to protobuf upgrade winds up early, without any
>>>> side
>>>> effects, I am more than happy to land it in trunk itself.
>>>>
>>>> Thanks,
>>>> Vinay
>>>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org>
>>>> wrote:
>>>>
>>>> > Thanks Vinay for starting the thread.
>>>> >
>>>> > I agree to Anu's view point related to protobuf. And with the
>>>> suggestion
>>>> > pointed out by Duo Zhang, if we can make use
>>>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>>>> 3.0.0
>>>> > of protobuf will also be more easier.
>>>> >
>>>> > However i think its better to do this effort in trunk itself.
>>>> > In offline talks, few members were interested to start 3.3.0 release.
>>>> And
>>>> > given that happens soon, I feel its better
>>>> > we do this task in trunk itself as branch diverge is very much
>>>> possible.
>>>> > And to bring to call a merge on such a big branch will be even more
>>>> tough
>>>> > task.
>>>> >
>>>> > my 2 cents.
>>>> >
>>>> > Thanks
>>>> > Sunil
>>>> >
>>>> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
>>>> > wrote:
>>>> >
>>>> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>>>> >> generate
>>>> >> the protobuf code. It will download the protoc binary from the maven
>>>> >> central so we do not need to install protoc on the build machine any
>>>> more.
>>>> >>
>>>> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>>>> >>
>>>> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been
>>>> failling
>>>> >> for
>>>> >> > over 2 month related to the Protobuf problem .
>>>> >> > According to the latest successful build log:
>>>> >> >
>>>> >>
>>>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>>>> >> > the
>>>> >> > os version was ubuntu 14.04 and for the jobs that are failling now
>>>> such
>>>> >> > as:
>>>> https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
>>>> >> > the os version is 18.04. I'm not very familiar with the version
>>>> changing
>>>> >> > for the jobs but I did a little search, according to:
>>>> >> >
>>>> >> >
>>>> >>
>>>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>>>> >> > &
>>>> >> >
>>>> >> >
>>>> >>
>>>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>>>> >> > it both said that the version of libprotc-dev and protobuf-compiler
>>>> >> > available for ubuntu 18.04 is 3.0.0
>>>> >> >
>>>> >> >
>>>> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
>>>> wrote:
>>>> >> >
>>>> >> >> Thanx Vinay for the initiative, Makes sense to add support for
>>>> >> different
>>>> >> >> architectures.
>>>> >> >>
>>>> >> >> +1, for the branch idea.
>>>> >> >> Good Luck!!!
>>>> >> >>
>>>> >> >> -Ayush
>>>> >> >>
>>>> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <
>>>> palomino219@gmail.com>
>>>> >> >> wrote:
>>>> >> >> >
>>>> >> >> > For HBase, we purged all the protobuf related things from the
>>>> public
>>>> >> >> API,
>>>> >> >> > and then upgraded to a shaded and relocated version of
>>>> protobuf. We
>>>> >> have
>>>> >> >> > created a repo for this:
>>>> >> >> >
>>>> >> >> > https://github.com/apache/hbase-thirdparty
>>>> >> >> >
>>>> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>>>> >> jars,
>>>> >> >> our
>>>> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened
>>>> a
>>>> >> >> discuss
>>>> >> >> > on how to deal with the upgrading of coprocessor. Glad to see
>>>> that
>>>> >> the
>>>> >> >> > hadoop community is also willing to solve the problem.
>>>> >> >> >
>>>> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
>>>> 上午1:23写道：
>>>> >> >> >
>>>> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is
>>>> proving
>>>> >> that
>>>> >> >> >> Hadoop and the downstream projects work correctly after you
>>>> upgrade
>>>> >> >> core
>>>> >> >> >> components like Protobuf.
>>>> >> >> >> So while branching and working on a branch is easy, merging
>>>> back
>>>> >> after
>>>> >> >> you
>>>> >> >> >> upgrade some of these core components is insanely hard. You
>>>> might
>>>> >> want
>>>> >> >> to
>>>> >> >> >> make sure that community buys into upgrading these components
>>>> in the
>>>> >> >> trunk.
>>>> >> >> >> That way we will get testing and downstream components will
>>>> notice
>>>> >> when
>>>> >> >> >> things break.
>>>> >> >> >>
>>>> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a
>>>> really
>>>> >> long
>>>> >> >> >> time; I have argued that 2.5 is out of support and we cannot
>>>> stay on
>>>> >> >> that
>>>> >> >> >> branch forever; or we need to take ownership of the Protobuf
>>>> 2.5
>>>> >> code
>>>> >> >> base.
>>>> >> >> >> It has been rightly pointed to me that while all the arguments
>>>> I
>>>> >> make
>>>> >> >> is
>>>> >> >> >> correct; it is a very complicated task to upgrade Protobuf,
>>>> and the
>>>> >> >> worst
>>>> >> >> >> part is we will not even know what breaks until downstream
>>>> projects
>>>> >> >> pick up
>>>> >> >> >> these changes and work against us.
>>>> >> >> >>
>>>> >> >> >> If we work off the Hadoop version 3 — and assume that we have
>>>> >> >> "shading" in
>>>> >> >> >> place for all deployments; it might be possible to get there;
>>>> still
>>>> >> a
>>>> >> >> >> daunting task.
>>>> >> >> >>
>>>> >> >> >> So best of luck with the branch approach — But please remember,
>>>> >> Merging
>>>> >> >> >> back will be hard, Just my 2 cents.
>>>> >> >> >>
>>>> >> >> >> — Anu
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>>>> >> zhengzhenyulixi@gmail.com
>>>> >> >> >
>>>> >> >> >> wrote:
>>>> >> >> >>
>>>> >> >> >>> Hi,
>>>> >> >> >>>
>>>> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the
>>>> idea. A
>>>> >> >> separate
>>>> >> >> >>> branch with it's own ARM CI seems a really good idea.
>>>> >> >> >>> By doing this we won't break any of the undergoing
>>>> development in
>>>> >> >> trunk
>>>> >> >> >> and
>>>> >> >> >>> a CI can be a very good way to show what are the
>>>> >> >> >>> current problems and what have been fixed, it will also
>>>> provide a
>>>> >> very
>>>> >> >> >> good
>>>> >> >> >>> view for contributors that are intrested to working on
>>>> >> >> >>> this. We can finally merge back the branch to trunk until the
>>>> >> >> community
>>>> >> >> >>> thinks it is good enough and stable enough. We can donate
>>>> >> >> >>> ARM machines to the existing CI system for the job.
>>>> >> >> >>>
>>>> >> >> >>> I wonder if this approch possible?
>>>> >> >> >>>
>>>> >> >> >>> BR,
>>>> >> >> >>>
>>>> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>>>> >> liusheng2048@gmail.com>
>>>> >> >> >>> wrote:
>>>> >> >> >>>
>>>> >> >> >>>> Hi,
>>>> >> >> >>>>
>>>> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>>>> >> community
>>>> >> >> >>>> mentioned by Vinay. I am working on building and
>>>> >> >> >>>> testing Hadoop components on aarch64 server these days,
>>>> besides
>>>> >> the
>>>> >> >> >>> missing
>>>> >> >> >>>> dependices of ARM platform issues #1 #2 #3
>>>> >> >> >>>> mentioned by Vinay, other similar issue has also be found,
>>>> such as
>>>> >> >> the
>>>> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>>>> >> >> >>>>
>>>> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>>>> >> hoped to
>>>> >> >> >> add
>>>> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
>>>> >> >> >>>> sure about if there is any potential effect or confilict on
>>>> the
>>>> >> trunk
>>>> >> >> >>>> branch, so maybe creating a ARM specific branch for doing
>>>> these
>>>> >> stuff
>>>> >> >> >>>> is a better choice, what do you think?
>>>> >> >> >>>>
>>>> >> >> >>>> Hope to hear thoughts from you :)
>>>> >> >> >>>>
>>>> >> >> >>>> BR,
>>>> >> >> >>>> Liu sheng
>>>> >> >> >>>>
>>>> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二
>>>> 上午5:34写道：
>>>> >> >> >>>>
>>>> >> >> >>>>> Hi Folks,
>>>> >> >> >>>>>
>>>> >> >> >>>>> ARM is becoming famous lately in its processing capability
>>>> and
>>>> >> has
>>>> >> >> >> got
>>>> >> >> >>>> the
>>>> >> >> >>>>> potential to run Bigdata workloads.
>>>> >> >> >>>>> Many users have been moving to ARM machines due to its low
>>>> cost.
>>>> >> >> >>>>>
>>>> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>>>> >> (Rasberry
>>>> >> >> >> PI)
>>>> >> >> >>>> for
>>>> >> >> >>>>> experimental purposes. Today ARM architecture is taking
>>>> some of
>>>> >> the
>>>> >> >> >>>>> serverside processing as well. So there will be/is a real
>>>> need of
>>>> >> >> >>> Hadoop
>>>> >> >> >>>> to
>>>> >> >> >>>>> support ARM architecture as well.
>>>> >> >> >>>>>
>>>> >> >> >>>>> There are bunch of users who are trying out building Hadoop
>>>> on
>>>> >> ARM,
>>>> >> >> >>>> trying
>>>> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>>>> >> >> >>>>>
>>>> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
>>>> issues,
>>>> >> >> >> found
>>>> >> >> >>>> from
>>>> >> >> >>>>> testing done in openlab in [2].
>>>> >> >> >>>>>
>>>> >> >> >>>>> 1. Protobuf :
>>>> >> >> >>>>> -------------------
>>>> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>>>> >> protobuf
>>>> >> >> >>>> 2.5.0
>>>> >> >> >>>>> version, due to backward compatibility reasons.
>>>> Protobuf-2.5.0 is
>>>> >> >> not
>>>> >> >> >>>> being
>>>> >> >> >>>>> maintained in the community. While protobuf 3.x is being
>>>> actively
>>>> >> >> >>> adopted
>>>> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
>>>> proto2
>>>> >> >> >>>> messages.
>>>> >> >> >>>>> Due to some compilation issues in the generated java code,
>>>> which
>>>> >> can
>>>> >> >> >>>> induce
>>>> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
>>>> from
>>>> >> >> >> 2.5.0
>>>> >> >> >>>> was
>>>> >> >> >>>>> not taken up.
>>>> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to
>>>> avoid
>>>> >> >> >>> classpath
>>>> >> >> >>>>> problem in downstream projects.
>>>> >> >> >>>>>    There are patches available to fix compilation in
>>>> Hadoop. But
>>>> >> >> >> need
>>>> >> >> >>> to
>>>> >> >> >>>>> find a way to upgrade protobuf to latest version and still
>>>> >> maintain
>>>> >> >> >> the
>>>> >> >> >>>>> downstream's classpath using shading feature of Hadoop
>>>> build.
>>>> >> >> >>>>>
>>>> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even
>>>> before
>>>> >> >> >> shade
>>>> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira
>>>> and
>>>> >> >> >> continue
>>>> >> >> >>>>> explore possibilities.
>>>> >> >> >>>>>
>>>> >> >> >>>>> 2. leveldbjni:
>>>> >> >> >>>>> ---------------
>>>> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>>>> >> architecture,
>>>> >> >> >>>> need
>>>> >> >> >>>>> to check whether any of the future versions support ARM and
>>>> can
>>>> >> >> >> hadoop
>>>> >> >> >>>>> upgrade to that version.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> 3. hadoop-yarn-csi's dependency
>>>> 'protoc-gen-grpc-java:1.15.1'
>>>> >> >> >>>>> -------------------------
>>>> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM
>>>> executable by
>>>> >> >> >>> default
>>>> >> >> >>>> in
>>>> >> >> >>>>> the maven repository. Workaround is to build it locally and
>>>> keep
>>>> >> in
>>>> >> >> >>> local
>>>> >> >> >>>>> maven repository.
>>>> >> >> >>>>> Need to check whether any future versions of
>>>> >> 'protoc-gen-grpc-java'
>>>> >> >> >> is
>>>> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can
>>>> upgrade it?
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> Once the compilation issues are solved, then there might be
>>>> many
>>>> >> >> >> native
>>>> >> >> >>>>> code related issues due to different architectures.
>>>> >> >> >>>>> So to explore everything, need to join hands together and
>>>> >> proceed.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> Let us discuss and check, whether any body else out there
>>>> who
>>>> >> also
>>>> >> >> >> need
>>>> >> >> >>>> the
>>>> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend
>>>> their
>>>> >> hands
>>>> >> >> >>> and
>>>> >> >> >>>>> time in this work.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>>>> >> >> >>>>> [2]
>>>> >> >> >>
>>>> >> >>
>>>> >>
>>>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>>>> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>>>> >> >> >>>>>
>>>> >> >> >>>>> -Vinay
>>>> >> >> >>
>>>> >> >>
>>>> >> >
>>>> >>
>>>> >
>>>>
>>>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B <vi...@apache.org>]

Thanks Zhenyu Zheng for the offer to donate machines for ARM CI.

We will definitely make use of it. Let me check how we can set up Jenkins
job for ARM.

-Thanks
Vinay


On Tue, 22 Oct 2019, 1:04 pm Zhenyu Zheng, <zh...@gmail.com>
wrote:

> Hi All,
>
> Some updates for ARM CI, our team has succesfully donated ARM resources
> and setup an ARM CI for Apache Spark:
> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/
> it will set to periodic job and then PR trigger when we think it is stable
> enough.
>
> I really hope we can do the same for Hadoop.
>
> BR,
>
> On Fri, Sep 6, 2019 at 6:11 AM Vinayakumar B <vi...@apache.org>
> wrote:
>
>> Thanks @Anu
>>
>> I understand the concern. I took it in different manner.
>>
>> Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
>> changes as early as possible, its better to do it trunk itself.
>>
>> I was able to come to successfull attempt of upgrading protobuf as per
>> suggestion of stack in Jira
>> https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
>>  .
>>
>> Have created the PR. Please review.  Changes looks huge, because all
>> references of "com.google.protobuf" relocated to
>> "o.a.h.shaded.com.google.protobuf".
>> Otherwise changes are reasonable.
>>
>> This change is with still keeping the current 2.5.0 dependency, for
>> downstream builds. So essentially nothing should be changed for downstreams.
>>
>> -Vinay
>>
>>
>> On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer <ae...@cloudera.com>
>> wrote:
>>
>>> Yes, I think that is what Sunil and I are trying to suggest; the complex
>>> dependencies like Protobuf, if you do it in the trunk you have a better
>>> change of getting it done. Otherwise, at merge point random downstream
>>> applications which you have never heard of will object, and Hadoop
>>> compatibility rules are very clear so you cannot fix it.
>>>
>>> With that said, even doing this in the trunk is complex; It might be
>>> good for you to host a meeting and get some feedback. I have openly said it
>>> is a great idea like "belling the cat", but the effort is in getting the
>>> community to agree and align. Solve that, most of your technical problems
>>> will be easier to solve.
>>>
>>> If you go into a branch, it might be that the community might forget
>>> about your work; and when you come in to merge you will see issues which
>>> you did not think about.
>>>
>>> So, Here is what would be great if you can make this happen; for ARM
>>> work, get a list of dependencies that needed to be upgraded; see if you can
>>> get the community aligned with this goal; since ARM might not be in the
>>> target for many users. You need to convince users that even without ARM,
>>> this is a great idea.
>>>
>>> If you like we can get together during one of the HDFS meetups hosted by
>>> Wei-chiu.
>>>
>>> Thanks
>>> Anu
>>>
>>>
>>>
>>> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Thanks for the response.
>>>> As I see, protobuf upgrade is long pending and most awaited one.
>>>>
>>>> @Sunil
>>>> Protobuf upgrade looks to be a non-trivial task.
>>>> Thanks @Duo Zhang for the suggestion of
>>>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>>>> problem
>>>> of dependency on build environment.
>>>> However more problem lies in upgrade protobuf without breaking the
>>>> downstream builds.
>>>> Reason is many downstream projects directly refers server specific jars
>>>> and
>>>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>>>> dependency.
>>>>
>>>> There are some historical discussions and suggestions on
>>>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>>>> upgrade.
>>>> Recommended path for solution is, try to upgrade protobuf using shading
>>>> of
>>>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>>>> dependencies for downstreams.
>>>> I am trying out ideas on this and if it gets completed within time, may
>>>> be
>>>> we can target trunk itself for the protobuf upgrade.
>>>>
>>>> separate branch idea is suggested for the overall ARM support including
>>>> protobuf upgrade and other problems mentioned in the discussion above.
>>>>
>>>> I dont expect separate branch to have a huge changes, apart from bug
>>>> fixes,
>>>> since there are no separate features specific to ARM is being planned.
>>>> So timely rebase of separate branch would reduce the overhead on branch
>>>> review/merge task.
>>>>
>>>> Still, if the solution to protobuf upgrade winds up early, without any
>>>> side
>>>> effects, I am more than happy to land it in trunk itself.
>>>>
>>>> Thanks,
>>>> Vinay
>>>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org>
>>>> wrote:
>>>>
>>>> > Thanks Vinay for starting the thread.
>>>> >
>>>> > I agree to Anu's view point related to protobuf. And with the
>>>> suggestion
>>>> > pointed out by Duo Zhang, if we can make use
>>>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>>>> 3.0.0
>>>> > of protobuf will also be more easier.
>>>> >
>>>> > However i think its better to do this effort in trunk itself.
>>>> > In offline talks, few members were interested to start 3.3.0 release.
>>>> And
>>>> > given that happens soon, I feel its better
>>>> > we do this task in trunk itself as branch diverge is very much
>>>> possible.
>>>> > And to bring to call a merge on such a big branch will be even more
>>>> tough
>>>> > task.
>>>> >
>>>> > my 2 cents.
>>>> >
>>>> > Thanks
>>>> > Sunil
>>>> >
>>>> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
>>>> > wrote:
>>>> >
>>>> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>>>> >> generate
>>>> >> the protobuf code. It will download the protoc binary from the maven
>>>> >> central so we do not need to install protoc on the build machine any
>>>> more.
>>>> >>
>>>> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>>>> >>
>>>> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been
>>>> failling
>>>> >> for
>>>> >> > over 2 month related to the Protobuf problem .
>>>> >> > According to the latest successful build log:
>>>> >> >
>>>> >>
>>>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>>>> >> > the
>>>> >> > os version was ubuntu 14.04 and for the jobs that are failling now
>>>> such
>>>> >> > as:
>>>> https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
>>>> >> > the os version is 18.04. I'm not very familiar with the version
>>>> changing
>>>> >> > for the jobs but I did a little search, according to:
>>>> >> >
>>>> >> >
>>>> >>
>>>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>>>> >> > &
>>>> >> >
>>>> >> >
>>>> >>
>>>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>>>> >> > it both said that the version of libprotc-dev and protobuf-compiler
>>>> >> > available for ubuntu 18.04 is 3.0.0
>>>> >> >
>>>> >> >
>>>> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
>>>> wrote:
>>>> >> >
>>>> >> >> Thanx Vinay for the initiative, Makes sense to add support for
>>>> >> different
>>>> >> >> architectures.
>>>> >> >>
>>>> >> >> +1, for the branch idea.
>>>> >> >> Good Luck!!!
>>>> >> >>
>>>> >> >> -Ayush
>>>> >> >>
>>>> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <
>>>> palomino219@gmail.com>
>>>> >> >> wrote:
>>>> >> >> >
>>>> >> >> > For HBase, we purged all the protobuf related things from the
>>>> public
>>>> >> >> API,
>>>> >> >> > and then upgraded to a shaded and relocated version of
>>>> protobuf. We
>>>> >> have
>>>> >> >> > created a repo for this:
>>>> >> >> >
>>>> >> >> > https://github.com/apache/hbase-thirdparty
>>>> >> >> >
>>>> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>>>> >> jars,
>>>> >> >> our
>>>> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened
>>>> a
>>>> >> >> discuss
>>>> >> >> > on how to deal with the upgrading of coprocessor. Glad to see
>>>> that
>>>> >> the
>>>> >> >> > hadoop community is also willing to solve the problem.
>>>> >> >> >
>>>> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
>>>> 上午1:23写道：
>>>> >> >> >
>>>> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is
>>>> proving
>>>> >> that
>>>> >> >> >> Hadoop and the downstream projects work correctly after you
>>>> upgrade
>>>> >> >> core
>>>> >> >> >> components like Protobuf.
>>>> >> >> >> So while branching and working on a branch is easy, merging
>>>> back
>>>> >> after
>>>> >> >> you
>>>> >> >> >> upgrade some of these core components is insanely hard. You
>>>> might
>>>> >> want
>>>> >> >> to
>>>> >> >> >> make sure that community buys into upgrading these components
>>>> in the
>>>> >> >> trunk.
>>>> >> >> >> That way we will get testing and downstream components will
>>>> notice
>>>> >> when
>>>> >> >> >> things break.
>>>> >> >> >>
>>>> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a
>>>> really
>>>> >> long
>>>> >> >> >> time; I have argued that 2.5 is out of support and we cannot
>>>> stay on
>>>> >> >> that
>>>> >> >> >> branch forever; or we need to take ownership of the Protobuf
>>>> 2.5
>>>> >> code
>>>> >> >> base.
>>>> >> >> >> It has been rightly pointed to me that while all the arguments
>>>> I
>>>> >> make
>>>> >> >> is
>>>> >> >> >> correct; it is a very complicated task to upgrade Protobuf,
>>>> and the
>>>> >> >> worst
>>>> >> >> >> part is we will not even know what breaks until downstream
>>>> projects
>>>> >> >> pick up
>>>> >> >> >> these changes and work against us.
>>>> >> >> >>
>>>> >> >> >> If we work off the Hadoop version 3 — and assume that we have
>>>> >> >> "shading" in
>>>> >> >> >> place for all deployments; it might be possible to get there;
>>>> still
>>>> >> a
>>>> >> >> >> daunting task.
>>>> >> >> >>
>>>> >> >> >> So best of luck with the branch approach — But please remember,
>>>> >> Merging
>>>> >> >> >> back will be hard, Just my 2 cents.
>>>> >> >> >>
>>>> >> >> >> — Anu
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>>>> >> zhengzhenyulixi@gmail.com
>>>> >> >> >
>>>> >> >> >> wrote:
>>>> >> >> >>
>>>> >> >> >>> Hi,
>>>> >> >> >>>
>>>> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the
>>>> idea. A
>>>> >> >> separate
>>>> >> >> >>> branch with it's own ARM CI seems a really good idea.
>>>> >> >> >>> By doing this we won't break any of the undergoing
>>>> development in
>>>> >> >> trunk
>>>> >> >> >> and
>>>> >> >> >>> a CI can be a very good way to show what are the
>>>> >> >> >>> current problems and what have been fixed, it will also
>>>> provide a
>>>> >> very
>>>> >> >> >> good
>>>> >> >> >>> view for contributors that are intrested to working on
>>>> >> >> >>> this. We can finally merge back the branch to trunk until the
>>>> >> >> community
>>>> >> >> >>> thinks it is good enough and stable enough. We can donate
>>>> >> >> >>> ARM machines to the existing CI system for the job.
>>>> >> >> >>>
>>>> >> >> >>> I wonder if this approch possible?
>>>> >> >> >>>
>>>> >> >> >>> BR,
>>>> >> >> >>>
>>>> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>>>> >> liusheng2048@gmail.com>
>>>> >> >> >>> wrote:
>>>> >> >> >>>
>>>> >> >> >>>> Hi,
>>>> >> >> >>>>
>>>> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>>>> >> community
>>>> >> >> >>>> mentioned by Vinay. I am working on building and
>>>> >> >> >>>> testing Hadoop components on aarch64 server these days,
>>>> besides
>>>> >> the
>>>> >> >> >>> missing
>>>> >> >> >>>> dependices of ARM platform issues #1 #2 #3
>>>> >> >> >>>> mentioned by Vinay, other similar issue has also be found,
>>>> such as
>>>> >> >> the
>>>> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>>>> >> >> >>>>
>>>> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>>>> >> hoped to
>>>> >> >> >> add
>>>> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
>>>> >> >> >>>> sure about if there is any potential effect or confilict on
>>>> the
>>>> >> trunk
>>>> >> >> >>>> branch, so maybe creating a ARM specific branch for doing
>>>> these
>>>> >> stuff
>>>> >> >> >>>> is a better choice, what do you think?
>>>> >> >> >>>>
>>>> >> >> >>>> Hope to hear thoughts from you :)
>>>> >> >> >>>>
>>>> >> >> >>>> BR,
>>>> >> >> >>>> Liu sheng
>>>> >> >> >>>>
>>>> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二
>>>> 上午5:34写道：
>>>> >> >> >>>>
>>>> >> >> >>>>> Hi Folks,
>>>> >> >> >>>>>
>>>> >> >> >>>>> ARM is becoming famous lately in its processing capability
>>>> and
>>>> >> has
>>>> >> >> >> got
>>>> >> >> >>>> the
>>>> >> >> >>>>> potential to run Bigdata workloads.
>>>> >> >> >>>>> Many users have been moving to ARM machines due to its low
>>>> cost.
>>>> >> >> >>>>>
>>>> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>>>> >> (Rasberry
>>>> >> >> >> PI)
>>>> >> >> >>>> for
>>>> >> >> >>>>> experimental purposes. Today ARM architecture is taking
>>>> some of
>>>> >> the
>>>> >> >> >>>>> serverside processing as well. So there will be/is a real
>>>> need of
>>>> >> >> >>> Hadoop
>>>> >> >> >>>> to
>>>> >> >> >>>>> support ARM architecture as well.
>>>> >> >> >>>>>
>>>> >> >> >>>>> There are bunch of users who are trying out building Hadoop
>>>> on
>>>> >> ARM,
>>>> >> >> >>>> trying
>>>> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>>>> >> >> >>>>>
>>>> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
>>>> issues,
>>>> >> >> >> found
>>>> >> >> >>>> from
>>>> >> >> >>>>> testing done in openlab in [2].
>>>> >> >> >>>>>
>>>> >> >> >>>>> 1. Protobuf :
>>>> >> >> >>>>> -------------------
>>>> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>>>> >> protobuf
>>>> >> >> >>>> 2.5.0
>>>> >> >> >>>>> version, due to backward compatibility reasons.
>>>> Protobuf-2.5.0 is
>>>> >> >> not
>>>> >> >> >>>> being
>>>> >> >> >>>>> maintained in the community. While protobuf 3.x is being
>>>> actively
>>>> >> >> >>> adopted
>>>> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
>>>> proto2
>>>> >> >> >>>> messages.
>>>> >> >> >>>>> Due to some compilation issues in the generated java code,
>>>> which
>>>> >> can
>>>> >> >> >>>> induce
>>>> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
>>>> from
>>>> >> >> >> 2.5.0
>>>> >> >> >>>> was
>>>> >> >> >>>>> not taken up.
>>>> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to
>>>> avoid
>>>> >> >> >>> classpath
>>>> >> >> >>>>> problem in downstream projects.
>>>> >> >> >>>>>    There are patches available to fix compilation in
>>>> Hadoop. But
>>>> >> >> >> need
>>>> >> >> >>> to
>>>> >> >> >>>>> find a way to upgrade protobuf to latest version and still
>>>> >> maintain
>>>> >> >> >> the
>>>> >> >> >>>>> downstream's classpath using shading feature of Hadoop
>>>> build.
>>>> >> >> >>>>>
>>>> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even
>>>> before
>>>> >> >> >> shade
>>>> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira
>>>> and
>>>> >> >> >> continue
>>>> >> >> >>>>> explore possibilities.
>>>> >> >> >>>>>
>>>> >> >> >>>>> 2. leveldbjni:
>>>> >> >> >>>>> ---------------
>>>> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>>>> >> architecture,
>>>> >> >> >>>> need
>>>> >> >> >>>>> to check whether any of the future versions support ARM and
>>>> can
>>>> >> >> >> hadoop
>>>> >> >> >>>>> upgrade to that version.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> 3. hadoop-yarn-csi's dependency
>>>> 'protoc-gen-grpc-java:1.15.1'
>>>> >> >> >>>>> -------------------------
>>>> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM
>>>> executable by
>>>> >> >> >>> default
>>>> >> >> >>>> in
>>>> >> >> >>>>> the maven repository. Workaround is to build it locally and
>>>> keep
>>>> >> in
>>>> >> >> >>> local
>>>> >> >> >>>>> maven repository.
>>>> >> >> >>>>> Need to check whether any future versions of
>>>> >> 'protoc-gen-grpc-java'
>>>> >> >> >> is
>>>> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can
>>>> upgrade it?
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> Once the compilation issues are solved, then there might be
>>>> many
>>>> >> >> >> native
>>>> >> >> >>>>> code related issues due to different architectures.
>>>> >> >> >>>>> So to explore everything, need to join hands together and
>>>> >> proceed.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> Let us discuss and check, whether any body else out there
>>>> who
>>>> >> also
>>>> >> >> >> need
>>>> >> >> >>>> the
>>>> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend
>>>> their
>>>> >> hands
>>>> >> >> >>> and
>>>> >> >> >>>>> time in this work.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>>>> >> >> >>>>> [2]
>>>> >> >> >>
>>>> >> >>
>>>> >>
>>>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>>>> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>>>> >> >> >>>>>
>>>> >> >> >>>>> -Vinay
>>>> >> >> >>
>>>> >> >>
>>>> >> >
>>>> >>
>>>> >
>>>>
>>>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B <vi...@apache.org>]

Thanks Zhenyu Zheng for the offer to donate machines for ARM CI.

We will definitely make use of it. Let me check how we can set up Jenkins
job for ARM.

-Thanks
Vinay


On Tue, 22 Oct 2019, 1:04 pm Zhenyu Zheng, <zh...@gmail.com>
wrote:

> Hi All,
>
> Some updates for ARM CI, our team has succesfully donated ARM resources
> and setup an ARM CI for Apache Spark:
> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/
> it will set to periodic job and then PR trigger when we think it is stable
> enough.
>
> I really hope we can do the same for Hadoop.
>
> BR,
>
> On Fri, Sep 6, 2019 at 6:11 AM Vinayakumar B <vi...@apache.org>
> wrote:
>
>> Thanks @Anu
>>
>> I understand the concern. I took it in different manner.
>>
>> Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
>> changes as early as possible, its better to do it trunk itself.
>>
>> I was able to come to successfull attempt of upgrading protobuf as per
>> suggestion of stack in Jira
>> https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
>>  .
>>
>> Have created the PR. Please review.  Changes looks huge, because all
>> references of "com.google.protobuf" relocated to
>> "o.a.h.shaded.com.google.protobuf".
>> Otherwise changes are reasonable.
>>
>> This change is with still keeping the current 2.5.0 dependency, for
>> downstream builds. So essentially nothing should be changed for downstreams.
>>
>> -Vinay
>>
>>
>> On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer <ae...@cloudera.com>
>> wrote:
>>
>>> Yes, I think that is what Sunil and I are trying to suggest; the complex
>>> dependencies like Protobuf, if you do it in the trunk you have a better
>>> change of getting it done. Otherwise, at merge point random downstream
>>> applications which you have never heard of will object, and Hadoop
>>> compatibility rules are very clear so you cannot fix it.
>>>
>>> With that said, even doing this in the trunk is complex; It might be
>>> good for you to host a meeting and get some feedback. I have openly said it
>>> is a great idea like "belling the cat", but the effort is in getting the
>>> community to agree and align. Solve that, most of your technical problems
>>> will be easier to solve.
>>>
>>> If you go into a branch, it might be that the community might forget
>>> about your work; and when you come in to merge you will see issues which
>>> you did not think about.
>>>
>>> So, Here is what would be great if you can make this happen; for ARM
>>> work, get a list of dependencies that needed to be upgraded; see if you can
>>> get the community aligned with this goal; since ARM might not be in the
>>> target for many users. You need to convince users that even without ARM,
>>> this is a great idea.
>>>
>>> If you like we can get together during one of the HDFS meetups hosted by
>>> Wei-chiu.
>>>
>>> Thanks
>>> Anu
>>>
>>>
>>>
>>> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Thanks for the response.
>>>> As I see, protobuf upgrade is long pending and most awaited one.
>>>>
>>>> @Sunil
>>>> Protobuf upgrade looks to be a non-trivial task.
>>>> Thanks @Duo Zhang for the suggestion of
>>>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>>>> problem
>>>> of dependency on build environment.
>>>> However more problem lies in upgrade protobuf without breaking the
>>>> downstream builds.
>>>> Reason is many downstream projects directly refers server specific jars
>>>> and
>>>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>>>> dependency.
>>>>
>>>> There are some historical discussions and suggestions on
>>>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>>>> upgrade.
>>>> Recommended path for solution is, try to upgrade protobuf using shading
>>>> of
>>>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>>>> dependencies for downstreams.
>>>> I am trying out ideas on this and if it gets completed within time, may
>>>> be
>>>> we can target trunk itself for the protobuf upgrade.
>>>>
>>>> separate branch idea is suggested for the overall ARM support including
>>>> protobuf upgrade and other problems mentioned in the discussion above.
>>>>
>>>> I dont expect separate branch to have a huge changes, apart from bug
>>>> fixes,
>>>> since there are no separate features specific to ARM is being planned.
>>>> So timely rebase of separate branch would reduce the overhead on branch
>>>> review/merge task.
>>>>
>>>> Still, if the solution to protobuf upgrade winds up early, without any
>>>> side
>>>> effects, I am more than happy to land it in trunk itself.
>>>>
>>>> Thanks,
>>>> Vinay
>>>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org>
>>>> wrote:
>>>>
>>>> > Thanks Vinay for starting the thread.
>>>> >
>>>> > I agree to Anu's view point related to protobuf. And with the
>>>> suggestion
>>>> > pointed out by Duo Zhang, if we can make use
>>>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>>>> 3.0.0
>>>> > of protobuf will also be more easier.
>>>> >
>>>> > However i think its better to do this effort in trunk itself.
>>>> > In offline talks, few members were interested to start 3.3.0 release.
>>>> And
>>>> > given that happens soon, I feel its better
>>>> > we do this task in trunk itself as branch diverge is very much
>>>> possible.
>>>> > And to bring to call a merge on such a big branch will be even more
>>>> tough
>>>> > task.
>>>> >
>>>> > my 2 cents.
>>>> >
>>>> > Thanks
>>>> > Sunil
>>>> >
>>>> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
>>>> > wrote:
>>>> >
>>>> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>>>> >> generate
>>>> >> the protobuf code. It will download the protoc binary from the maven
>>>> >> central so we do not need to install protoc on the build machine any
>>>> more.
>>>> >>
>>>> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>>>> >>
>>>> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been
>>>> failling
>>>> >> for
>>>> >> > over 2 month related to the Protobuf problem .
>>>> >> > According to the latest successful build log:
>>>> >> >
>>>> >>
>>>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>>>> >> > the
>>>> >> > os version was ubuntu 14.04 and for the jobs that are failling now
>>>> such
>>>> >> > as:
>>>> https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
>>>> >> > the os version is 18.04. I'm not very familiar with the version
>>>> changing
>>>> >> > for the jobs but I did a little search, according to:
>>>> >> >
>>>> >> >
>>>> >>
>>>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>>>> >> > &
>>>> >> >
>>>> >> >
>>>> >>
>>>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>>>> >> > it both said that the version of libprotc-dev and protobuf-compiler
>>>> >> > available for ubuntu 18.04 is 3.0.0
>>>> >> >
>>>> >> >
>>>> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
>>>> wrote:
>>>> >> >
>>>> >> >> Thanx Vinay for the initiative, Makes sense to add support for
>>>> >> different
>>>> >> >> architectures.
>>>> >> >>
>>>> >> >> +1, for the branch idea.
>>>> >> >> Good Luck!!!
>>>> >> >>
>>>> >> >> -Ayush
>>>> >> >>
>>>> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <
>>>> palomino219@gmail.com>
>>>> >> >> wrote:
>>>> >> >> >
>>>> >> >> > For HBase, we purged all the protobuf related things from the
>>>> public
>>>> >> >> API,
>>>> >> >> > and then upgraded to a shaded and relocated version of
>>>> protobuf. We
>>>> >> have
>>>> >> >> > created a repo for this:
>>>> >> >> >
>>>> >> >> > https://github.com/apache/hbase-thirdparty
>>>> >> >> >
>>>> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>>>> >> jars,
>>>> >> >> our
>>>> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened
>>>> a
>>>> >> >> discuss
>>>> >> >> > on how to deal with the upgrading of coprocessor. Glad to see
>>>> that
>>>> >> the
>>>> >> >> > hadoop community is also willing to solve the problem.
>>>> >> >> >
>>>> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
>>>> 上午1:23写道：
>>>> >> >> >
>>>> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is
>>>> proving
>>>> >> that
>>>> >> >> >> Hadoop and the downstream projects work correctly after you
>>>> upgrade
>>>> >> >> core
>>>> >> >> >> components like Protobuf.
>>>> >> >> >> So while branching and working on a branch is easy, merging
>>>> back
>>>> >> after
>>>> >> >> you
>>>> >> >> >> upgrade some of these core components is insanely hard. You
>>>> might
>>>> >> want
>>>> >> >> to
>>>> >> >> >> make sure that community buys into upgrading these components
>>>> in the
>>>> >> >> trunk.
>>>> >> >> >> That way we will get testing and downstream components will
>>>> notice
>>>> >> when
>>>> >> >> >> things break.
>>>> >> >> >>
>>>> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a
>>>> really
>>>> >> long
>>>> >> >> >> time; I have argued that 2.5 is out of support and we cannot
>>>> stay on
>>>> >> >> that
>>>> >> >> >> branch forever; or we need to take ownership of the Protobuf
>>>> 2.5
>>>> >> code
>>>> >> >> base.
>>>> >> >> >> It has been rightly pointed to me that while all the arguments
>>>> I
>>>> >> make
>>>> >> >> is
>>>> >> >> >> correct; it is a very complicated task to upgrade Protobuf,
>>>> and the
>>>> >> >> worst
>>>> >> >> >> part is we will not even know what breaks until downstream
>>>> projects
>>>> >> >> pick up
>>>> >> >> >> these changes and work against us.
>>>> >> >> >>
>>>> >> >> >> If we work off the Hadoop version 3 — and assume that we have
>>>> >> >> "shading" in
>>>> >> >> >> place for all deployments; it might be possible to get there;
>>>> still
>>>> >> a
>>>> >> >> >> daunting task.
>>>> >> >> >>
>>>> >> >> >> So best of luck with the branch approach — But please remember,
>>>> >> Merging
>>>> >> >> >> back will be hard, Just my 2 cents.
>>>> >> >> >>
>>>> >> >> >> — Anu
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>>>> >> zhengzhenyulixi@gmail.com
>>>> >> >> >
>>>> >> >> >> wrote:
>>>> >> >> >>
>>>> >> >> >>> Hi,
>>>> >> >> >>>
>>>> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the
>>>> idea. A
>>>> >> >> separate
>>>> >> >> >>> branch with it's own ARM CI seems a really good idea.
>>>> >> >> >>> By doing this we won't break any of the undergoing
>>>> development in
>>>> >> >> trunk
>>>> >> >> >> and
>>>> >> >> >>> a CI can be a very good way to show what are the
>>>> >> >> >>> current problems and what have been fixed, it will also
>>>> provide a
>>>> >> very
>>>> >> >> >> good
>>>> >> >> >>> view for contributors that are intrested to working on
>>>> >> >> >>> this. We can finally merge back the branch to trunk until the
>>>> >> >> community
>>>> >> >> >>> thinks it is good enough and stable enough. We can donate
>>>> >> >> >>> ARM machines to the existing CI system for the job.
>>>> >> >> >>>
>>>> >> >> >>> I wonder if this approch possible?
>>>> >> >> >>>
>>>> >> >> >>> BR,
>>>> >> >> >>>
>>>> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>>>> >> liusheng2048@gmail.com>
>>>> >> >> >>> wrote:
>>>> >> >> >>>
>>>> >> >> >>>> Hi,
>>>> >> >> >>>>
>>>> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>>>> >> community
>>>> >> >> >>>> mentioned by Vinay. I am working on building and
>>>> >> >> >>>> testing Hadoop components on aarch64 server these days,
>>>> besides
>>>> >> the
>>>> >> >> >>> missing
>>>> >> >> >>>> dependices of ARM platform issues #1 #2 #3
>>>> >> >> >>>> mentioned by Vinay, other similar issue has also be found,
>>>> such as
>>>> >> >> the
>>>> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>>>> >> >> >>>>
>>>> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>>>> >> hoped to
>>>> >> >> >> add
>>>> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
>>>> >> >> >>>> sure about if there is any potential effect or confilict on
>>>> the
>>>> >> trunk
>>>> >> >> >>>> branch, so maybe creating a ARM specific branch for doing
>>>> these
>>>> >> stuff
>>>> >> >> >>>> is a better choice, what do you think?
>>>> >> >> >>>>
>>>> >> >> >>>> Hope to hear thoughts from you :)
>>>> >> >> >>>>
>>>> >> >> >>>> BR,
>>>> >> >> >>>> Liu sheng
>>>> >> >> >>>>
>>>> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二
>>>> 上午5:34写道：
>>>> >> >> >>>>
>>>> >> >> >>>>> Hi Folks,
>>>> >> >> >>>>>
>>>> >> >> >>>>> ARM is becoming famous lately in its processing capability
>>>> and
>>>> >> has
>>>> >> >> >> got
>>>> >> >> >>>> the
>>>> >> >> >>>>> potential to run Bigdata workloads.
>>>> >> >> >>>>> Many users have been moving to ARM machines due to its low
>>>> cost.
>>>> >> >> >>>>>
>>>> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>>>> >> (Rasberry
>>>> >> >> >> PI)
>>>> >> >> >>>> for
>>>> >> >> >>>>> experimental purposes. Today ARM architecture is taking
>>>> some of
>>>> >> the
>>>> >> >> >>>>> serverside processing as well. So there will be/is a real
>>>> need of
>>>> >> >> >>> Hadoop
>>>> >> >> >>>> to
>>>> >> >> >>>>> support ARM architecture as well.
>>>> >> >> >>>>>
>>>> >> >> >>>>> There are bunch of users who are trying out building Hadoop
>>>> on
>>>> >> ARM,
>>>> >> >> >>>> trying
>>>> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>>>> >> >> >>>>>
>>>> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
>>>> issues,
>>>> >> >> >> found
>>>> >> >> >>>> from
>>>> >> >> >>>>> testing done in openlab in [2].
>>>> >> >> >>>>>
>>>> >> >> >>>>> 1. Protobuf :
>>>> >> >> >>>>> -------------------
>>>> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>>>> >> protobuf
>>>> >> >> >>>> 2.5.0
>>>> >> >> >>>>> version, due to backward compatibility reasons.
>>>> Protobuf-2.5.0 is
>>>> >> >> not
>>>> >> >> >>>> being
>>>> >> >> >>>>> maintained in the community. While protobuf 3.x is being
>>>> actively
>>>> >> >> >>> adopted
>>>> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
>>>> proto2
>>>> >> >> >>>> messages.
>>>> >> >> >>>>> Due to some compilation issues in the generated java code,
>>>> which
>>>> >> can
>>>> >> >> >>>> induce
>>>> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
>>>> from
>>>> >> >> >> 2.5.0
>>>> >> >> >>>> was
>>>> >> >> >>>>> not taken up.
>>>> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to
>>>> avoid
>>>> >> >> >>> classpath
>>>> >> >> >>>>> problem in downstream projects.
>>>> >> >> >>>>>    There are patches available to fix compilation in
>>>> Hadoop. But
>>>> >> >> >> need
>>>> >> >> >>> to
>>>> >> >> >>>>> find a way to upgrade protobuf to latest version and still
>>>> >> maintain
>>>> >> >> >> the
>>>> >> >> >>>>> downstream's classpath using shading feature of Hadoop
>>>> build.
>>>> >> >> >>>>>
>>>> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even
>>>> before
>>>> >> >> >> shade
>>>> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira
>>>> and
>>>> >> >> >> continue
>>>> >> >> >>>>> explore possibilities.
>>>> >> >> >>>>>
>>>> >> >> >>>>> 2. leveldbjni:
>>>> >> >> >>>>> ---------------
>>>> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>>>> >> architecture,
>>>> >> >> >>>> need
>>>> >> >> >>>>> to check whether any of the future versions support ARM and
>>>> can
>>>> >> >> >> hadoop
>>>> >> >> >>>>> upgrade to that version.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> 3. hadoop-yarn-csi's dependency
>>>> 'protoc-gen-grpc-java:1.15.1'
>>>> >> >> >>>>> -------------------------
>>>> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM
>>>> executable by
>>>> >> >> >>> default
>>>> >> >> >>>> in
>>>> >> >> >>>>> the maven repository. Workaround is to build it locally and
>>>> keep
>>>> >> in
>>>> >> >> >>> local
>>>> >> >> >>>>> maven repository.
>>>> >> >> >>>>> Need to check whether any future versions of
>>>> >> 'protoc-gen-grpc-java'
>>>> >> >> >> is
>>>> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can
>>>> upgrade it?
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> Once the compilation issues are solved, then there might be
>>>> many
>>>> >> >> >> native
>>>> >> >> >>>>> code related issues due to different architectures.
>>>> >> >> >>>>> So to explore everything, need to join hands together and
>>>> >> proceed.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> Let us discuss and check, whether any body else out there
>>>> who
>>>> >> also
>>>> >> >> >> need
>>>> >> >> >>>> the
>>>> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend
>>>> their
>>>> >> hands
>>>> >> >> >>> and
>>>> >> >> >>>>> time in this work.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>>>> >> >> >>>>> [2]
>>>> >> >> >>
>>>> >> >>
>>>> >>
>>>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>>>> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>>>> >> >> >>>>>
>>>> >> >> >>>>> -Vinay
>>>> >> >> >>
>>>> >> >>
>>>> >> >
>>>> >>
>>>> >
>>>>
>>>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B <vi...@apache.org>]

Thanks Zhenyu Zheng for the offer to donate machines for ARM CI.

We will definitely make use of it. Let me check how we can set up Jenkins
job for ARM.

-Thanks
Vinay


On Tue, 22 Oct 2019, 1:04 pm Zhenyu Zheng, <zh...@gmail.com>
wrote:

> Hi All,
>
> Some updates for ARM CI, our team has succesfully donated ARM resources
> and setup an ARM CI for Apache Spark:
> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/
> it will set to periodic job and then PR trigger when we think it is stable
> enough.
>
> I really hope we can do the same for Hadoop.
>
> BR,
>
> On Fri, Sep 6, 2019 at 6:11 AM Vinayakumar B <vi...@apache.org>
> wrote:
>
>> Thanks @Anu
>>
>> I understand the concern. I took it in different manner.
>>
>> Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
>> changes as early as possible, its better to do it trunk itself.
>>
>> I was able to come to successfull attempt of upgrading protobuf as per
>> suggestion of stack in Jira
>> https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
>>  .
>>
>> Have created the PR. Please review.  Changes looks huge, because all
>> references of "com.google.protobuf" relocated to
>> "o.a.h.shaded.com.google.protobuf".
>> Otherwise changes are reasonable.
>>
>> This change is with still keeping the current 2.5.0 dependency, for
>> downstream builds. So essentially nothing should be changed for downstreams.
>>
>> -Vinay
>>
>>
>> On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer <ae...@cloudera.com>
>> wrote:
>>
>>> Yes, I think that is what Sunil and I are trying to suggest; the complex
>>> dependencies like Protobuf, if you do it in the trunk you have a better
>>> change of getting it done. Otherwise, at merge point random downstream
>>> applications which you have never heard of will object, and Hadoop
>>> compatibility rules are very clear so you cannot fix it.
>>>
>>> With that said, even doing this in the trunk is complex; It might be
>>> good for you to host a meeting and get some feedback. I have openly said it
>>> is a great idea like "belling the cat", but the effort is in getting the
>>> community to agree and align. Solve that, most of your technical problems
>>> will be easier to solve.
>>>
>>> If you go into a branch, it might be that the community might forget
>>> about your work; and when you come in to merge you will see issues which
>>> you did not think about.
>>>
>>> So, Here is what would be great if you can make this happen; for ARM
>>> work, get a list of dependencies that needed to be upgraded; see if you can
>>> get the community aligned with this goal; since ARM might not be in the
>>> target for many users. You need to convince users that even without ARM,
>>> this is a great idea.
>>>
>>> If you like we can get together during one of the HDFS meetups hosted by
>>> Wei-chiu.
>>>
>>> Thanks
>>> Anu
>>>
>>>
>>>
>>> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Thanks for the response.
>>>> As I see, protobuf upgrade is long pending and most awaited one.
>>>>
>>>> @Sunil
>>>> Protobuf upgrade looks to be a non-trivial task.
>>>> Thanks @Duo Zhang for the suggestion of
>>>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>>>> problem
>>>> of dependency on build environment.
>>>> However more problem lies in upgrade protobuf without breaking the
>>>> downstream builds.
>>>> Reason is many downstream projects directly refers server specific jars
>>>> and
>>>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>>>> dependency.
>>>>
>>>> There are some historical discussions and suggestions on
>>>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>>>> upgrade.
>>>> Recommended path for solution is, try to upgrade protobuf using shading
>>>> of
>>>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>>>> dependencies for downstreams.
>>>> I am trying out ideas on this and if it gets completed within time, may
>>>> be
>>>> we can target trunk itself for the protobuf upgrade.
>>>>
>>>> separate branch idea is suggested for the overall ARM support including
>>>> protobuf upgrade and other problems mentioned in the discussion above.
>>>>
>>>> I dont expect separate branch to have a huge changes, apart from bug
>>>> fixes,
>>>> since there are no separate features specific to ARM is being planned.
>>>> So timely rebase of separate branch would reduce the overhead on branch
>>>> review/merge task.
>>>>
>>>> Still, if the solution to protobuf upgrade winds up early, without any
>>>> side
>>>> effects, I am more than happy to land it in trunk itself.
>>>>
>>>> Thanks,
>>>> Vinay
>>>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org>
>>>> wrote:
>>>>
>>>> > Thanks Vinay for starting the thread.
>>>> >
>>>> > I agree to Anu's view point related to protobuf. And with the
>>>> suggestion
>>>> > pointed out by Duo Zhang, if we can make use
>>>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>>>> 3.0.0
>>>> > of protobuf will also be more easier.
>>>> >
>>>> > However i think its better to do this effort in trunk itself.
>>>> > In offline talks, few members were interested to start 3.3.0 release.
>>>> And
>>>> > given that happens soon, I feel its better
>>>> > we do this task in trunk itself as branch diverge is very much
>>>> possible.
>>>> > And to bring to call a merge on such a big branch will be even more
>>>> tough
>>>> > task.
>>>> >
>>>> > my 2 cents.
>>>> >
>>>> > Thanks
>>>> > Sunil
>>>> >
>>>> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
>>>> > wrote:
>>>> >
>>>> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>>>> >> generate
>>>> >> the protobuf code. It will download the protoc binary from the maven
>>>> >> central so we do not need to install protoc on the build machine any
>>>> more.
>>>> >>
>>>> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>>>> >>
>>>> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been
>>>> failling
>>>> >> for
>>>> >> > over 2 month related to the Protobuf problem .
>>>> >> > According to the latest successful build log:
>>>> >> >
>>>> >>
>>>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>>>> >> > the
>>>> >> > os version was ubuntu 14.04 and for the jobs that are failling now
>>>> such
>>>> >> > as:
>>>> https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
>>>> >> > the os version is 18.04. I'm not very familiar with the version
>>>> changing
>>>> >> > for the jobs but I did a little search, according to:
>>>> >> >
>>>> >> >
>>>> >>
>>>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>>>> >> > &
>>>> >> >
>>>> >> >
>>>> >>
>>>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>>>> >> > it both said that the version of libprotc-dev and protobuf-compiler
>>>> >> > available for ubuntu 18.04 is 3.0.0
>>>> >> >
>>>> >> >
>>>> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
>>>> wrote:
>>>> >> >
>>>> >> >> Thanx Vinay for the initiative, Makes sense to add support for
>>>> >> different
>>>> >> >> architectures.
>>>> >> >>
>>>> >> >> +1, for the branch idea.
>>>> >> >> Good Luck!!!
>>>> >> >>
>>>> >> >> -Ayush
>>>> >> >>
>>>> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <
>>>> palomino219@gmail.com>
>>>> >> >> wrote:
>>>> >> >> >
>>>> >> >> > For HBase, we purged all the protobuf related things from the
>>>> public
>>>> >> >> API,
>>>> >> >> > and then upgraded to a shaded and relocated version of
>>>> protobuf. We
>>>> >> have
>>>> >> >> > created a repo for this:
>>>> >> >> >
>>>> >> >> > https://github.com/apache/hbase-thirdparty
>>>> >> >> >
>>>> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>>>> >> jars,
>>>> >> >> our
>>>> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened
>>>> a
>>>> >> >> discuss
>>>> >> >> > on how to deal with the upgrading of coprocessor. Glad to see
>>>> that
>>>> >> the
>>>> >> >> > hadoop community is also willing to solve the problem.
>>>> >> >> >
>>>> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
>>>> 上午1:23写道：
>>>> >> >> >
>>>> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is
>>>> proving
>>>> >> that
>>>> >> >> >> Hadoop and the downstream projects work correctly after you
>>>> upgrade
>>>> >> >> core
>>>> >> >> >> components like Protobuf.
>>>> >> >> >> So while branching and working on a branch is easy, merging
>>>> back
>>>> >> after
>>>> >> >> you
>>>> >> >> >> upgrade some of these core components is insanely hard. You
>>>> might
>>>> >> want
>>>> >> >> to
>>>> >> >> >> make sure that community buys into upgrading these components
>>>> in the
>>>> >> >> trunk.
>>>> >> >> >> That way we will get testing and downstream components will
>>>> notice
>>>> >> when
>>>> >> >> >> things break.
>>>> >> >> >>
>>>> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a
>>>> really
>>>> >> long
>>>> >> >> >> time; I have argued that 2.5 is out of support and we cannot
>>>> stay on
>>>> >> >> that
>>>> >> >> >> branch forever; or we need to take ownership of the Protobuf
>>>> 2.5
>>>> >> code
>>>> >> >> base.
>>>> >> >> >> It has been rightly pointed to me that while all the arguments
>>>> I
>>>> >> make
>>>> >> >> is
>>>> >> >> >> correct; it is a very complicated task to upgrade Protobuf,
>>>> and the
>>>> >> >> worst
>>>> >> >> >> part is we will not even know what breaks until downstream
>>>> projects
>>>> >> >> pick up
>>>> >> >> >> these changes and work against us.
>>>> >> >> >>
>>>> >> >> >> If we work off the Hadoop version 3 — and assume that we have
>>>> >> >> "shading" in
>>>> >> >> >> place for all deployments; it might be possible to get there;
>>>> still
>>>> >> a
>>>> >> >> >> daunting task.
>>>> >> >> >>
>>>> >> >> >> So best of luck with the branch approach — But please remember,
>>>> >> Merging
>>>> >> >> >> back will be hard, Just my 2 cents.
>>>> >> >> >>
>>>> >> >> >> — Anu
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>>>> >> zhengzhenyulixi@gmail.com
>>>> >> >> >
>>>> >> >> >> wrote:
>>>> >> >> >>
>>>> >> >> >>> Hi,
>>>> >> >> >>>
>>>> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the
>>>> idea. A
>>>> >> >> separate
>>>> >> >> >>> branch with it's own ARM CI seems a really good idea.
>>>> >> >> >>> By doing this we won't break any of the undergoing
>>>> development in
>>>> >> >> trunk
>>>> >> >> >> and
>>>> >> >> >>> a CI can be a very good way to show what are the
>>>> >> >> >>> current problems and what have been fixed, it will also
>>>> provide a
>>>> >> very
>>>> >> >> >> good
>>>> >> >> >>> view for contributors that are intrested to working on
>>>> >> >> >>> this. We can finally merge back the branch to trunk until the
>>>> >> >> community
>>>> >> >> >>> thinks it is good enough and stable enough. We can donate
>>>> >> >> >>> ARM machines to the existing CI system for the job.
>>>> >> >> >>>
>>>> >> >> >>> I wonder if this approch possible?
>>>> >> >> >>>
>>>> >> >> >>> BR,
>>>> >> >> >>>
>>>> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>>>> >> liusheng2048@gmail.com>
>>>> >> >> >>> wrote:
>>>> >> >> >>>
>>>> >> >> >>>> Hi,
>>>> >> >> >>>>
>>>> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>>>> >> community
>>>> >> >> >>>> mentioned by Vinay. I am working on building and
>>>> >> >> >>>> testing Hadoop components on aarch64 server these days,
>>>> besides
>>>> >> the
>>>> >> >> >>> missing
>>>> >> >> >>>> dependices of ARM platform issues #1 #2 #3
>>>> >> >> >>>> mentioned by Vinay, other similar issue has also be found,
>>>> such as
>>>> >> >> the
>>>> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>>>> >> >> >>>>
>>>> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>>>> >> hoped to
>>>> >> >> >> add
>>>> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
>>>> >> >> >>>> sure about if there is any potential effect or confilict on
>>>> the
>>>> >> trunk
>>>> >> >> >>>> branch, so maybe creating a ARM specific branch for doing
>>>> these
>>>> >> stuff
>>>> >> >> >>>> is a better choice, what do you think?
>>>> >> >> >>>>
>>>> >> >> >>>> Hope to hear thoughts from you :)
>>>> >> >> >>>>
>>>> >> >> >>>> BR,
>>>> >> >> >>>> Liu sheng
>>>> >> >> >>>>
>>>> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二
>>>> 上午5:34写道：
>>>> >> >> >>>>
>>>> >> >> >>>>> Hi Folks,
>>>> >> >> >>>>>
>>>> >> >> >>>>> ARM is becoming famous lately in its processing capability
>>>> and
>>>> >> has
>>>> >> >> >> got
>>>> >> >> >>>> the
>>>> >> >> >>>>> potential to run Bigdata workloads.
>>>> >> >> >>>>> Many users have been moving to ARM machines due to its low
>>>> cost.
>>>> >> >> >>>>>
>>>> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>>>> >> (Rasberry
>>>> >> >> >> PI)
>>>> >> >> >>>> for
>>>> >> >> >>>>> experimental purposes. Today ARM architecture is taking
>>>> some of
>>>> >> the
>>>> >> >> >>>>> serverside processing as well. So there will be/is a real
>>>> need of
>>>> >> >> >>> Hadoop
>>>> >> >> >>>> to
>>>> >> >> >>>>> support ARM architecture as well.
>>>> >> >> >>>>>
>>>> >> >> >>>>> There are bunch of users who are trying out building Hadoop
>>>> on
>>>> >> ARM,
>>>> >> >> >>>> trying
>>>> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>>>> >> >> >>>>>
>>>> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
>>>> issues,
>>>> >> >> >> found
>>>> >> >> >>>> from
>>>> >> >> >>>>> testing done in openlab in [2].
>>>> >> >> >>>>>
>>>> >> >> >>>>> 1. Protobuf :
>>>> >> >> >>>>> -------------------
>>>> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>>>> >> protobuf
>>>> >> >> >>>> 2.5.0
>>>> >> >> >>>>> version, due to backward compatibility reasons.
>>>> Protobuf-2.5.0 is
>>>> >> >> not
>>>> >> >> >>>> being
>>>> >> >> >>>>> maintained in the community. While protobuf 3.x is being
>>>> actively
>>>> >> >> >>> adopted
>>>> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
>>>> proto2
>>>> >> >> >>>> messages.
>>>> >> >> >>>>> Due to some compilation issues in the generated java code,
>>>> which
>>>> >> can
>>>> >> >> >>>> induce
>>>> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
>>>> from
>>>> >> >> >> 2.5.0
>>>> >> >> >>>> was
>>>> >> >> >>>>> not taken up.
>>>> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to
>>>> avoid
>>>> >> >> >>> classpath
>>>> >> >> >>>>> problem in downstream projects.
>>>> >> >> >>>>>    There are patches available to fix compilation in
>>>> Hadoop. But
>>>> >> >> >> need
>>>> >> >> >>> to
>>>> >> >> >>>>> find a way to upgrade protobuf to latest version and still
>>>> >> maintain
>>>> >> >> >> the
>>>> >> >> >>>>> downstream's classpath using shading feature of Hadoop
>>>> build.
>>>> >> >> >>>>>
>>>> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even
>>>> before
>>>> >> >> >> shade
>>>> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira
>>>> and
>>>> >> >> >> continue
>>>> >> >> >>>>> explore possibilities.
>>>> >> >> >>>>>
>>>> >> >> >>>>> 2. leveldbjni:
>>>> >> >> >>>>> ---------------
>>>> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>>>> >> architecture,
>>>> >> >> >>>> need
>>>> >> >> >>>>> to check whether any of the future versions support ARM and
>>>> can
>>>> >> >> >> hadoop
>>>> >> >> >>>>> upgrade to that version.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> 3. hadoop-yarn-csi's dependency
>>>> 'protoc-gen-grpc-java:1.15.1'
>>>> >> >> >>>>> -------------------------
>>>> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM
>>>> executable by
>>>> >> >> >>> default
>>>> >> >> >>>> in
>>>> >> >> >>>>> the maven repository. Workaround is to build it locally and
>>>> keep
>>>> >> in
>>>> >> >> >>> local
>>>> >> >> >>>>> maven repository.
>>>> >> >> >>>>> Need to check whether any future versions of
>>>> >> 'protoc-gen-grpc-java'
>>>> >> >> >> is
>>>> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can
>>>> upgrade it?
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> Once the compilation issues are solved, then there might be
>>>> many
>>>> >> >> >> native
>>>> >> >> >>>>> code related issues due to different architectures.
>>>> >> >> >>>>> So to explore everything, need to join hands together and
>>>> >> proceed.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> Let us discuss and check, whether any body else out there
>>>> who
>>>> >> also
>>>> >> >> >> need
>>>> >> >> >>>> the
>>>> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend
>>>> their
>>>> >> hands
>>>> >> >> >>> and
>>>> >> >> >>>>> time in this work.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>>>> >> >> >>>>> [2]
>>>> >> >> >>
>>>> >> >>
>>>> >>
>>>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>>>> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>>>> >> >> >>>>>
>>>> >> >> >>>>> -Vinay
>>>> >> >> >>
>>>> >> >>
>>>> >> >
>>>> >>
>>>> >
>>>>
>>>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Zhenyu Zheng <zh...@gmail.com>]

Hi All,

Some updates for ARM CI, our team has succesfully donated ARM resources and
setup an ARM CI for Apache Spark:
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/
it will set to periodic job and then PR trigger when we think it is stable
enough.

I really hope we can do the same for Hadoop.

BR,

On Fri, Sep 6, 2019 at 6:11 AM Vinayakumar B <vi...@apache.org>
wrote:

> Thanks @Anu
>
> I understand the concern. I took it in different manner.
>
> Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
> changes as early as possible, its better to do it trunk itself.
>
> I was able to come to successfull attempt of upgrading protobuf as per
> suggestion of stack in Jira
> https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
>  .
>
> Have created the PR. Please review.  Changes looks huge, because all
> references of "com.google.protobuf" relocated to
> "o.a.h.shaded.com.google.protobuf".
> Otherwise changes are reasonable.
>
> This change is with still keeping the current 2.5.0 dependency, for
> downstream builds. So essentially nothing should be changed for downstreams.
>
> -Vinay
>
>
> On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer <ae...@cloudera.com>
> wrote:
>
>> Yes, I think that is what Sunil and I are trying to suggest; the complex
>> dependencies like Protobuf, if you do it in the trunk you have a better
>> change of getting it done. Otherwise, at merge point random downstream
>> applications which you have never heard of will object, and Hadoop
>> compatibility rules are very clear so you cannot fix it.
>>
>> With that said, even doing this in the trunk is complex; It might be
>> good for you to host a meeting and get some feedback. I have openly said it
>> is a great idea like "belling the cat", but the effort is in getting the
>> community to agree and align. Solve that, most of your technical problems
>> will be easier to solve.
>>
>> If you go into a branch, it might be that the community might forget
>> about your work; and when you come in to merge you will see issues which
>> you did not think about.
>>
>> So, Here is what would be great if you can make this happen; for ARM
>> work, get a list of dependencies that needed to be upgraded; see if you can
>> get the community aligned with this goal; since ARM might not be in the
>> target for many users. You need to convince users that even without ARM,
>> this is a great idea.
>>
>> If you like we can get together during one of the HDFS meetups hosted by
>> Wei-chiu.
>>
>> Thanks
>> Anu
>>
>>
>>
>> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
>> wrote:
>>
>>> Hi all,
>>>
>>> Thanks for the response.
>>> As I see, protobuf upgrade is long pending and most awaited one.
>>>
>>> @Sunil
>>> Protobuf upgrade looks to be a non-trivial task.
>>> Thanks @Duo Zhang for the suggestion of
>>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>>> problem
>>> of dependency on build environment.
>>> However more problem lies in upgrade protobuf without breaking the
>>> downstream builds.
>>> Reason is many downstream projects directly refers server specific jars
>>> and
>>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>>> dependency.
>>>
>>> There are some historical discussions and suggestions on
>>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>>> upgrade.
>>> Recommended path for solution is, try to upgrade protobuf using shading
>>> of
>>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>>> dependencies for downstreams.
>>> I am trying out ideas on this and if it gets completed within time, may
>>> be
>>> we can target trunk itself for the protobuf upgrade.
>>>
>>> separate branch idea is suggested for the overall ARM support including
>>> protobuf upgrade and other problems mentioned in the discussion above.
>>>
>>> I dont expect separate branch to have a huge changes, apart from bug
>>> fixes,
>>> since there are no separate features specific to ARM is being planned.
>>> So timely rebase of separate branch would reduce the overhead on branch
>>> review/merge task.
>>>
>>> Still, if the solution to protobuf upgrade winds up early, without any
>>> side
>>> effects, I am more than happy to land it in trunk itself.
>>>
>>> Thanks,
>>> Vinay
>>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:
>>>
>>> > Thanks Vinay for starting the thread.
>>> >
>>> > I agree to Anu's view point related to protobuf. And with the
>>> suggestion
>>> > pointed out by Duo Zhang, if we can make use
>>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>>> 3.0.0
>>> > of protobuf will also be more easier.
>>> >
>>> > However i think its better to do this effort in trunk itself.
>>> > In offline talks, few members were interested to start 3.3.0 release.
>>> And
>>> > given that happens soon, I feel its better
>>> > we do this task in trunk itself as branch diverge is very much
>>> possible.
>>> > And to bring to call a merge on such a big branch will be even more
>>> tough
>>> > task.
>>> >
>>> > my 2 cents.
>>> >
>>> > Thanks
>>> > Sunil
>>> >
>>> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
>>> > wrote:
>>> >
>>> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>>> >> generate
>>> >> the protobuf code. It will download the protoc binary from the maven
>>> >> central so we do not need to install protoc on the build machine any
>>> more.
>>> >>
>>> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>>> >>
>>> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been
>>> failling
>>> >> for
>>> >> > over 2 month related to the Protobuf problem .
>>> >> > According to the latest successful build log:
>>> >> >
>>> >>
>>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>>> >> > the
>>> >> > os version was ubuntu 14.04 and for the jobs that are failling now
>>> such
>>> >> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console
>>> ,
>>> >> > the os version is 18.04. I'm not very familiar with the version
>>> changing
>>> >> > for the jobs but I did a little search, according to:
>>> >> >
>>> >> >
>>> >>
>>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>>> >> > &
>>> >> >
>>> >> >
>>> >>
>>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>>> >> > it both said that the version of libprotc-dev and protobuf-compiler
>>> >> > available for ubuntu 18.04 is 3.0.0
>>> >> >
>>> >> >
>>> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
>>> wrote:
>>> >> >
>>> >> >> Thanx Vinay for the initiative, Makes sense to add support for
>>> >> different
>>> >> >> architectures.
>>> >> >>
>>> >> >> +1, for the branch idea.
>>> >> >> Good Luck!!!
>>> >> >>
>>> >> >> -Ayush
>>> >> >>
>>> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <palomino219@gmail.com
>>> >
>>> >> >> wrote:
>>> >> >> >
>>> >> >> > For HBase, we purged all the protobuf related things from the
>>> public
>>> >> >> API,
>>> >> >> > and then upgraded to a shaded and relocated version of protobuf.
>>> We
>>> >> have
>>> >> >> > created a repo for this:
>>> >> >> >
>>> >> >> > https://github.com/apache/hbase-thirdparty
>>> >> >> >
>>> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>>> >> jars,
>>> >> >> our
>>> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
>>> >> >> discuss
>>> >> >> > on how to deal with the upgrading of coprocessor. Glad to see
>>> that
>>> >> the
>>> >> >> > hadoop community is also willing to solve the problem.
>>> >> >> >
>>> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
>>> 上午1:23写道：
>>> >> >> >
>>> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is
>>> proving
>>> >> that
>>> >> >> >> Hadoop and the downstream projects work correctly after you
>>> upgrade
>>> >> >> core
>>> >> >> >> components like Protobuf.
>>> >> >> >> So while branching and working on a branch is easy, merging back
>>> >> after
>>> >> >> you
>>> >> >> >> upgrade some of these core components is insanely hard. You
>>> might
>>> >> want
>>> >> >> to
>>> >> >> >> make sure that community buys into upgrading these components
>>> in the
>>> >> >> trunk.
>>> >> >> >> That way we will get testing and downstream components will
>>> notice
>>> >> when
>>> >> >> >> things break.
>>> >> >> >>
>>> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a
>>> really
>>> >> long
>>> >> >> >> time; I have argued that 2.5 is out of support and we cannot
>>> stay on
>>> >> >> that
>>> >> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>>> >> code
>>> >> >> base.
>>> >> >> >> It has been rightly pointed to me that while all the arguments I
>>> >> make
>>> >> >> is
>>> >> >> >> correct; it is a very complicated task to upgrade Protobuf, and
>>> the
>>> >> >> worst
>>> >> >> >> part is we will not even know what breaks until downstream
>>> projects
>>> >> >> pick up
>>> >> >> >> these changes and work against us.
>>> >> >> >>
>>> >> >> >> If we work off the Hadoop version 3 — and assume that we have
>>> >> >> "shading" in
>>> >> >> >> place for all deployments; it might be possible to get there;
>>> still
>>> >> a
>>> >> >> >> daunting task.
>>> >> >> >>
>>> >> >> >> So best of luck with the branch approach — But please remember,
>>> >> Merging
>>> >> >> >> back will be hard, Just my 2 cents.
>>> >> >> >>
>>> >> >> >> — Anu
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>>> >> zhengzhenyulixi@gmail.com
>>> >> >> >
>>> >> >> >> wrote:
>>> >> >> >>
>>> >> >> >>> Hi,
>>> >> >> >>>
>>> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea.
>>> A
>>> >> >> separate
>>> >> >> >>> branch with it's own ARM CI seems a really good idea.
>>> >> >> >>> By doing this we won't break any of the undergoing development
>>> in
>>> >> >> trunk
>>> >> >> >> and
>>> >> >> >>> a CI can be a very good way to show what are the
>>> >> >> >>> current problems and what have been fixed, it will also
>>> provide a
>>> >> very
>>> >> >> >> good
>>> >> >> >>> view for contributors that are intrested to working on
>>> >> >> >>> this. We can finally merge back the branch to trunk until the
>>> >> >> community
>>> >> >> >>> thinks it is good enough and stable enough. We can donate
>>> >> >> >>> ARM machines to the existing CI system for the job.
>>> >> >> >>>
>>> >> >> >>> I wonder if this approch possible?
>>> >> >> >>>
>>> >> >> >>> BR,
>>> >> >> >>>
>>> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>>> >> liusheng2048@gmail.com>
>>> >> >> >>> wrote:
>>> >> >> >>>
>>> >> >> >>>> Hi,
>>> >> >> >>>>
>>> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>>> >> community
>>> >> >> >>>> mentioned by Vinay. I am working on building and
>>> >> >> >>>> testing Hadoop components on aarch64 server these days,
>>> besides
>>> >> the
>>> >> >> >>> missing
>>> >> >> >>>> dependices of ARM platform issues #1 #2 #3
>>> >> >> >>>> mentioned by Vinay, other similar issue has also be found,
>>> such as
>>> >> >> the
>>> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>>> >> >> >>>>
>>> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>>> >> hoped to
>>> >> >> >> add
>>> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
>>> >> >> >>>> sure about if there is any potential effect or confilict on
>>> the
>>> >> trunk
>>> >> >> >>>> branch, so maybe creating a ARM specific branch for doing
>>> these
>>> >> stuff
>>> >> >> >>>> is a better choice, what do you think?
>>> >> >> >>>>
>>> >> >> >>>> Hope to hear thoughts from you :)
>>> >> >> >>>>
>>> >> >> >>>> BR,
>>> >> >> >>>> Liu sheng
>>> >> >> >>>>
>>> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二
>>> 上午5:34写道：
>>> >> >> >>>>
>>> >> >> >>>>> Hi Folks,
>>> >> >> >>>>>
>>> >> >> >>>>> ARM is becoming famous lately in its processing capability
>>> and
>>> >> has
>>> >> >> >> got
>>> >> >> >>>> the
>>> >> >> >>>>> potential to run Bigdata workloads.
>>> >> >> >>>>> Many users have been moving to ARM machines due to its low
>>> cost.
>>> >> >> >>>>>
>>> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>>> >> (Rasberry
>>> >> >> >> PI)
>>> >> >> >>>> for
>>> >> >> >>>>> experimental purposes. Today ARM architecture is taking some
>>> of
>>> >> the
>>> >> >> >>>>> serverside processing as well. So there will be/is a real
>>> need of
>>> >> >> >>> Hadoop
>>> >> >> >>>> to
>>> >> >> >>>>> support ARM architecture as well.
>>> >> >> >>>>>
>>> >> >> >>>>> There are bunch of users who are trying out building Hadoop
>>> on
>>> >> ARM,
>>> >> >> >>>> trying
>>> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>>> >> >> >>>>>
>>> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
>>> issues,
>>> >> >> >> found
>>> >> >> >>>> from
>>> >> >> >>>>> testing done in openlab in [2].
>>> >> >> >>>>>
>>> >> >> >>>>> 1. Protobuf :
>>> >> >> >>>>> -------------------
>>> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>>> >> protobuf
>>> >> >> >>>> 2.5.0
>>> >> >> >>>>> version, due to backward compatibility reasons.
>>> Protobuf-2.5.0 is
>>> >> >> not
>>> >> >> >>>> being
>>> >> >> >>>>> maintained in the community. While protobuf 3.x is being
>>> actively
>>> >> >> >>> adopted
>>> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
>>> proto2
>>> >> >> >>>> messages.
>>> >> >> >>>>> Due to some compilation issues in the generated java code,
>>> which
>>> >> can
>>> >> >> >>>> induce
>>> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
>>> from
>>> >> >> >> 2.5.0
>>> >> >> >>>> was
>>> >> >> >>>>> not taken up.
>>> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to
>>> avoid
>>> >> >> >>> classpath
>>> >> >> >>>>> problem in downstream projects.
>>> >> >> >>>>>    There are patches available to fix compilation in Hadoop.
>>> But
>>> >> >> >> need
>>> >> >> >>> to
>>> >> >> >>>>> find a way to upgrade protobuf to latest version and still
>>> >> maintain
>>> >> >> >> the
>>> >> >> >>>>> downstream's classpath using shading feature of Hadoop build.
>>> >> >> >>>>>
>>> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even
>>> before
>>> >> >> >> shade
>>> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>>> >> >> >> continue
>>> >> >> >>>>> explore possibilities.
>>> >> >> >>>>>
>>> >> >> >>>>> 2. leveldbjni:
>>> >> >> >>>>> ---------------
>>> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>>> >> architecture,
>>> >> >> >>>> need
>>> >> >> >>>>> to check whether any of the future versions support ARM and
>>> can
>>> >> >> >> hadoop
>>> >> >> >>>>> upgrade to that version.
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>>> >> >> >>>>> -------------------------
>>> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM
>>> executable by
>>> >> >> >>> default
>>> >> >> >>>> in
>>> >> >> >>>>> the maven repository. Workaround is to build it locally and
>>> keep
>>> >> in
>>> >> >> >>> local
>>> >> >> >>>>> maven repository.
>>> >> >> >>>>> Need to check whether any future versions of
>>> >> 'protoc-gen-grpc-java'
>>> >> >> >> is
>>> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can
>>> upgrade it?
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> Once the compilation issues are solved, then there might be
>>> many
>>> >> >> >> native
>>> >> >> >>>>> code related issues due to different architectures.
>>> >> >> >>>>> So to explore everything, need to join hands together and
>>> >> proceed.
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> Let us discuss and check, whether any body else out there who
>>> >> also
>>> >> >> >> need
>>> >> >> >>>> the
>>> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend
>>> their
>>> >> hands
>>> >> >> >>> and
>>> >> >> >>>>> time in this work.
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>>> >> >> >>>>> [2]
>>> >> >> >>
>>> >> >>
>>> >>
>>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>>> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>>> >> >> >>>>>
>>> >> >> >>>>> -Vinay
>>> >> >> >>
>>> >> >>
>>> >> >
>>> >>
>>> >
>>>
>>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Zhenyu Zheng <zh...@gmail.com>]

Hi All,

Thanks Vinaya bring the whole thing up and everyone else for discussing,
providing thoughts and doing actuall coding.

I noticed that protobuf upgrading has been the hottest discussion point
about the whole thread and  Vinayakumar, Duo Zhang and Akira Ajisaka have
done alot about protocbuf and the progress seems well:
https://issues.apache.org/jira/browse/HADOOP-13363

During the same time, we have keep trying to build and test hadoop
components, we have been debugging for all the problems occurred, and
providing possible solutions like:
https://issues.apache.org/jira/browse/HADOOP-16614
we would be really apappreciated if some one could check on the probosed
solution and provide feedbacks and thoughts.

Currently, we mainly focused on YARN module and we have finishing testing
and debugging, we can now passing all tests with fixes, we also found root
causes and possible sulotions to some already known problems like:
https://issues.apache.org/jira/browse/YARN-9511

I think on some level this could prove that we are willing to maintain and
improve the stuff in long term.

*Depending on this, I would like to propose adding an ARM CI to Hadoop
community, we can first start as a periodic job, or maybe for only one
component(YARN for example). We will donate machines to the current CI
system and we will maintain it and keep solve problems. By adding a CI, it
will also allow others that are interested in this to help review and debug
problems occoured. We are also willing to provide machines for contibutors
that are intrested or want to debuging for ARM related issues.*

And another thing I want to add on is, I can see from the thread and
previous discussions in ML and Jira, not much people seems interested in
things like make Hadoop working on Aarch64 platform. Here I want to provide
some ideas that might make this more attractive. When speaking of ARM or
Aarch64, devices like raspberry pi may be the first thing that pop up to
our head, we might think 'oh, it could be running on raspberry pi,
interesting' and treated it as a toy or test and then we go back to our
normal life. But actually, we can do much more. As we all know, over 90
percent of the mobile device uses ARM architecture, and there are more and
more ARM base datacenter servers are begaining to show up at the market.
With the increasing computing capability of mobile devices and increasing
speed of wireless connection like 5G, we could imagine what kind of
possiblity could we have if we can have Hadoop running on both datacenter
and mobile devices at edge. I'm not a Hadoop expert but I can imagine that
there is a ton of possiblilities and opportunities for new bussiness models
using Hadoop in the future. Probably some of them will change our life.
Just my 2 cents.

I have also updated in the issue that I've originally submitted, please
feel free to leave comments:
https://issues.apache.org/jira/browse/HADOOP-16358

BR,


On Fri, Sep 6, 2019 at 6:11 AM Vinayakumar B <vi...@apache.org>
wrote:

> Thanks @Anu
>
> I understand the concern. I took it in different manner.
>
> Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
> changes as early as possible, its better to do it trunk itself.
>
> I was able to come to successfull attempt of upgrading protobuf as per
> suggestion of stack in Jira
> https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
>  .
>
> Have created the PR. Please review.  Changes looks huge, because all
> references of "com.google.protobuf" relocated to
> "o.a.h.shaded.com.google.protobuf".
> Otherwise changes are reasonable.
>
> This change is with still keeping the current 2.5.0 dependency, for
> downstream builds. So essentially nothing should be changed for downstreams.
>
> -Vinay
>
>
> On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer <ae...@cloudera.com>
> wrote:
>
>> Yes, I think that is what Sunil and I are trying to suggest; the complex
>> dependencies like Protobuf, if you do it in the trunk you have a better
>> change of getting it done. Otherwise, at merge point random downstream
>> applications which you have never heard of will object, and Hadoop
>> compatibility rules are very clear so you cannot fix it.
>>
>> With that said, even doing this in the trunk is complex; It might be
>> good for you to host a meeting and get some feedback. I have openly said it
>> is a great idea like "belling the cat", but the effort is in getting the
>> community to agree and align. Solve that, most of your technical problems
>> will be easier to solve.
>>
>> If you go into a branch, it might be that the community might forget
>> about your work; and when you come in to merge you will see issues which
>> you did not think about.
>>
>> So, Here is what would be great if you can make this happen; for ARM
>> work, get a list of dependencies that needed to be upgraded; see if you can
>> get the community aligned with this goal; since ARM might not be in the
>> target for many users. You need to convince users that even without ARM,
>> this is a great idea.
>>
>> If you like we can get together during one of the HDFS meetups hosted by
>> Wei-chiu.
>>
>> Thanks
>> Anu
>>
>>
>>
>> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
>> wrote:
>>
>>> Hi all,
>>>
>>> Thanks for the response.
>>> As I see, protobuf upgrade is long pending and most awaited one.
>>>
>>> @Sunil
>>> Protobuf upgrade looks to be a non-trivial task.
>>> Thanks @Duo Zhang for the suggestion of
>>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>>> problem
>>> of dependency on build environment.
>>> However more problem lies in upgrade protobuf without breaking the
>>> downstream builds.
>>> Reason is many downstream projects directly refers server specific jars
>>> and
>>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>>> dependency.
>>>
>>> There are some historical discussions and suggestions on
>>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>>> upgrade.
>>> Recommended path for solution is, try to upgrade protobuf using shading
>>> of
>>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>>> dependencies for downstreams.
>>> I am trying out ideas on this and if it gets completed within time, may
>>> be
>>> we can target trunk itself for the protobuf upgrade.
>>>
>>> separate branch idea is suggested for the overall ARM support including
>>> protobuf upgrade and other problems mentioned in the discussion above.
>>>
>>> I dont expect separate branch to have a huge changes, apart from bug
>>> fixes,
>>> since there are no separate features specific to ARM is being planned.
>>> So timely rebase of separate branch would reduce the overhead on branch
>>> review/merge task.
>>>
>>> Still, if the solution to protobuf upgrade winds up early, without any
>>> side
>>> effects, I am more than happy to land it in trunk itself.
>>>
>>> Thanks,
>>> Vinay
>>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:
>>>
>>> > Thanks Vinay for starting the thread.
>>> >
>>> > I agree to Anu's view point related to protobuf. And with the
>>> suggestion
>>> > pointed out by Duo Zhang, if we can make use
>>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>>> 3.0.0
>>> > of protobuf will also be more easier.
>>> >
>>> > However i think its better to do this effort in trunk itself.
>>> > In offline talks, few members were interested to start 3.3.0 release.
>>> And
>>> > given that happens soon, I feel its better
>>> > we do this task in trunk itself as branch diverge is very much
>>> possible.
>>> > And to bring to call a merge on such a big branch will be even more
>>> tough
>>> > task.
>>> >
>>> > my 2 cents.
>>> >
>>> > Thanks
>>> > Sunil
>>> >
>>> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
>>> > wrote:
>>> >
>>> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>>> >> generate
>>> >> the protobuf code. It will download the protoc binary from the maven
>>> >> central so we do not need to install protoc on the build machine any
>>> more.
>>> >>
>>> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>>> >>
>>> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been
>>> failling
>>> >> for
>>> >> > over 2 month related to the Protobuf problem .
>>> >> > According to the latest successful build log:
>>> >> >
>>> >>
>>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>>> >> > the
>>> >> > os version was ubuntu 14.04 and for the jobs that are failling now
>>> such
>>> >> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console
>>> ,
>>> >> > the os version is 18.04. I'm not very familiar with the version
>>> changing
>>> >> > for the jobs but I did a little search, according to:
>>> >> >
>>> >> >
>>> >>
>>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>>> >> > &
>>> >> >
>>> >> >
>>> >>
>>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>>> >> > it both said that the version of libprotc-dev and protobuf-compiler
>>> >> > available for ubuntu 18.04 is 3.0.0
>>> >> >
>>> >> >
>>> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
>>> wrote:
>>> >> >
>>> >> >> Thanx Vinay for the initiative, Makes sense to add support for
>>> >> different
>>> >> >> architectures.
>>> >> >>
>>> >> >> +1, for the branch idea.
>>> >> >> Good Luck!!!
>>> >> >>
>>> >> >> -Ayush
>>> >> >>
>>> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <palomino219@gmail.com
>>> >
>>> >> >> wrote:
>>> >> >> >
>>> >> >> > For HBase, we purged all the protobuf related things from the
>>> public
>>> >> >> API,
>>> >> >> > and then upgraded to a shaded and relocated version of protobuf.
>>> We
>>> >> have
>>> >> >> > created a repo for this:
>>> >> >> >
>>> >> >> > https://github.com/apache/hbase-thirdparty
>>> >> >> >
>>> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>>> >> jars,
>>> >> >> our
>>> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
>>> >> >> discuss
>>> >> >> > on how to deal with the upgrading of coprocessor. Glad to see
>>> that
>>> >> the
>>> >> >> > hadoop community is also willing to solve the problem.
>>> >> >> >
>>> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
>>> 上午1:23写道：
>>> >> >> >
>>> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is
>>> proving
>>> >> that
>>> >> >> >> Hadoop and the downstream projects work correctly after you
>>> upgrade
>>> >> >> core
>>> >> >> >> components like Protobuf.
>>> >> >> >> So while branching and working on a branch is easy, merging back
>>> >> after
>>> >> >> you
>>> >> >> >> upgrade some of these core components is insanely hard. You
>>> might
>>> >> want
>>> >> >> to
>>> >> >> >> make sure that community buys into upgrading these components
>>> in the
>>> >> >> trunk.
>>> >> >> >> That way we will get testing and downstream components will
>>> notice
>>> >> when
>>> >> >> >> things break.
>>> >> >> >>
>>> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a
>>> really
>>> >> long
>>> >> >> >> time; I have argued that 2.5 is out of support and we cannot
>>> stay on
>>> >> >> that
>>> >> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>>> >> code
>>> >> >> base.
>>> >> >> >> It has been rightly pointed to me that while all the arguments I
>>> >> make
>>> >> >> is
>>> >> >> >> correct; it is a very complicated task to upgrade Protobuf, and
>>> the
>>> >> >> worst
>>> >> >> >> part is we will not even know what breaks until downstream
>>> projects
>>> >> >> pick up
>>> >> >> >> these changes and work against us.
>>> >> >> >>
>>> >> >> >> If we work off the Hadoop version 3 — and assume that we have
>>> >> >> "shading" in
>>> >> >> >> place for all deployments; it might be possible to get there;
>>> still
>>> >> a
>>> >> >> >> daunting task.
>>> >> >> >>
>>> >> >> >> So best of luck with the branch approach — But please remember,
>>> >> Merging
>>> >> >> >> back will be hard, Just my 2 cents.
>>> >> >> >>
>>> >> >> >> — Anu
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>>> >> zhengzhenyulixi@gmail.com
>>> >> >> >
>>> >> >> >> wrote:
>>> >> >> >>
>>> >> >> >>> Hi,
>>> >> >> >>>
>>> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea.
>>> A
>>> >> >> separate
>>> >> >> >>> branch with it's own ARM CI seems a really good idea.
>>> >> >> >>> By doing this we won't break any of the undergoing development
>>> in
>>> >> >> trunk
>>> >> >> >> and
>>> >> >> >>> a CI can be a very good way to show what are the
>>> >> >> >>> current problems and what have been fixed, it will also
>>> provide a
>>> >> very
>>> >> >> >> good
>>> >> >> >>> view for contributors that are intrested to working on
>>> >> >> >>> this. We can finally merge back the branch to trunk until the
>>> >> >> community
>>> >> >> >>> thinks it is good enough and stable enough. We can donate
>>> >> >> >>> ARM machines to the existing CI system for the job.
>>> >> >> >>>
>>> >> >> >>> I wonder if this approch possible?
>>> >> >> >>>
>>> >> >> >>> BR,
>>> >> >> >>>
>>> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>>> >> liusheng2048@gmail.com>
>>> >> >> >>> wrote:
>>> >> >> >>>
>>> >> >> >>>> Hi,
>>> >> >> >>>>
>>> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>>> >> community
>>> >> >> >>>> mentioned by Vinay. I am working on building and
>>> >> >> >>>> testing Hadoop components on aarch64 server these days,
>>> besides
>>> >> the
>>> >> >> >>> missing
>>> >> >> >>>> dependices of ARM platform issues #1 #2 #3
>>> >> >> >>>> mentioned by Vinay, other similar issue has also be found,
>>> such as
>>> >> >> the
>>> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>>> >> >> >>>>
>>> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>>> >> hoped to
>>> >> >> >> add
>>> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
>>> >> >> >>>> sure about if there is any potential effect or confilict on
>>> the
>>> >> trunk
>>> >> >> >>>> branch, so maybe creating a ARM specific branch for doing
>>> these
>>> >> stuff
>>> >> >> >>>> is a better choice, what do you think?
>>> >> >> >>>>
>>> >> >> >>>> Hope to hear thoughts from you :)
>>> >> >> >>>>
>>> >> >> >>>> BR,
>>> >> >> >>>> Liu sheng
>>> >> >> >>>>
>>> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二
>>> 上午5:34写道：
>>> >> >> >>>>
>>> >> >> >>>>> Hi Folks,
>>> >> >> >>>>>
>>> >> >> >>>>> ARM is becoming famous lately in its processing capability
>>> and
>>> >> has
>>> >> >> >> got
>>> >> >> >>>> the
>>> >> >> >>>>> potential to run Bigdata workloads.
>>> >> >> >>>>> Many users have been moving to ARM machines due to its low
>>> cost.
>>> >> >> >>>>>
>>> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>>> >> (Rasberry
>>> >> >> >> PI)
>>> >> >> >>>> for
>>> >> >> >>>>> experimental purposes. Today ARM architecture is taking some
>>> of
>>> >> the
>>> >> >> >>>>> serverside processing as well. So there will be/is a real
>>> need of
>>> >> >> >>> Hadoop
>>> >> >> >>>> to
>>> >> >> >>>>> support ARM architecture as well.
>>> >> >> >>>>>
>>> >> >> >>>>> There are bunch of users who are trying out building Hadoop
>>> on
>>> >> ARM,
>>> >> >> >>>> trying
>>> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>>> >> >> >>>>>
>>> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
>>> issues,
>>> >> >> >> found
>>> >> >> >>>> from
>>> >> >> >>>>> testing done in openlab in [2].
>>> >> >> >>>>>
>>> >> >> >>>>> 1. Protobuf :
>>> >> >> >>>>> -------------------
>>> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>>> >> protobuf
>>> >> >> >>>> 2.5.0
>>> >> >> >>>>> version, due to backward compatibility reasons.
>>> Protobuf-2.5.0 is
>>> >> >> not
>>> >> >> >>>> being
>>> >> >> >>>>> maintained in the community. While protobuf 3.x is being
>>> actively
>>> >> >> >>> adopted
>>> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
>>> proto2
>>> >> >> >>>> messages.
>>> >> >> >>>>> Due to some compilation issues in the generated java code,
>>> which
>>> >> can
>>> >> >> >>>> induce
>>> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
>>> from
>>> >> >> >> 2.5.0
>>> >> >> >>>> was
>>> >> >> >>>>> not taken up.
>>> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to
>>> avoid
>>> >> >> >>> classpath
>>> >> >> >>>>> problem in downstream projects.
>>> >> >> >>>>>    There are patches available to fix compilation in Hadoop.
>>> But
>>> >> >> >> need
>>> >> >> >>> to
>>> >> >> >>>>> find a way to upgrade protobuf to latest version and still
>>> >> maintain
>>> >> >> >> the
>>> >> >> >>>>> downstream's classpath using shading feature of Hadoop build.
>>> >> >> >>>>>
>>> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even
>>> before
>>> >> >> >> shade
>>> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>>> >> >> >> continue
>>> >> >> >>>>> explore possibilities.
>>> >> >> >>>>>
>>> >> >> >>>>> 2. leveldbjni:
>>> >> >> >>>>> ---------------
>>> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>>> >> architecture,
>>> >> >> >>>> need
>>> >> >> >>>>> to check whether any of the future versions support ARM and
>>> can
>>> >> >> >> hadoop
>>> >> >> >>>>> upgrade to that version.
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>>> >> >> >>>>> -------------------------
>>> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM
>>> executable by
>>> >> >> >>> default
>>> >> >> >>>> in
>>> >> >> >>>>> the maven repository. Workaround is to build it locally and
>>> keep
>>> >> in
>>> >> >> >>> local
>>> >> >> >>>>> maven repository.
>>> >> >> >>>>> Need to check whether any future versions of
>>> >> 'protoc-gen-grpc-java'
>>> >> >> >> is
>>> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can
>>> upgrade it?
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> Once the compilation issues are solved, then there might be
>>> many
>>> >> >> >> native
>>> >> >> >>>>> code related issues due to different architectures.
>>> >> >> >>>>> So to explore everything, need to join hands together and
>>> >> proceed.
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> Let us discuss and check, whether any body else out there who
>>> >> also
>>> >> >> >> need
>>> >> >> >>>> the
>>> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend
>>> their
>>> >> hands
>>> >> >> >>> and
>>> >> >> >>>>> time in this work.
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>>> >> >> >>>>> [2]
>>> >> >> >>
>>> >> >>
>>> >>
>>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>>> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>>> >> >> >>>>>
>>> >> >> >>>>> -Vinay
>>> >> >> >>
>>> >> >>
>>> >> >
>>> >>
>>> >
>>>
>>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Zhenyu Zheng <zh...@gmail.com>]

Hi All,

Some updates for ARM CI, our team has succesfully donated ARM resources and
setup an ARM CI for Apache Spark:
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/
it will set to periodic job and then PR trigger when we think it is stable
enough.

I really hope we can do the same for Hadoop.

BR,

On Fri, Sep 6, 2019 at 6:11 AM Vinayakumar B <vi...@apache.org>
wrote:

> Thanks @Anu
>
> I understand the concern. I took it in different manner.
>
> Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
> changes as early as possible, its better to do it trunk itself.
>
> I was able to come to successfull attempt of upgrading protobuf as per
> suggestion of stack in Jira
> https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
>  .
>
> Have created the PR. Please review.  Changes looks huge, because all
> references of "com.google.protobuf" relocated to
> "o.a.h.shaded.com.google.protobuf".
> Otherwise changes are reasonable.
>
> This change is with still keeping the current 2.5.0 dependency, for
> downstream builds. So essentially nothing should be changed for downstreams.
>
> -Vinay
>
>
> On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer <ae...@cloudera.com>
> wrote:
>
>> Yes, I think that is what Sunil and I are trying to suggest; the complex
>> dependencies like Protobuf, if you do it in the trunk you have a better
>> change of getting it done. Otherwise, at merge point random downstream
>> applications which you have never heard of will object, and Hadoop
>> compatibility rules are very clear so you cannot fix it.
>>
>> With that said, even doing this in the trunk is complex; It might be
>> good for you to host a meeting and get some feedback. I have openly said it
>> is a great idea like "belling the cat", but the effort is in getting the
>> community to agree and align. Solve that, most of your technical problems
>> will be easier to solve.
>>
>> If you go into a branch, it might be that the community might forget
>> about your work; and when you come in to merge you will see issues which
>> you did not think about.
>>
>> So, Here is what would be great if you can make this happen; for ARM
>> work, get a list of dependencies that needed to be upgraded; see if you can
>> get the community aligned with this goal; since ARM might not be in the
>> target for many users. You need to convince users that even without ARM,
>> this is a great idea.
>>
>> If you like we can get together during one of the HDFS meetups hosted by
>> Wei-chiu.
>>
>> Thanks
>> Anu
>>
>>
>>
>> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
>> wrote:
>>
>>> Hi all,
>>>
>>> Thanks for the response.
>>> As I see, protobuf upgrade is long pending and most awaited one.
>>>
>>> @Sunil
>>> Protobuf upgrade looks to be a non-trivial task.
>>> Thanks @Duo Zhang for the suggestion of
>>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>>> problem
>>> of dependency on build environment.
>>> However more problem lies in upgrade protobuf without breaking the
>>> downstream builds.
>>> Reason is many downstream projects directly refers server specific jars
>>> and
>>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>>> dependency.
>>>
>>> There are some historical discussions and suggestions on
>>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>>> upgrade.
>>> Recommended path for solution is, try to upgrade protobuf using shading
>>> of
>>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>>> dependencies for downstreams.
>>> I am trying out ideas on this and if it gets completed within time, may
>>> be
>>> we can target trunk itself for the protobuf upgrade.
>>>
>>> separate branch idea is suggested for the overall ARM support including
>>> protobuf upgrade and other problems mentioned in the discussion above.
>>>
>>> I dont expect separate branch to have a huge changes, apart from bug
>>> fixes,
>>> since there are no separate features specific to ARM is being planned.
>>> So timely rebase of separate branch would reduce the overhead on branch
>>> review/merge task.
>>>
>>> Still, if the solution to protobuf upgrade winds up early, without any
>>> side
>>> effects, I am more than happy to land it in trunk itself.
>>>
>>> Thanks,
>>> Vinay
>>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:
>>>
>>> > Thanks Vinay for starting the thread.
>>> >
>>> > I agree to Anu's view point related to protobuf. And with the
>>> suggestion
>>> > pointed out by Duo Zhang, if we can make use
>>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>>> 3.0.0
>>> > of protobuf will also be more easier.
>>> >
>>> > However i think its better to do this effort in trunk itself.
>>> > In offline talks, few members were interested to start 3.3.0 release.
>>> And
>>> > given that happens soon, I feel its better
>>> > we do this task in trunk itself as branch diverge is very much
>>> possible.
>>> > And to bring to call a merge on such a big branch will be even more
>>> tough
>>> > task.
>>> >
>>> > my 2 cents.
>>> >
>>> > Thanks
>>> > Sunil
>>> >
>>> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
>>> > wrote:
>>> >
>>> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>>> >> generate
>>> >> the protobuf code. It will download the protoc binary from the maven
>>> >> central so we do not need to install protoc on the build machine any
>>> more.
>>> >>
>>> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>>> >>
>>> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been
>>> failling
>>> >> for
>>> >> > over 2 month related to the Protobuf problem .
>>> >> > According to the latest successful build log:
>>> >> >
>>> >>
>>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>>> >> > the
>>> >> > os version was ubuntu 14.04 and for the jobs that are failling now
>>> such
>>> >> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console
>>> ,
>>> >> > the os version is 18.04. I'm not very familiar with the version
>>> changing
>>> >> > for the jobs but I did a little search, according to:
>>> >> >
>>> >> >
>>> >>
>>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>>> >> > &
>>> >> >
>>> >> >
>>> >>
>>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>>> >> > it both said that the version of libprotc-dev and protobuf-compiler
>>> >> > available for ubuntu 18.04 is 3.0.0
>>> >> >
>>> >> >
>>> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
>>> wrote:
>>> >> >
>>> >> >> Thanx Vinay for the initiative, Makes sense to add support for
>>> >> different
>>> >> >> architectures.
>>> >> >>
>>> >> >> +1, for the branch idea.
>>> >> >> Good Luck!!!
>>> >> >>
>>> >> >> -Ayush
>>> >> >>
>>> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <palomino219@gmail.com
>>> >
>>> >> >> wrote:
>>> >> >> >
>>> >> >> > For HBase, we purged all the protobuf related things from the
>>> public
>>> >> >> API,
>>> >> >> > and then upgraded to a shaded and relocated version of protobuf.
>>> We
>>> >> have
>>> >> >> > created a repo for this:
>>> >> >> >
>>> >> >> > https://github.com/apache/hbase-thirdparty
>>> >> >> >
>>> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>>> >> jars,
>>> >> >> our
>>> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
>>> >> >> discuss
>>> >> >> > on how to deal with the upgrading of coprocessor. Glad to see
>>> that
>>> >> the
>>> >> >> > hadoop community is also willing to solve the problem.
>>> >> >> >
>>> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
>>> 上午1:23写道：
>>> >> >> >
>>> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is
>>> proving
>>> >> that
>>> >> >> >> Hadoop and the downstream projects work correctly after you
>>> upgrade
>>> >> >> core
>>> >> >> >> components like Protobuf.
>>> >> >> >> So while branching and working on a branch is easy, merging back
>>> >> after
>>> >> >> you
>>> >> >> >> upgrade some of these core components is insanely hard. You
>>> might
>>> >> want
>>> >> >> to
>>> >> >> >> make sure that community buys into upgrading these components
>>> in the
>>> >> >> trunk.
>>> >> >> >> That way we will get testing and downstream components will
>>> notice
>>> >> when
>>> >> >> >> things break.
>>> >> >> >>
>>> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a
>>> really
>>> >> long
>>> >> >> >> time; I have argued that 2.5 is out of support and we cannot
>>> stay on
>>> >> >> that
>>> >> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>>> >> code
>>> >> >> base.
>>> >> >> >> It has been rightly pointed to me that while all the arguments I
>>> >> make
>>> >> >> is
>>> >> >> >> correct; it is a very complicated task to upgrade Protobuf, and
>>> the
>>> >> >> worst
>>> >> >> >> part is we will not even know what breaks until downstream
>>> projects
>>> >> >> pick up
>>> >> >> >> these changes and work against us.
>>> >> >> >>
>>> >> >> >> If we work off the Hadoop version 3 — and assume that we have
>>> >> >> "shading" in
>>> >> >> >> place for all deployments; it might be possible to get there;
>>> still
>>> >> a
>>> >> >> >> daunting task.
>>> >> >> >>
>>> >> >> >> So best of luck with the branch approach — But please remember,
>>> >> Merging
>>> >> >> >> back will be hard, Just my 2 cents.
>>> >> >> >>
>>> >> >> >> — Anu
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>>> >> zhengzhenyulixi@gmail.com
>>> >> >> >
>>> >> >> >> wrote:
>>> >> >> >>
>>> >> >> >>> Hi,
>>> >> >> >>>
>>> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea.
>>> A
>>> >> >> separate
>>> >> >> >>> branch with it's own ARM CI seems a really good idea.
>>> >> >> >>> By doing this we won't break any of the undergoing development
>>> in
>>> >> >> trunk
>>> >> >> >> and
>>> >> >> >>> a CI can be a very good way to show what are the
>>> >> >> >>> current problems and what have been fixed, it will also
>>> provide a
>>> >> very
>>> >> >> >> good
>>> >> >> >>> view for contributors that are intrested to working on
>>> >> >> >>> this. We can finally merge back the branch to trunk until the
>>> >> >> community
>>> >> >> >>> thinks it is good enough and stable enough. We can donate
>>> >> >> >>> ARM machines to the existing CI system for the job.
>>> >> >> >>>
>>> >> >> >>> I wonder if this approch possible?
>>> >> >> >>>
>>> >> >> >>> BR,
>>> >> >> >>>
>>> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>>> >> liusheng2048@gmail.com>
>>> >> >> >>> wrote:
>>> >> >> >>>
>>> >> >> >>>> Hi,
>>> >> >> >>>>
>>> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>>> >> community
>>> >> >> >>>> mentioned by Vinay. I am working on building and
>>> >> >> >>>> testing Hadoop components on aarch64 server these days,
>>> besides
>>> >> the
>>> >> >> >>> missing
>>> >> >> >>>> dependices of ARM platform issues #1 #2 #3
>>> >> >> >>>> mentioned by Vinay, other similar issue has also be found,
>>> such as
>>> >> >> the
>>> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>>> >> >> >>>>
>>> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>>> >> hoped to
>>> >> >> >> add
>>> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
>>> >> >> >>>> sure about if there is any potential effect or confilict on
>>> the
>>> >> trunk
>>> >> >> >>>> branch, so maybe creating a ARM specific branch for doing
>>> these
>>> >> stuff
>>> >> >> >>>> is a better choice, what do you think?
>>> >> >> >>>>
>>> >> >> >>>> Hope to hear thoughts from you :)
>>> >> >> >>>>
>>> >> >> >>>> BR,
>>> >> >> >>>> Liu sheng
>>> >> >> >>>>
>>> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二
>>> 上午5:34写道：
>>> >> >> >>>>
>>> >> >> >>>>> Hi Folks,
>>> >> >> >>>>>
>>> >> >> >>>>> ARM is becoming famous lately in its processing capability
>>> and
>>> >> has
>>> >> >> >> got
>>> >> >> >>>> the
>>> >> >> >>>>> potential to run Bigdata workloads.
>>> >> >> >>>>> Many users have been moving to ARM machines due to its low
>>> cost.
>>> >> >> >>>>>
>>> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>>> >> (Rasberry
>>> >> >> >> PI)
>>> >> >> >>>> for
>>> >> >> >>>>> experimental purposes. Today ARM architecture is taking some
>>> of
>>> >> the
>>> >> >> >>>>> serverside processing as well. So there will be/is a real
>>> need of
>>> >> >> >>> Hadoop
>>> >> >> >>>> to
>>> >> >> >>>>> support ARM architecture as well.
>>> >> >> >>>>>
>>> >> >> >>>>> There are bunch of users who are trying out building Hadoop
>>> on
>>> >> ARM,
>>> >> >> >>>> trying
>>> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>>> >> >> >>>>>
>>> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
>>> issues,
>>> >> >> >> found
>>> >> >> >>>> from
>>> >> >> >>>>> testing done in openlab in [2].
>>> >> >> >>>>>
>>> >> >> >>>>> 1. Protobuf :
>>> >> >> >>>>> -------------------
>>> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>>> >> protobuf
>>> >> >> >>>> 2.5.0
>>> >> >> >>>>> version, due to backward compatibility reasons.
>>> Protobuf-2.5.0 is
>>> >> >> not
>>> >> >> >>>> being
>>> >> >> >>>>> maintained in the community. While protobuf 3.x is being
>>> actively
>>> >> >> >>> adopted
>>> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
>>> proto2
>>> >> >> >>>> messages.
>>> >> >> >>>>> Due to some compilation issues in the generated java code,
>>> which
>>> >> can
>>> >> >> >>>> induce
>>> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
>>> from
>>> >> >> >> 2.5.0
>>> >> >> >>>> was
>>> >> >> >>>>> not taken up.
>>> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to
>>> avoid
>>> >> >> >>> classpath
>>> >> >> >>>>> problem in downstream projects.
>>> >> >> >>>>>    There are patches available to fix compilation in Hadoop.
>>> But
>>> >> >> >> need
>>> >> >> >>> to
>>> >> >> >>>>> find a way to upgrade protobuf to latest version and still
>>> >> maintain
>>> >> >> >> the
>>> >> >> >>>>> downstream's classpath using shading feature of Hadoop build.
>>> >> >> >>>>>
>>> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even
>>> before
>>> >> >> >> shade
>>> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>>> >> >> >> continue
>>> >> >> >>>>> explore possibilities.
>>> >> >> >>>>>
>>> >> >> >>>>> 2. leveldbjni:
>>> >> >> >>>>> ---------------
>>> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>>> >> architecture,
>>> >> >> >>>> need
>>> >> >> >>>>> to check whether any of the future versions support ARM and
>>> can
>>> >> >> >> hadoop
>>> >> >> >>>>> upgrade to that version.
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>>> >> >> >>>>> -------------------------
>>> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM
>>> executable by
>>> >> >> >>> default
>>> >> >> >>>> in
>>> >> >> >>>>> the maven repository. Workaround is to build it locally and
>>> keep
>>> >> in
>>> >> >> >>> local
>>> >> >> >>>>> maven repository.
>>> >> >> >>>>> Need to check whether any future versions of
>>> >> 'protoc-gen-grpc-java'
>>> >> >> >> is
>>> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can
>>> upgrade it?
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> Once the compilation issues are solved, then there might be
>>> many
>>> >> >> >> native
>>> >> >> >>>>> code related issues due to different architectures.
>>> >> >> >>>>> So to explore everything, need to join hands together and
>>> >> proceed.
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> Let us discuss and check, whether any body else out there who
>>> >> also
>>> >> >> >> need
>>> >> >> >>>> the
>>> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend
>>> their
>>> >> hands
>>> >> >> >>> and
>>> >> >> >>>>> time in this work.
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>>> >> >> >>>>> [2]
>>> >> >> >>
>>> >> >>
>>> >>
>>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>>> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>>> >> >> >>>>>
>>> >> >> >>>>> -Vinay
>>> >> >> >>
>>> >> >>
>>> >> >
>>> >>
>>> >
>>>
>>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B <vi...@apache.org>]

Thanks @Anu

I understand the concern. I took it in different manner.

Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
changes as early as possible, its better to do it trunk itself.

I was able to come to successfull attempt of upgrading protobuf as per
suggestion of stack in Jira
https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
 .

Have created the PR. Please review.  Changes looks huge, because all
references of "com.google.protobuf" relocated to
"o.a.h.shaded.com.google.protobuf".
Otherwise changes are reasonable.

This change is with still keeping the current 2.5.0 dependency, for
downstream builds. So essentially nothing should be changed for downstreams.

-Vinay


On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer <ae...@cloudera.com> wrote:

> Yes, I think that is what Sunil and I are trying to suggest; the complex
> dependencies like Protobuf, if you do it in the trunk you have a better
> change of getting it done. Otherwise, at merge point random downstream
> applications which you have never heard of will object, and Hadoop
> compatibility rules are very clear so you cannot fix it.
>
> With that said, even doing this in the trunk is complex; It might be good
> for you to host a meeting and get some feedback. I have openly said it is a
> great idea like "belling the cat", but the effort is in getting the
> community to agree and align. Solve that, most of your technical problems
> will be easier to solve.
>
> If you go into a branch, it might be that the community might forget about
> your work; and when you come in to merge you will see issues which you did
> not think about.
>
> So, Here is what would be great if you can make this happen; for ARM work,
> get a list of dependencies that needed to be upgraded; see if you can get
> the community aligned with this goal; since ARM might not be in the target
> for many users. You need to convince users that even without ARM, this is a
> great idea.
>
> If you like we can get together during one of the HDFS meetups hosted by
> Wei-chiu.
>
> Thanks
> Anu
>
>
>
> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
> wrote:
>
>> Hi all,
>>
>> Thanks for the response.
>> As I see, protobuf upgrade is long pending and most awaited one.
>>
>> @Sunil
>> Protobuf upgrade looks to be a non-trivial task.
>> Thanks @Duo Zhang for the suggestion of
>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>> problem
>> of dependency on build environment.
>> However more problem lies in upgrade protobuf without breaking the
>> downstream builds.
>> Reason is many downstream projects directly refers server specific jars
>> and
>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>> dependency.
>>
>> There are some historical discussions and suggestions on
>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>> upgrade.
>> Recommended path for solution is, try to upgrade protobuf using shading of
>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>> dependencies for downstreams.
>> I am trying out ideas on this and if it gets completed within time, may be
>> we can target trunk itself for the protobuf upgrade.
>>
>> separate branch idea is suggested for the overall ARM support including
>> protobuf upgrade and other problems mentioned in the discussion above.
>>
>> I dont expect separate branch to have a huge changes, apart from bug
>> fixes,
>> since there are no separate features specific to ARM is being planned.
>> So timely rebase of separate branch would reduce the overhead on branch
>> review/merge task.
>>
>> Still, if the solution to protobuf upgrade winds up early, without any
>> side
>> effects, I am more than happy to land it in trunk itself.
>>
>> Thanks,
>> Vinay
>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:
>>
>> > Thanks Vinay for starting the thread.
>> >
>> > I agree to Anu's view point related to protobuf. And with the suggestion
>> > pointed out by Duo Zhang, if we can make use
>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>> 3.0.0
>> > of protobuf will also be more easier.
>> >
>> > However i think its better to do this effort in trunk itself.
>> > In offline talks, few members were interested to start 3.3.0 release.
>> And
>> > given that happens soon, I feel its better
>> > we do this task in trunk itself as branch diverge is very much possible.
>> > And to bring to call a merge on such a big branch will be even more
>> tough
>> > task.
>> >
>> > my 2 cents.
>> >
>> > Thanks
>> > Sunil
>> >
>> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
>> > wrote:
>> >
>> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>> >> generate
>> >> the protobuf code. It will download the protoc binary from the maven
>> >> central so we do not need to install protoc on the build machine any
>> more.
>> >>
>> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>> >>
>> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been
>> failling
>> >> for
>> >> > over 2 month related to the Protobuf problem .
>> >> > According to the latest successful build log:
>> >> >
>> >>
>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>> >> > the
>> >> > os version was ubuntu 14.04 and for the jobs that are failling now
>> such
>> >> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
>> >> > the os version is 18.04. I'm not very familiar with the version
>> changing
>> >> > for the jobs but I did a little search, according to:
>> >> >
>> >> >
>> >>
>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>> >> > &
>> >> >
>> >> >
>> >>
>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>> >> > it both said that the version of libprotc-dev and protobuf-compiler
>> >> > available for ubuntu 18.04 is 3.0.0
>> >> >
>> >> >
>> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
>> wrote:
>> >> >
>> >> >> Thanx Vinay for the initiative, Makes sense to add support for
>> >> different
>> >> >> architectures.
>> >> >>
>> >> >> +1, for the branch idea.
>> >> >> Good Luck!!!
>> >> >>
>> >> >> -Ayush
>> >> >>
>> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
>> >> >> wrote:
>> >> >> >
>> >> >> > For HBase, we purged all the protobuf related things from the
>> public
>> >> >> API,
>> >> >> > and then upgraded to a shaded and relocated version of protobuf.
>> We
>> >> have
>> >> >> > created a repo for this:
>> >> >> >
>> >> >> > https://github.com/apache/hbase-thirdparty
>> >> >> >
>> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>> >> jars,
>> >> >> our
>> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
>> >> >> discuss
>> >> >> > on how to deal with the upgrading of coprocessor. Glad to see that
>> >> the
>> >> >> > hadoop community is also willing to solve the problem.
>> >> >> >
>> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
>> 上午1:23写道：
>> >> >> >
>> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is
>> proving
>> >> that
>> >> >> >> Hadoop and the downstream projects work correctly after you
>> upgrade
>> >> >> core
>> >> >> >> components like Protobuf.
>> >> >> >> So while branching and working on a branch is easy, merging back
>> >> after
>> >> >> you
>> >> >> >> upgrade some of these core components is insanely hard. You might
>> >> want
>> >> >> to
>> >> >> >> make sure that community buys into upgrading these components in
>> the
>> >> >> trunk.
>> >> >> >> That way we will get testing and downstream components will
>> notice
>> >> when
>> >> >> >> things break.
>> >> >> >>
>> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a
>> really
>> >> long
>> >> >> >> time; I have argued that 2.5 is out of support and we cannot
>> stay on
>> >> >> that
>> >> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>> >> code
>> >> >> base.
>> >> >> >> It has been rightly pointed to me that while all the arguments I
>> >> make
>> >> >> is
>> >> >> >> correct; it is a very complicated task to upgrade Protobuf, and
>> the
>> >> >> worst
>> >> >> >> part is we will not even know what breaks until downstream
>> projects
>> >> >> pick up
>> >> >> >> these changes and work against us.
>> >> >> >>
>> >> >> >> If we work off the Hadoop version 3 — and assume that we have
>> >> >> "shading" in
>> >> >> >> place for all deployments; it might be possible to get there;
>> still
>> >> a
>> >> >> >> daunting task.
>> >> >> >>
>> >> >> >> So best of luck with the branch approach — But please remember,
>> >> Merging
>> >> >> >> back will be hard, Just my 2 cents.
>> >> >> >>
>> >> >> >> — Anu
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>> >> zhengzhenyulixi@gmail.com
>> >> >> >
>> >> >> >> wrote:
>> >> >> >>
>> >> >> >>> Hi,
>> >> >> >>>
>> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
>> >> >> separate
>> >> >> >>> branch with it's own ARM CI seems a really good idea.
>> >> >> >>> By doing this we won't break any of the undergoing development
>> in
>> >> >> trunk
>> >> >> >> and
>> >> >> >>> a CI can be a very good way to show what are the
>> >> >> >>> current problems and what have been fixed, it will also provide
>> a
>> >> very
>> >> >> >> good
>> >> >> >>> view for contributors that are intrested to working on
>> >> >> >>> this. We can finally merge back the branch to trunk until the
>> >> >> community
>> >> >> >>> thinks it is good enough and stable enough. We can donate
>> >> >> >>> ARM machines to the existing CI system for the job.
>> >> >> >>>
>> >> >> >>> I wonder if this approch possible?
>> >> >> >>>
>> >> >> >>> BR,
>> >> >> >>>
>> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>> >> liusheng2048@gmail.com>
>> >> >> >>> wrote:
>> >> >> >>>
>> >> >> >>>> Hi,
>> >> >> >>>>
>> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>> >> community
>> >> >> >>>> mentioned by Vinay. I am working on building and
>> >> >> >>>> testing Hadoop components on aarch64 server these days, besides
>> >> the
>> >> >> >>> missing
>> >> >> >>>> dependices of ARM platform issues #1 #2 #3
>> >> >> >>>> mentioned by Vinay, other similar issue has also be found,
>> such as
>> >> >> the
>> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >> >> >>>>
>> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>> >> hoped to
>> >> >> >> add
>> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
>> >> >> >>>> sure about if there is any potential effect or confilict on the
>> >> trunk
>> >> >> >>>> branch, so maybe creating a ARM specific branch for doing these
>> >> stuff
>> >> >> >>>> is a better choice, what do you think?
>> >> >> >>>>
>> >> >> >>>> Hope to hear thoughts from you :)
>> >> >> >>>>
>> >> >> >>>> BR,
>> >> >> >>>> Liu sheng
>> >> >> >>>>
>> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二
>> 上午5:34写道：
>> >> >> >>>>
>> >> >> >>>>> Hi Folks,
>> >> >> >>>>>
>> >> >> >>>>> ARM is becoming famous lately in its processing capability and
>> >> has
>> >> >> >> got
>> >> >> >>>> the
>> >> >> >>>>> potential to run Bigdata workloads.
>> >> >> >>>>> Many users have been moving to ARM machines due to its low
>> cost.
>> >> >> >>>>>
>> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>> >> (Rasberry
>> >> >> >> PI)
>> >> >> >>>> for
>> >> >> >>>>> experimental purposes. Today ARM architecture is taking some
>> of
>> >> the
>> >> >> >>>>> serverside processing as well. So there will be/is a real
>> need of
>> >> >> >>> Hadoop
>> >> >> >>>> to
>> >> >> >>>>> support ARM architecture as well.
>> >> >> >>>>>
>> >> >> >>>>> There are bunch of users who are trying out building Hadoop on
>> >> ARM,
>> >> >> >>>> trying
>> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>> >> >> >>>>>
>> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
>> issues,
>> >> >> >> found
>> >> >> >>>> from
>> >> >> >>>>> testing done in openlab in [2].
>> >> >> >>>>>
>> >> >> >>>>> 1. Protobuf :
>> >> >> >>>>> -------------------
>> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>> >> protobuf
>> >> >> >>>> 2.5.0
>> >> >> >>>>> version, due to backward compatibility reasons.
>> Protobuf-2.5.0 is
>> >> >> not
>> >> >> >>>> being
>> >> >> >>>>> maintained in the community. While protobuf 3.x is being
>> actively
>> >> >> >>> adopted
>> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
>> proto2
>> >> >> >>>> messages.
>> >> >> >>>>> Due to some compilation issues in the generated java code,
>> which
>> >> can
>> >> >> >>>> induce
>> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
>> from
>> >> >> >> 2.5.0
>> >> >> >>>> was
>> >> >> >>>>> not taken up.
>> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to
>> avoid
>> >> >> >>> classpath
>> >> >> >>>>> problem in downstream projects.
>> >> >> >>>>>    There are patches available to fix compilation in Hadoop.
>> But
>> >> >> >> need
>> >> >> >>> to
>> >> >> >>>>> find a way to upgrade protobuf to latest version and still
>> >> maintain
>> >> >> >> the
>> >> >> >>>>> downstream's classpath using shading feature of Hadoop build.
>> >> >> >>>>>
>> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even
>> before
>> >> >> >> shade
>> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>> >> >> >> continue
>> >> >> >>>>> explore possibilities.
>> >> >> >>>>>
>> >> >> >>>>> 2. leveldbjni:
>> >> >> >>>>> ---------------
>> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>> >> architecture,
>> >> >> >>>> need
>> >> >> >>>>> to check whether any of the future versions support ARM and
>> can
>> >> >> >> hadoop
>> >> >> >>>>> upgrade to that version.
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>> >> >> >>>>> -------------------------
>> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable
>> by
>> >> >> >>> default
>> >> >> >>>> in
>> >> >> >>>>> the maven repository. Workaround is to build it locally and
>> keep
>> >> in
>> >> >> >>> local
>> >> >> >>>>> maven repository.
>> >> >> >>>>> Need to check whether any future versions of
>> >> 'protoc-gen-grpc-java'
>> >> >> >> is
>> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade
>> it?
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> Once the compilation issues are solved, then there might be
>> many
>> >> >> >> native
>> >> >> >>>>> code related issues due to different architectures.
>> >> >> >>>>> So to explore everything, need to join hands together and
>> >> proceed.
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> Let us discuss and check, whether any body else out there who
>> >> also
>> >> >> >> need
>> >> >> >>>> the
>> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
>> >> hands
>> >> >> >>> and
>> >> >> >>>>> time in this work.
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>> >> >> >>>>> [2]
>> >> >> >>
>> >> >>
>> >>
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>> >> >> >>>>>
>> >> >> >>>>> -Vinay
>> >> >> >>
>> >> >>
>> >> >
>> >>
>> >
>>
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B <vi...@apache.org>]

Thanks @Anu

I understand the concern. I took it in different manner.

Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
changes as early as possible, its better to do it trunk itself.

I was able to come to successfull attempt of upgrading protobuf as per
suggestion of stack in Jira
https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
 .

Have created the PR. Please review.  Changes looks huge, because all
references of "com.google.protobuf" relocated to
"o.a.h.shaded.com.google.protobuf".
Otherwise changes are reasonable.

This change is with still keeping the current 2.5.0 dependency, for
downstream builds. So essentially nothing should be changed for downstreams.

-Vinay


On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer <ae...@cloudera.com> wrote:

> Yes, I think that is what Sunil and I are trying to suggest; the complex
> dependencies like Protobuf, if you do it in the trunk you have a better
> change of getting it done. Otherwise, at merge point random downstream
> applications which you have never heard of will object, and Hadoop
> compatibility rules are very clear so you cannot fix it.
>
> With that said, even doing this in the trunk is complex; It might be good
> for you to host a meeting and get some feedback. I have openly said it is a
> great idea like "belling the cat", but the effort is in getting the
> community to agree and align. Solve that, most of your technical problems
> will be easier to solve.
>
> If you go into a branch, it might be that the community might forget about
> your work; and when you come in to merge you will see issues which you did
> not think about.
>
> So, Here is what would be great if you can make this happen; for ARM work,
> get a list of dependencies that needed to be upgraded; see if you can get
> the community aligned with this goal; since ARM might not be in the target
> for many users. You need to convince users that even without ARM, this is a
> great idea.
>
> If you like we can get together during one of the HDFS meetups hosted by
> Wei-chiu.
>
> Thanks
> Anu
>
>
>
> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
> wrote:
>
>> Hi all,
>>
>> Thanks for the response.
>> As I see, protobuf upgrade is long pending and most awaited one.
>>
>> @Sunil
>> Protobuf upgrade looks to be a non-trivial task.
>> Thanks @Duo Zhang for the suggestion of
>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>> problem
>> of dependency on build environment.
>> However more problem lies in upgrade protobuf without breaking the
>> downstream builds.
>> Reason is many downstream projects directly refers server specific jars
>> and
>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>> dependency.
>>
>> There are some historical discussions and suggestions on
>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>> upgrade.
>> Recommended path for solution is, try to upgrade protobuf using shading of
>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>> dependencies for downstreams.
>> I am trying out ideas on this and if it gets completed within time, may be
>> we can target trunk itself for the protobuf upgrade.
>>
>> separate branch idea is suggested for the overall ARM support including
>> protobuf upgrade and other problems mentioned in the discussion above.
>>
>> I dont expect separate branch to have a huge changes, apart from bug
>> fixes,
>> since there are no separate features specific to ARM is being planned.
>> So timely rebase of separate branch would reduce the overhead on branch
>> review/merge task.
>>
>> Still, if the solution to protobuf upgrade winds up early, without any
>> side
>> effects, I am more than happy to land it in trunk itself.
>>
>> Thanks,
>> Vinay
>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:
>>
>> > Thanks Vinay for starting the thread.
>> >
>> > I agree to Anu's view point related to protobuf. And with the suggestion
>> > pointed out by Duo Zhang, if we can make use
>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>> 3.0.0
>> > of protobuf will also be more easier.
>> >
>> > However i think its better to do this effort in trunk itself.
>> > In offline talks, few members were interested to start 3.3.0 release.
>> And
>> > given that happens soon, I feel its better
>> > we do this task in trunk itself as branch diverge is very much possible.
>> > And to bring to call a merge on such a big branch will be even more
>> tough
>> > task.
>> >
>> > my 2 cents.
>> >
>> > Thanks
>> > Sunil
>> >
>> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
>> > wrote:
>> >
>> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>> >> generate
>> >> the protobuf code. It will download the protoc binary from the maven
>> >> central so we do not need to install protoc on the build machine any
>> more.
>> >>
>> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>> >>
>> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been
>> failling
>> >> for
>> >> > over 2 month related to the Protobuf problem .
>> >> > According to the latest successful build log:
>> >> >
>> >>
>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>> >> > the
>> >> > os version was ubuntu 14.04 and for the jobs that are failling now
>> such
>> >> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
>> >> > the os version is 18.04. I'm not very familiar with the version
>> changing
>> >> > for the jobs but I did a little search, according to:
>> >> >
>> >> >
>> >>
>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>> >> > &
>> >> >
>> >> >
>> >>
>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>> >> > it both said that the version of libprotc-dev and protobuf-compiler
>> >> > available for ubuntu 18.04 is 3.0.0
>> >> >
>> >> >
>> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
>> wrote:
>> >> >
>> >> >> Thanx Vinay for the initiative, Makes sense to add support for
>> >> different
>> >> >> architectures.
>> >> >>
>> >> >> +1, for the branch idea.
>> >> >> Good Luck!!!
>> >> >>
>> >> >> -Ayush
>> >> >>
>> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
>> >> >> wrote:
>> >> >> >
>> >> >> > For HBase, we purged all the protobuf related things from the
>> public
>> >> >> API,
>> >> >> > and then upgraded to a shaded and relocated version of protobuf.
>> We
>> >> have
>> >> >> > created a repo for this:
>> >> >> >
>> >> >> > https://github.com/apache/hbase-thirdparty
>> >> >> >
>> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>> >> jars,
>> >> >> our
>> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
>> >> >> discuss
>> >> >> > on how to deal with the upgrading of coprocessor. Glad to see that
>> >> the
>> >> >> > hadoop community is also willing to solve the problem.
>> >> >> >
>> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
>> 上午1:23写道：
>> >> >> >
>> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is
>> proving
>> >> that
>> >> >> >> Hadoop and the downstream projects work correctly after you
>> upgrade
>> >> >> core
>> >> >> >> components like Protobuf.
>> >> >> >> So while branching and working on a branch is easy, merging back
>> >> after
>> >> >> you
>> >> >> >> upgrade some of these core components is insanely hard. You might
>> >> want
>> >> >> to
>> >> >> >> make sure that community buys into upgrading these components in
>> the
>> >> >> trunk.
>> >> >> >> That way we will get testing and downstream components will
>> notice
>> >> when
>> >> >> >> things break.
>> >> >> >>
>> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a
>> really
>> >> long
>> >> >> >> time; I have argued that 2.5 is out of support and we cannot
>> stay on
>> >> >> that
>> >> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>> >> code
>> >> >> base.
>> >> >> >> It has been rightly pointed to me that while all the arguments I
>> >> make
>> >> >> is
>> >> >> >> correct; it is a very complicated task to upgrade Protobuf, and
>> the
>> >> >> worst
>> >> >> >> part is we will not even know what breaks until downstream
>> projects
>> >> >> pick up
>> >> >> >> these changes and work against us.
>> >> >> >>
>> >> >> >> If we work off the Hadoop version 3 — and assume that we have
>> >> >> "shading" in
>> >> >> >> place for all deployments; it might be possible to get there;
>> still
>> >> a
>> >> >> >> daunting task.
>> >> >> >>
>> >> >> >> So best of luck with the branch approach — But please remember,
>> >> Merging
>> >> >> >> back will be hard, Just my 2 cents.
>> >> >> >>
>> >> >> >> — Anu
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>> >> zhengzhenyulixi@gmail.com
>> >> >> >
>> >> >> >> wrote:
>> >> >> >>
>> >> >> >>> Hi,
>> >> >> >>>
>> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
>> >> >> separate
>> >> >> >>> branch with it's own ARM CI seems a really good idea.
>> >> >> >>> By doing this we won't break any of the undergoing development
>> in
>> >> >> trunk
>> >> >> >> and
>> >> >> >>> a CI can be a very good way to show what are the
>> >> >> >>> current problems and what have been fixed, it will also provide
>> a
>> >> very
>> >> >> >> good
>> >> >> >>> view for contributors that are intrested to working on
>> >> >> >>> this. We can finally merge back the branch to trunk until the
>> >> >> community
>> >> >> >>> thinks it is good enough and stable enough. We can donate
>> >> >> >>> ARM machines to the existing CI system for the job.
>> >> >> >>>
>> >> >> >>> I wonder if this approch possible?
>> >> >> >>>
>> >> >> >>> BR,
>> >> >> >>>
>> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>> >> liusheng2048@gmail.com>
>> >> >> >>> wrote:
>> >> >> >>>
>> >> >> >>>> Hi,
>> >> >> >>>>
>> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>> >> community
>> >> >> >>>> mentioned by Vinay. I am working on building and
>> >> >> >>>> testing Hadoop components on aarch64 server these days, besides
>> >> the
>> >> >> >>> missing
>> >> >> >>>> dependices of ARM platform issues #1 #2 #3
>> >> >> >>>> mentioned by Vinay, other similar issue has also be found,
>> such as
>> >> >> the
>> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >> >> >>>>
>> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>> >> hoped to
>> >> >> >> add
>> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
>> >> >> >>>> sure about if there is any potential effect or confilict on the
>> >> trunk
>> >> >> >>>> branch, so maybe creating a ARM specific branch for doing these
>> >> stuff
>> >> >> >>>> is a better choice, what do you think?
>> >> >> >>>>
>> >> >> >>>> Hope to hear thoughts from you :)
>> >> >> >>>>
>> >> >> >>>> BR,
>> >> >> >>>> Liu sheng
>> >> >> >>>>
>> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二
>> 上午5:34写道：
>> >> >> >>>>
>> >> >> >>>>> Hi Folks,
>> >> >> >>>>>
>> >> >> >>>>> ARM is becoming famous lately in its processing capability and
>> >> has
>> >> >> >> got
>> >> >> >>>> the
>> >> >> >>>>> potential to run Bigdata workloads.
>> >> >> >>>>> Many users have been moving to ARM machines due to its low
>> cost.
>> >> >> >>>>>
>> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>> >> (Rasberry
>> >> >> >> PI)
>> >> >> >>>> for
>> >> >> >>>>> experimental purposes. Today ARM architecture is taking some
>> of
>> >> the
>> >> >> >>>>> serverside processing as well. So there will be/is a real
>> need of
>> >> >> >>> Hadoop
>> >> >> >>>> to
>> >> >> >>>>> support ARM architecture as well.
>> >> >> >>>>>
>> >> >> >>>>> There are bunch of users who are trying out building Hadoop on
>> >> ARM,
>> >> >> >>>> trying
>> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>> >> >> >>>>>
>> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
>> issues,
>> >> >> >> found
>> >> >> >>>> from
>> >> >> >>>>> testing done in openlab in [2].
>> >> >> >>>>>
>> >> >> >>>>> 1. Protobuf :
>> >> >> >>>>> -------------------
>> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>> >> protobuf
>> >> >> >>>> 2.5.0
>> >> >> >>>>> version, due to backward compatibility reasons.
>> Protobuf-2.5.0 is
>> >> >> not
>> >> >> >>>> being
>> >> >> >>>>> maintained in the community. While protobuf 3.x is being
>> actively
>> >> >> >>> adopted
>> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
>> proto2
>> >> >> >>>> messages.
>> >> >> >>>>> Due to some compilation issues in the generated java code,
>> which
>> >> can
>> >> >> >>>> induce
>> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
>> from
>> >> >> >> 2.5.0
>> >> >> >>>> was
>> >> >> >>>>> not taken up.
>> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to
>> avoid
>> >> >> >>> classpath
>> >> >> >>>>> problem in downstream projects.
>> >> >> >>>>>    There are patches available to fix compilation in Hadoop.
>> But
>> >> >> >> need
>> >> >> >>> to
>> >> >> >>>>> find a way to upgrade protobuf to latest version and still
>> >> maintain
>> >> >> >> the
>> >> >> >>>>> downstream's classpath using shading feature of Hadoop build.
>> >> >> >>>>>
>> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even
>> before
>> >> >> >> shade
>> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>> >> >> >> continue
>> >> >> >>>>> explore possibilities.
>> >> >> >>>>>
>> >> >> >>>>> 2. leveldbjni:
>> >> >> >>>>> ---------------
>> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>> >> architecture,
>> >> >> >>>> need
>> >> >> >>>>> to check whether any of the future versions support ARM and
>> can
>> >> >> >> hadoop
>> >> >> >>>>> upgrade to that version.
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>> >> >> >>>>> -------------------------
>> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable
>> by
>> >> >> >>> default
>> >> >> >>>> in
>> >> >> >>>>> the maven repository. Workaround is to build it locally and
>> keep
>> >> in
>> >> >> >>> local
>> >> >> >>>>> maven repository.
>> >> >> >>>>> Need to check whether any future versions of
>> >> 'protoc-gen-grpc-java'
>> >> >> >> is
>> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade
>> it?
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> Once the compilation issues are solved, then there might be
>> many
>> >> >> >> native
>> >> >> >>>>> code related issues due to different architectures.
>> >> >> >>>>> So to explore everything, need to join hands together and
>> >> proceed.
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> Let us discuss and check, whether any body else out there who
>> >> also
>> >> >> >> need
>> >> >> >>>> the
>> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
>> >> hands
>> >> >> >>> and
>> >> >> >>>>> time in this work.
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>> >> >> >>>>> [2]
>> >> >> >>
>> >> >>
>> >>
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>> >> >> >>>>>
>> >> >> >>>>> -Vinay
>> >> >> >>
>> >> >>
>> >> >
>> >>
>> >
>>
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B <vi...@apache.org>]

Thanks @Anu

I understand the concern. I took it in different manner.

Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
changes as early as possible, its better to do it trunk itself.

I was able to come to successfull attempt of upgrading protobuf as per
suggestion of stack in Jira
https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
 .

Have created the PR. Please review.  Changes looks huge, because all
references of "com.google.protobuf" relocated to
"o.a.h.shaded.com.google.protobuf".
Otherwise changes are reasonable.

This change is with still keeping the current 2.5.0 dependency, for
downstream builds. So essentially nothing should be changed for downstreams.

-Vinay


On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer <ae...@cloudera.com> wrote:

> Yes, I think that is what Sunil and I are trying to suggest; the complex
> dependencies like Protobuf, if you do it in the trunk you have a better
> change of getting it done. Otherwise, at merge point random downstream
> applications which you have never heard of will object, and Hadoop
> compatibility rules are very clear so you cannot fix it.
>
> With that said, even doing this in the trunk is complex; It might be good
> for you to host a meeting and get some feedback. I have openly said it is a
> great idea like "belling the cat", but the effort is in getting the
> community to agree and align. Solve that, most of your technical problems
> will be easier to solve.
>
> If you go into a branch, it might be that the community might forget about
> your work; and when you come in to merge you will see issues which you did
> not think about.
>
> So, Here is what would be great if you can make this happen; for ARM work,
> get a list of dependencies that needed to be upgraded; see if you can get
> the community aligned with this goal; since ARM might not be in the target
> for many users. You need to convince users that even without ARM, this is a
> great idea.
>
> If you like we can get together during one of the HDFS meetups hosted by
> Wei-chiu.
>
> Thanks
> Anu
>
>
>
> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
> wrote:
>
>> Hi all,
>>
>> Thanks for the response.
>> As I see, protobuf upgrade is long pending and most awaited one.
>>
>> @Sunil
>> Protobuf upgrade looks to be a non-trivial task.
>> Thanks @Duo Zhang for the suggestion of
>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>> problem
>> of dependency on build environment.
>> However more problem lies in upgrade protobuf without breaking the
>> downstream builds.
>> Reason is many downstream projects directly refers server specific jars
>> and
>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>> dependency.
>>
>> There are some historical discussions and suggestions on
>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>> upgrade.
>> Recommended path for solution is, try to upgrade protobuf using shading of
>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>> dependencies for downstreams.
>> I am trying out ideas on this and if it gets completed within time, may be
>> we can target trunk itself for the protobuf upgrade.
>>
>> separate branch idea is suggested for the overall ARM support including
>> protobuf upgrade and other problems mentioned in the discussion above.
>>
>> I dont expect separate branch to have a huge changes, apart from bug
>> fixes,
>> since there are no separate features specific to ARM is being planned.
>> So timely rebase of separate branch would reduce the overhead on branch
>> review/merge task.
>>
>> Still, if the solution to protobuf upgrade winds up early, without any
>> side
>> effects, I am more than happy to land it in trunk itself.
>>
>> Thanks,
>> Vinay
>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:
>>
>> > Thanks Vinay for starting the thread.
>> >
>> > I agree to Anu's view point related to protobuf. And with the suggestion
>> > pointed out by Duo Zhang, if we can make use
>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>> 3.0.0
>> > of protobuf will also be more easier.
>> >
>> > However i think its better to do this effort in trunk itself.
>> > In offline talks, few members were interested to start 3.3.0 release.
>> And
>> > given that happens soon, I feel its better
>> > we do this task in trunk itself as branch diverge is very much possible.
>> > And to bring to call a merge on such a big branch will be even more
>> tough
>> > task.
>> >
>> > my 2 cents.
>> >
>> > Thanks
>> > Sunil
>> >
>> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
>> > wrote:
>> >
>> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>> >> generate
>> >> the protobuf code. It will download the protoc binary from the maven
>> >> central so we do not need to install protoc on the build machine any
>> more.
>> >>
>> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>> >>
>> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been
>> failling
>> >> for
>> >> > over 2 month related to the Protobuf problem .
>> >> > According to the latest successful build log:
>> >> >
>> >>
>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>> >> > the
>> >> > os version was ubuntu 14.04 and for the jobs that are failling now
>> such
>> >> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
>> >> > the os version is 18.04. I'm not very familiar with the version
>> changing
>> >> > for the jobs but I did a little search, according to:
>> >> >
>> >> >
>> >>
>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>> >> > &
>> >> >
>> >> >
>> >>
>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>> >> > it both said that the version of libprotc-dev and protobuf-compiler
>> >> > available for ubuntu 18.04 is 3.0.0
>> >> >
>> >> >
>> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
>> wrote:
>> >> >
>> >> >> Thanx Vinay for the initiative, Makes sense to add support for
>> >> different
>> >> >> architectures.
>> >> >>
>> >> >> +1, for the branch idea.
>> >> >> Good Luck!!!
>> >> >>
>> >> >> -Ayush
>> >> >>
>> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
>> >> >> wrote:
>> >> >> >
>> >> >> > For HBase, we purged all the protobuf related things from the
>> public
>> >> >> API,
>> >> >> > and then upgraded to a shaded and relocated version of protobuf.
>> We
>> >> have
>> >> >> > created a repo for this:
>> >> >> >
>> >> >> > https://github.com/apache/hbase-thirdparty
>> >> >> >
>> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>> >> jars,
>> >> >> our
>> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
>> >> >> discuss
>> >> >> > on how to deal with the upgrading of coprocessor. Glad to see that
>> >> the
>> >> >> > hadoop community is also willing to solve the problem.
>> >> >> >
>> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
>> 上午1:23写道：
>> >> >> >
>> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is
>> proving
>> >> that
>> >> >> >> Hadoop and the downstream projects work correctly after you
>> upgrade
>> >> >> core
>> >> >> >> components like Protobuf.
>> >> >> >> So while branching and working on a branch is easy, merging back
>> >> after
>> >> >> you
>> >> >> >> upgrade some of these core components is insanely hard. You might
>> >> want
>> >> >> to
>> >> >> >> make sure that community buys into upgrading these components in
>> the
>> >> >> trunk.
>> >> >> >> That way we will get testing and downstream components will
>> notice
>> >> when
>> >> >> >> things break.
>> >> >> >>
>> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a
>> really
>> >> long
>> >> >> >> time; I have argued that 2.5 is out of support and we cannot
>> stay on
>> >> >> that
>> >> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>> >> code
>> >> >> base.
>> >> >> >> It has been rightly pointed to me that while all the arguments I
>> >> make
>> >> >> is
>> >> >> >> correct; it is a very complicated task to upgrade Protobuf, and
>> the
>> >> >> worst
>> >> >> >> part is we will not even know what breaks until downstream
>> projects
>> >> >> pick up
>> >> >> >> these changes and work against us.
>> >> >> >>
>> >> >> >> If we work off the Hadoop version 3 — and assume that we have
>> >> >> "shading" in
>> >> >> >> place for all deployments; it might be possible to get there;
>> still
>> >> a
>> >> >> >> daunting task.
>> >> >> >>
>> >> >> >> So best of luck with the branch approach — But please remember,
>> >> Merging
>> >> >> >> back will be hard, Just my 2 cents.
>> >> >> >>
>> >> >> >> — Anu
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>> >> zhengzhenyulixi@gmail.com
>> >> >> >
>> >> >> >> wrote:
>> >> >> >>
>> >> >> >>> Hi,
>> >> >> >>>
>> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
>> >> >> separate
>> >> >> >>> branch with it's own ARM CI seems a really good idea.
>> >> >> >>> By doing this we won't break any of the undergoing development
>> in
>> >> >> trunk
>> >> >> >> and
>> >> >> >>> a CI can be a very good way to show what are the
>> >> >> >>> current problems and what have been fixed, it will also provide
>> a
>> >> very
>> >> >> >> good
>> >> >> >>> view for contributors that are intrested to working on
>> >> >> >>> this. We can finally merge back the branch to trunk until the
>> >> >> community
>> >> >> >>> thinks it is good enough and stable enough. We can donate
>> >> >> >>> ARM machines to the existing CI system for the job.
>> >> >> >>>
>> >> >> >>> I wonder if this approch possible?
>> >> >> >>>
>> >> >> >>> BR,
>> >> >> >>>
>> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>> >> liusheng2048@gmail.com>
>> >> >> >>> wrote:
>> >> >> >>>
>> >> >> >>>> Hi,
>> >> >> >>>>
>> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>> >> community
>> >> >> >>>> mentioned by Vinay. I am working on building and
>> >> >> >>>> testing Hadoop components on aarch64 server these days, besides
>> >> the
>> >> >> >>> missing
>> >> >> >>>> dependices of ARM platform issues #1 #2 #3
>> >> >> >>>> mentioned by Vinay, other similar issue has also be found,
>> such as
>> >> >> the
>> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >> >> >>>>
>> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>> >> hoped to
>> >> >> >> add
>> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
>> >> >> >>>> sure about if there is any potential effect or confilict on the
>> >> trunk
>> >> >> >>>> branch, so maybe creating a ARM specific branch for doing these
>> >> stuff
>> >> >> >>>> is a better choice, what do you think?
>> >> >> >>>>
>> >> >> >>>> Hope to hear thoughts from you :)
>> >> >> >>>>
>> >> >> >>>> BR,
>> >> >> >>>> Liu sheng
>> >> >> >>>>
>> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二
>> 上午5:34写道：
>> >> >> >>>>
>> >> >> >>>>> Hi Folks,
>> >> >> >>>>>
>> >> >> >>>>> ARM is becoming famous lately in its processing capability and
>> >> has
>> >> >> >> got
>> >> >> >>>> the
>> >> >> >>>>> potential to run Bigdata workloads.
>> >> >> >>>>> Many users have been moving to ARM machines due to its low
>> cost.
>> >> >> >>>>>
>> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>> >> (Rasberry
>> >> >> >> PI)
>> >> >> >>>> for
>> >> >> >>>>> experimental purposes. Today ARM architecture is taking some
>> of
>> >> the
>> >> >> >>>>> serverside processing as well. So there will be/is a real
>> need of
>> >> >> >>> Hadoop
>> >> >> >>>> to
>> >> >> >>>>> support ARM architecture as well.
>> >> >> >>>>>
>> >> >> >>>>> There are bunch of users who are trying out building Hadoop on
>> >> ARM,
>> >> >> >>>> trying
>> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>> >> >> >>>>>
>> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
>> issues,
>> >> >> >> found
>> >> >> >>>> from
>> >> >> >>>>> testing done in openlab in [2].
>> >> >> >>>>>
>> >> >> >>>>> 1. Protobuf :
>> >> >> >>>>> -------------------
>> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>> >> protobuf
>> >> >> >>>> 2.5.0
>> >> >> >>>>> version, due to backward compatibility reasons.
>> Protobuf-2.5.0 is
>> >> >> not
>> >> >> >>>> being
>> >> >> >>>>> maintained in the community. While protobuf 3.x is being
>> actively
>> >> >> >>> adopted
>> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
>> proto2
>> >> >> >>>> messages.
>> >> >> >>>>> Due to some compilation issues in the generated java code,
>> which
>> >> can
>> >> >> >>>> induce
>> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
>> from
>> >> >> >> 2.5.0
>> >> >> >>>> was
>> >> >> >>>>> not taken up.
>> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to
>> avoid
>> >> >> >>> classpath
>> >> >> >>>>> problem in downstream projects.
>> >> >> >>>>>    There are patches available to fix compilation in Hadoop.
>> But
>> >> >> >> need
>> >> >> >>> to
>> >> >> >>>>> find a way to upgrade protobuf to latest version and still
>> >> maintain
>> >> >> >> the
>> >> >> >>>>> downstream's classpath using shading feature of Hadoop build.
>> >> >> >>>>>
>> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even
>> before
>> >> >> >> shade
>> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>> >> >> >> continue
>> >> >> >>>>> explore possibilities.
>> >> >> >>>>>
>> >> >> >>>>> 2. leveldbjni:
>> >> >> >>>>> ---------------
>> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>> >> architecture,
>> >> >> >>>> need
>> >> >> >>>>> to check whether any of the future versions support ARM and
>> can
>> >> >> >> hadoop
>> >> >> >>>>> upgrade to that version.
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>> >> >> >>>>> -------------------------
>> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable
>> by
>> >> >> >>> default
>> >> >> >>>> in
>> >> >> >>>>> the maven repository. Workaround is to build it locally and
>> keep
>> >> in
>> >> >> >>> local
>> >> >> >>>>> maven repository.
>> >> >> >>>>> Need to check whether any future versions of
>> >> 'protoc-gen-grpc-java'
>> >> >> >> is
>> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade
>> it?
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> Once the compilation issues are solved, then there might be
>> many
>> >> >> >> native
>> >> >> >>>>> code related issues due to different architectures.
>> >> >> >>>>> So to explore everything, need to join hands together and
>> >> proceed.
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> Let us discuss and check, whether any body else out there who
>> >> also
>> >> >> >> need
>> >> >> >>>> the
>> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
>> >> hands
>> >> >> >>> and
>> >> >> >>>>> time in this work.
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>> >> >> >>>>> [2]
>> >> >> >>
>> >> >>
>> >>
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>> >> >> >>>>>
>> >> >> >>>>> -Vinay
>> >> >> >>
>> >> >>
>> >> >
>> >>
>> >
>>
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B <vi...@apache.org>]

Thanks @Anu

I understand the concern. I took it in different manner.

Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
changes as early as possible, its better to do it trunk itself.

I was able to come to successfull attempt of upgrading protobuf as per
suggestion of stack in Jira
https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
 .

Have created the PR. Please review.  Changes looks huge, because all
references of "com.google.protobuf" relocated to
"o.a.h.shaded.com.google.protobuf".
Otherwise changes are reasonable.

This change is with still keeping the current 2.5.0 dependency, for
downstream builds. So essentially nothing should be changed for downstreams.

-Vinay


On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer <ae...@cloudera.com> wrote:

> Yes, I think that is what Sunil and I are trying to suggest; the complex
> dependencies like Protobuf, if you do it in the trunk you have a better
> change of getting it done. Otherwise, at merge point random downstream
> applications which you have never heard of will object, and Hadoop
> compatibility rules are very clear so you cannot fix it.
>
> With that said, even doing this in the trunk is complex; It might be good
> for you to host a meeting and get some feedback. I have openly said it is a
> great idea like "belling the cat", but the effort is in getting the
> community to agree and align. Solve that, most of your technical problems
> will be easier to solve.
>
> If you go into a branch, it might be that the community might forget about
> your work; and when you come in to merge you will see issues which you did
> not think about.
>
> So, Here is what would be great if you can make this happen; for ARM work,
> get a list of dependencies that needed to be upgraded; see if you can get
> the community aligned with this goal; since ARM might not be in the target
> for many users. You need to convince users that even without ARM, this is a
> great idea.
>
> If you like we can get together during one of the HDFS meetups hosted by
> Wei-chiu.
>
> Thanks
> Anu
>
>
>
> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
> wrote:
>
>> Hi all,
>>
>> Thanks for the response.
>> As I see, protobuf upgrade is long pending and most awaited one.
>>
>> @Sunil
>> Protobuf upgrade looks to be a non-trivial task.
>> Thanks @Duo Zhang for the suggestion of
>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>> problem
>> of dependency on build environment.
>> However more problem lies in upgrade protobuf without breaking the
>> downstream builds.
>> Reason is many downstream projects directly refers server specific jars
>> and
>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>> dependency.
>>
>> There are some historical discussions and suggestions on
>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>> upgrade.
>> Recommended path for solution is, try to upgrade protobuf using shading of
>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>> dependencies for downstreams.
>> I am trying out ideas on this and if it gets completed within time, may be
>> we can target trunk itself for the protobuf upgrade.
>>
>> separate branch idea is suggested for the overall ARM support including
>> protobuf upgrade and other problems mentioned in the discussion above.
>>
>> I dont expect separate branch to have a huge changes, apart from bug
>> fixes,
>> since there are no separate features specific to ARM is being planned.
>> So timely rebase of separate branch would reduce the overhead on branch
>> review/merge task.
>>
>> Still, if the solution to protobuf upgrade winds up early, without any
>> side
>> effects, I am more than happy to land it in trunk itself.
>>
>> Thanks,
>> Vinay
>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:
>>
>> > Thanks Vinay for starting the thread.
>> >
>> > I agree to Anu's view point related to protobuf. And with the suggestion
>> > pointed out by Duo Zhang, if we can make use
>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>> 3.0.0
>> > of protobuf will also be more easier.
>> >
>> > However i think its better to do this effort in trunk itself.
>> > In offline talks, few members were interested to start 3.3.0 release.
>> And
>> > given that happens soon, I feel its better
>> > we do this task in trunk itself as branch diverge is very much possible.
>> > And to bring to call a merge on such a big branch will be even more
>> tough
>> > task.
>> >
>> > my 2 cents.
>> >
>> > Thanks
>> > Sunil
>> >
>> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
>> > wrote:
>> >
>> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>> >> generate
>> >> the protobuf code. It will download the protoc binary from the maven
>> >> central so we do not need to install protoc on the build machine any
>> more.
>> >>
>> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>> >>
>> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been
>> failling
>> >> for
>> >> > over 2 month related to the Protobuf problem .
>> >> > According to the latest successful build log:
>> >> >
>> >>
>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>> >> > the
>> >> > os version was ubuntu 14.04 and for the jobs that are failling now
>> such
>> >> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
>> >> > the os version is 18.04. I'm not very familiar with the version
>> changing
>> >> > for the jobs but I did a little search, according to:
>> >> >
>> >> >
>> >>
>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>> >> > &
>> >> >
>> >> >
>> >>
>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>> >> > it both said that the version of libprotc-dev and protobuf-compiler
>> >> > available for ubuntu 18.04 is 3.0.0
>> >> >
>> >> >
>> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
>> wrote:
>> >> >
>> >> >> Thanx Vinay for the initiative, Makes sense to add support for
>> >> different
>> >> >> architectures.
>> >> >>
>> >> >> +1, for the branch idea.
>> >> >> Good Luck!!!
>> >> >>
>> >> >> -Ayush
>> >> >>
>> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
>> >> >> wrote:
>> >> >> >
>> >> >> > For HBase, we purged all the protobuf related things from the
>> public
>> >> >> API,
>> >> >> > and then upgraded to a shaded and relocated version of protobuf.
>> We
>> >> have
>> >> >> > created a repo for this:
>> >> >> >
>> >> >> > https://github.com/apache/hbase-thirdparty
>> >> >> >
>> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>> >> jars,
>> >> >> our
>> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
>> >> >> discuss
>> >> >> > on how to deal with the upgrading of coprocessor. Glad to see that
>> >> the
>> >> >> > hadoop community is also willing to solve the problem.
>> >> >> >
>> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
>> 上午1:23写道：
>> >> >> >
>> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is
>> proving
>> >> that
>> >> >> >> Hadoop and the downstream projects work correctly after you
>> upgrade
>> >> >> core
>> >> >> >> components like Protobuf.
>> >> >> >> So while branching and working on a branch is easy, merging back
>> >> after
>> >> >> you
>> >> >> >> upgrade some of these core components is insanely hard. You might
>> >> want
>> >> >> to
>> >> >> >> make sure that community buys into upgrading these components in
>> the
>> >> >> trunk.
>> >> >> >> That way we will get testing and downstream components will
>> notice
>> >> when
>> >> >> >> things break.
>> >> >> >>
>> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a
>> really
>> >> long
>> >> >> >> time; I have argued that 2.5 is out of support and we cannot
>> stay on
>> >> >> that
>> >> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>> >> code
>> >> >> base.
>> >> >> >> It has been rightly pointed to me that while all the arguments I
>> >> make
>> >> >> is
>> >> >> >> correct; it is a very complicated task to upgrade Protobuf, and
>> the
>> >> >> worst
>> >> >> >> part is we will not even know what breaks until downstream
>> projects
>> >> >> pick up
>> >> >> >> these changes and work against us.
>> >> >> >>
>> >> >> >> If we work off the Hadoop version 3 — and assume that we have
>> >> >> "shading" in
>> >> >> >> place for all deployments; it might be possible to get there;
>> still
>> >> a
>> >> >> >> daunting task.
>> >> >> >>
>> >> >> >> So best of luck with the branch approach — But please remember,
>> >> Merging
>> >> >> >> back will be hard, Just my 2 cents.
>> >> >> >>
>> >> >> >> — Anu
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>> >> zhengzhenyulixi@gmail.com
>> >> >> >
>> >> >> >> wrote:
>> >> >> >>
>> >> >> >>> Hi,
>> >> >> >>>
>> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
>> >> >> separate
>> >> >> >>> branch with it's own ARM CI seems a really good idea.
>> >> >> >>> By doing this we won't break any of the undergoing development
>> in
>> >> >> trunk
>> >> >> >> and
>> >> >> >>> a CI can be a very good way to show what are the
>> >> >> >>> current problems and what have been fixed, it will also provide
>> a
>> >> very
>> >> >> >> good
>> >> >> >>> view for contributors that are intrested to working on
>> >> >> >>> this. We can finally merge back the branch to trunk until the
>> >> >> community
>> >> >> >>> thinks it is good enough and stable enough. We can donate
>> >> >> >>> ARM machines to the existing CI system for the job.
>> >> >> >>>
>> >> >> >>> I wonder if this approch possible?
>> >> >> >>>
>> >> >> >>> BR,
>> >> >> >>>
>> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>> >> liusheng2048@gmail.com>
>> >> >> >>> wrote:
>> >> >> >>>
>> >> >> >>>> Hi,
>> >> >> >>>>
>> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>> >> community
>> >> >> >>>> mentioned by Vinay. I am working on building and
>> >> >> >>>> testing Hadoop components on aarch64 server these days, besides
>> >> the
>> >> >> >>> missing
>> >> >> >>>> dependices of ARM platform issues #1 #2 #3
>> >> >> >>>> mentioned by Vinay, other similar issue has also be found,
>> such as
>> >> >> the
>> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >> >> >>>>
>> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>> >> hoped to
>> >> >> >> add
>> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
>> >> >> >>>> sure about if there is any potential effect or confilict on the
>> >> trunk
>> >> >> >>>> branch, so maybe creating a ARM specific branch for doing these
>> >> stuff
>> >> >> >>>> is a better choice, what do you think?
>> >> >> >>>>
>> >> >> >>>> Hope to hear thoughts from you :)
>> >> >> >>>>
>> >> >> >>>> BR,
>> >> >> >>>> Liu sheng
>> >> >> >>>>
>> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二
>> 上午5:34写道：
>> >> >> >>>>
>> >> >> >>>>> Hi Folks,
>> >> >> >>>>>
>> >> >> >>>>> ARM is becoming famous lately in its processing capability and
>> >> has
>> >> >> >> got
>> >> >> >>>> the
>> >> >> >>>>> potential to run Bigdata workloads.
>> >> >> >>>>> Many users have been moving to ARM machines due to its low
>> cost.
>> >> >> >>>>>
>> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>> >> (Rasberry
>> >> >> >> PI)
>> >> >> >>>> for
>> >> >> >>>>> experimental purposes. Today ARM architecture is taking some
>> of
>> >> the
>> >> >> >>>>> serverside processing as well. So there will be/is a real
>> need of
>> >> >> >>> Hadoop
>> >> >> >>>> to
>> >> >> >>>>> support ARM architecture as well.
>> >> >> >>>>>
>> >> >> >>>>> There are bunch of users who are trying out building Hadoop on
>> >> ARM,
>> >> >> >>>> trying
>> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>> >> >> >>>>>
>> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
>> issues,
>> >> >> >> found
>> >> >> >>>> from
>> >> >> >>>>> testing done in openlab in [2].
>> >> >> >>>>>
>> >> >> >>>>> 1. Protobuf :
>> >> >> >>>>> -------------------
>> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>> >> protobuf
>> >> >> >>>> 2.5.0
>> >> >> >>>>> version, due to backward compatibility reasons.
>> Protobuf-2.5.0 is
>> >> >> not
>> >> >> >>>> being
>> >> >> >>>>> maintained in the community. While protobuf 3.x is being
>> actively
>> >> >> >>> adopted
>> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
>> proto2
>> >> >> >>>> messages.
>> >> >> >>>>> Due to some compilation issues in the generated java code,
>> which
>> >> can
>> >> >> >>>> induce
>> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
>> from
>> >> >> >> 2.5.0
>> >> >> >>>> was
>> >> >> >>>>> not taken up.
>> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to
>> avoid
>> >> >> >>> classpath
>> >> >> >>>>> problem in downstream projects.
>> >> >> >>>>>    There are patches available to fix compilation in Hadoop.
>> But
>> >> >> >> need
>> >> >> >>> to
>> >> >> >>>>> find a way to upgrade protobuf to latest version and still
>> >> maintain
>> >> >> >> the
>> >> >> >>>>> downstream's classpath using shading feature of Hadoop build.
>> >> >> >>>>>
>> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even
>> before
>> >> >> >> shade
>> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>> >> >> >> continue
>> >> >> >>>>> explore possibilities.
>> >> >> >>>>>
>> >> >> >>>>> 2. leveldbjni:
>> >> >> >>>>> ---------------
>> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>> >> architecture,
>> >> >> >>>> need
>> >> >> >>>>> to check whether any of the future versions support ARM and
>> can
>> >> >> >> hadoop
>> >> >> >>>>> upgrade to that version.
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>> >> >> >>>>> -------------------------
>> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable
>> by
>> >> >> >>> default
>> >> >> >>>> in
>> >> >> >>>>> the maven repository. Workaround is to build it locally and
>> keep
>> >> in
>> >> >> >>> local
>> >> >> >>>>> maven repository.
>> >> >> >>>>> Need to check whether any future versions of
>> >> 'protoc-gen-grpc-java'
>> >> >> >> is
>> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade
>> it?
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> Once the compilation issues are solved, then there might be
>> many
>> >> >> >> native
>> >> >> >>>>> code related issues due to different architectures.
>> >> >> >>>>> So to explore everything, need to join hands together and
>> >> proceed.
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> Let us discuss and check, whether any body else out there who
>> >> also
>> >> >> >> need
>> >> >> >>>> the
>> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
>> >> hands
>> >> >> >>> and
>> >> >> >>>>> time in this work.
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>> >> >> >>>>> [2]
>> >> >> >>
>> >> >>
>> >>
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>> >> >> >>>>>
>> >> >> >>>>> -Vinay
>> >> >> >>
>> >> >>
>> >> >
>> >>
>> >
>>
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Anu Engineer <ae...@cloudera.com.INVALID>]

Yes, I think that is what Sunil and I are trying to suggest; the complex
dependencies like Protobuf, if you do it in the trunk you have a better
change of getting it done. Otherwise, at merge point random downstream
applications which you have never heard of will object, and Hadoop
compatibility rules are very clear so you cannot fix it.

With that said, even doing this in the trunk is complex; It might be good
for you to host a meeting and get some feedback. I have openly said it is a
great idea like "belling the cat", but the effort is in getting the
community to agree and align. Solve that, most of your technical problems
will be easier to solve.

If you go into a branch, it might be that the community might forget about
your work; and when you come in to merge you will see issues which you did
not think about.

So, Here is what would be great if you can make this happen; for ARM work,
get a list of dependencies that needed to be upgraded; see if you can get
the community aligned with this goal; since ARM might not be in the target
for many users. You need to convince users that even without ARM, this is a
great idea.

If you like we can get together during one of the HDFS meetups hosted by
Wei-chiu.

Thanks
Anu



On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
wrote:

> Hi all,
>
> Thanks for the response.
> As I see, protobuf upgrade is long pending and most awaited one.
>
> @Sunil
> Protobuf upgrade looks to be a non-trivial task.
> Thanks @Duo Zhang for the suggestion of
> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the problem
> of dependency on build environment.
> However more problem lies in upgrade protobuf without breaking the
> downstream builds.
> Reason is many downstream projects directly refers server specific jars and
> they expect protobuf-2.5.0 jar to get added to classpath by transitive
> dependency.
>
> There are some historical discussions and suggestions on
> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
> upgrade.
> Recommended path for solution is, try to upgrade protobuf using shading of
> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
> dependencies for downstreams.
> I am trying out ideas on this and if it gets completed within time, may be
> we can target trunk itself for the protobuf upgrade.
>
> separate branch idea is suggested for the overall ARM support including
> protobuf upgrade and other problems mentioned in the discussion above.
>
> I dont expect separate branch to have a huge changes, apart from bug fixes,
> since there are no separate features specific to ARM is being planned.
> So timely rebase of separate branch would reduce the overhead on branch
> review/merge task.
>
> Still, if the solution to protobuf upgrade winds up early, without any side
> effects, I am more than happy to land it in trunk itself.
>
> Thanks,
> Vinay
> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:
>
> > Thanks Vinay for starting the thread.
> >
> > I agree to Anu's view point related to protobuf. And with the suggestion
> > pointed out by Duo Zhang, if we can make use
> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
> > of protobuf will also be more easier.
> >
> > However i think its better to do this effort in trunk itself.
> > In offline talks, few members were interested to start 3.3.0 release. And
> > given that happens soon, I feel its better
> > we do this task in trunk itself as branch diverge is very much possible.
> > And to bring to call a merge on such a big branch will be even more tough
> > task.
> >
> > my 2 cents.
> >
> > Thanks
> > Sunil
> >
> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> >
> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
> >> generate
> >> the protobuf code. It will download the protoc binary from the maven
> >> central so we do not need to install protoc on the build machine any
> more.
> >>
> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
> >>
> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
> >> for
> >> > over 2 month related to the Protobuf problem .
> >> > According to the latest successful build log:
> >> >
> >>
> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
> >> > the
> >> > os version was ubuntu 14.04 and for the jobs that are failling now
> such
> >> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
> >> > the os version is 18.04. I'm not very familiar with the version
> changing
> >> > for the jobs but I did a little search, according to:
> >> >
> >> >
> >>
> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
> >> > &
> >> >
> >> >
> >>
> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
> >> > it both said that the version of libprotc-dev and protobuf-compiler
> >> > available for ubuntu 18.04 is 3.0.0
> >> >
> >> >
> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
> wrote:
> >> >
> >> >> Thanx Vinay for the initiative, Makes sense to add support for
> >> different
> >> >> architectures.
> >> >>
> >> >> +1, for the branch idea.
> >> >> Good Luck!!!
> >> >>
> >> >> -Ayush
> >> >>
> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> > For HBase, we purged all the protobuf related things from the
> public
> >> >> API,
> >> >> > and then upgraded to a shaded and relocated version of protobuf. We
> >> have
> >> >> > created a repo for this:
> >> >> >
> >> >> > https://github.com/apache/hbase-thirdparty
> >> >> >
> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
> >> jars,
> >> >> our
> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
> >> >> discuss
> >> >> > on how to deal with the upgrading of coprocessor. Glad to see that
> >> the
> >> >> > hadoop community is also willing to solve the problem.
> >> >> >
> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
> 上午1:23写道：
> >> >> >
> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
> >> that
> >> >> >> Hadoop and the downstream projects work correctly after you
> upgrade
> >> >> core
> >> >> >> components like Protobuf.
> >> >> >> So while branching and working on a branch is easy, merging back
> >> after
> >> >> you
> >> >> >> upgrade some of these core components is insanely hard. You might
> >> want
> >> >> to
> >> >> >> make sure that community buys into upgrading these components in
> the
> >> >> trunk.
> >> >> >> That way we will get testing and downstream components will notice
> >> when
> >> >> >> things break.
> >> >> >>
> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
> >> long
> >> >> >> time; I have argued that 2.5 is out of support and we cannot stay
> on
> >> >> that
> >> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
> >> code
> >> >> base.
> >> >> >> It has been rightly pointed to me that while all the arguments I
> >> make
> >> >> is
> >> >> >> correct; it is a very complicated task to upgrade Protobuf, and
> the
> >> >> worst
> >> >> >> part is we will not even know what breaks until downstream
> projects
> >> >> pick up
> >> >> >> these changes and work against us.
> >> >> >>
> >> >> >> If we work off the Hadoop version 3 — and assume that we have
> >> >> "shading" in
> >> >> >> place for all deployments; it might be possible to get there;
> still
> >> a
> >> >> >> daunting task.
> >> >> >>
> >> >> >> So best of luck with the branch approach — But please remember,
> >> Merging
> >> >> >> back will be hard, Just my 2 cents.
> >> >> >>
> >> >> >> — Anu
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
> >> zhengzhenyulixi@gmail.com
> >> >> >
> >> >> >> wrote:
> >> >> >>
> >> >> >>> Hi,
> >> >> >>>
> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
> >> >> separate
> >> >> >>> branch with it's own ARM CI seems a really good idea.
> >> >> >>> By doing this we won't break any of the undergoing development in
> >> >> trunk
> >> >> >> and
> >> >> >>> a CI can be a very good way to show what are the
> >> >> >>> current problems and what have been fixed, it will also provide a
> >> very
> >> >> >> good
> >> >> >>> view for contributors that are intrested to working on
> >> >> >>> this. We can finally merge back the branch to trunk until the
> >> >> community
> >> >> >>> thinks it is good enough and stable enough. We can donate
> >> >> >>> ARM machines to the existing CI system for the job.
> >> >> >>>
> >> >> >>> I wonder if this approch possible?
> >> >> >>>
> >> >> >>> BR,
> >> >> >>>
> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
> >> liusheng2048@gmail.com>
> >> >> >>> wrote:
> >> >> >>>
> >> >> >>>> Hi,
> >> >> >>>>
> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
> >> community
> >> >> >>>> mentioned by Vinay. I am working on building and
> >> >> >>>> testing Hadoop components on aarch64 server these days, besides
> >> the
> >> >> >>> missing
> >> >> >>>> dependices of ARM platform issues #1 #2 #3
> >> >> >>>> mentioned by Vinay, other similar issue has also be found, such
> as
> >> >> the
> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
> >> >> >>>>
> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
> >> hoped to
> >> >> >> add
> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
> >> >> >>>> sure about if there is any potential effect or confilict on the
> >> trunk
> >> >> >>>> branch, so maybe creating a ARM specific branch for doing these
> >> stuff
> >> >> >>>> is a better choice, what do you think?
> >> >> >>>>
> >> >> >>>> Hope to hear thoughts from you :)
> >> >> >>>>
> >> >> >>>> BR,
> >> >> >>>> Liu sheng
> >> >> >>>>
> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> >> >> >>>>
> >> >> >>>>> Hi Folks,
> >> >> >>>>>
> >> >> >>>>> ARM is becoming famous lately in its processing capability and
> >> has
> >> >> >> got
> >> >> >>>> the
> >> >> >>>>> potential to run Bigdata workloads.
> >> >> >>>>> Many users have been moving to ARM machines due to its low
> cost.
> >> >> >>>>>
> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
> >> (Rasberry
> >> >> >> PI)
> >> >> >>>> for
> >> >> >>>>> experimental purposes. Today ARM architecture is taking some of
> >> the
> >> >> >>>>> serverside processing as well. So there will be/is a real need
> of
> >> >> >>> Hadoop
> >> >> >>>> to
> >> >> >>>>> support ARM architecture as well.
> >> >> >>>>>
> >> >> >>>>> There are bunch of users who are trying out building Hadoop on
> >> ARM,
> >> >> >>>> trying
> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
> >> >> >>>>>
> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
> issues,
> >> >> >> found
> >> >> >>>> from
> >> >> >>>>> testing done in openlab in [2].
> >> >> >>>>>
> >> >> >>>>> 1. Protobuf :
> >> >> >>>>> -------------------
> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
> >> protobuf
> >> >> >>>> 2.5.0
> >> >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0
> is
> >> >> not
> >> >> >>>> being
> >> >> >>>>> maintained in the community. While protobuf 3.x is being
> actively
> >> >> >>> adopted
> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
> proto2
> >> >> >>>> messages.
> >> >> >>>>> Due to some compilation issues in the generated java code,
> which
> >> can
> >> >> >>>> induce
> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
> from
> >> >> >> 2.5.0
> >> >> >>>> was
> >> >> >>>>> not taken up.
> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> >> >> >>> classpath
> >> >> >>>>> problem in downstream projects.
> >> >> >>>>>    There are patches available to fix compilation in Hadoop.
> But
> >> >> >> need
> >> >> >>> to
> >> >> >>>>> find a way to upgrade protobuf to latest version and still
> >> maintain
> >> >> >> the
> >> >> >>>>> downstream's classpath using shading feature of Hadoop build.
> >> >> >>>>>
> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even before
> >> >> >> shade
> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
> >> >> >> continue
> >> >> >>>>> explore possibilities.
> >> >> >>>>>
> >> >> >>>>> 2. leveldbjni:
> >> >> >>>>> ---------------
> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
> >> architecture,
> >> >> >>>> need
> >> >> >>>>> to check whether any of the future versions support ARM and can
> >> >> >> hadoop
> >> >> >>>>> upgrade to that version.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> >> >> >>>>> -------------------------
> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable
> by
> >> >> >>> default
> >> >> >>>> in
> >> >> >>>>> the maven repository. Workaround is to build it locally and
> keep
> >> in
> >> >> >>> local
> >> >> >>>>> maven repository.
> >> >> >>>>> Need to check whether any future versions of
> >> 'protoc-gen-grpc-java'
> >> >> >> is
> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade
> it?
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> Once the compilation issues are solved, then there might be
> many
> >> >> >> native
> >> >> >>>>> code related issues due to different architectures.
> >> >> >>>>> So to explore everything, need to join hands together and
> >> proceed.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> Let us discuss and check, whether any body else out there who
> >> also
> >> >> >> need
> >> >> >>>> the
> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
> >> hands
> >> >> >>> and
> >> >> >>>>> time in this work.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
> >> >> >>>>> [2]
> >> >> >>
> >> >>
> >>
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
> >> >> >>>>>
> >> >> >>>>> -Vinay
> >> >> >>
> >> >>
> >> >
> >>
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Anu Engineer <ae...@cloudera.com.INVALID>]

Yes, I think that is what Sunil and I are trying to suggest; the complex
dependencies like Protobuf, if you do it in the trunk you have a better
change of getting it done. Otherwise, at merge point random downstream
applications which you have never heard of will object, and Hadoop
compatibility rules are very clear so you cannot fix it.

With that said, even doing this in the trunk is complex; It might be good
for you to host a meeting and get some feedback. I have openly said it is a
great idea like "belling the cat", but the effort is in getting the
community to agree and align. Solve that, most of your technical problems
will be easier to solve.

If you go into a branch, it might be that the community might forget about
your work; and when you come in to merge you will see issues which you did
not think about.

So, Here is what would be great if you can make this happen; for ARM work,
get a list of dependencies that needed to be upgraded; see if you can get
the community aligned with this goal; since ARM might not be in the target
for many users. You need to convince users that even without ARM, this is a
great idea.

If you like we can get together during one of the HDFS meetups hosted by
Wei-chiu.

Thanks
Anu



On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
wrote:

> Hi all,
>
> Thanks for the response.
> As I see, protobuf upgrade is long pending and most awaited one.
>
> @Sunil
> Protobuf upgrade looks to be a non-trivial task.
> Thanks @Duo Zhang for the suggestion of
> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the problem
> of dependency on build environment.
> However more problem lies in upgrade protobuf without breaking the
> downstream builds.
> Reason is many downstream projects directly refers server specific jars and
> they expect protobuf-2.5.0 jar to get added to classpath by transitive
> dependency.
>
> There are some historical discussions and suggestions on
> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
> upgrade.
> Recommended path for solution is, try to upgrade protobuf using shading of
> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
> dependencies for downstreams.
> I am trying out ideas on this and if it gets completed within time, may be
> we can target trunk itself for the protobuf upgrade.
>
> separate branch idea is suggested for the overall ARM support including
> protobuf upgrade and other problems mentioned in the discussion above.
>
> I dont expect separate branch to have a huge changes, apart from bug fixes,
> since there are no separate features specific to ARM is being planned.
> So timely rebase of separate branch would reduce the overhead on branch
> review/merge task.
>
> Still, if the solution to protobuf upgrade winds up early, without any side
> effects, I am more than happy to land it in trunk itself.
>
> Thanks,
> Vinay
> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:
>
> > Thanks Vinay for starting the thread.
> >
> > I agree to Anu's view point related to protobuf. And with the suggestion
> > pointed out by Duo Zhang, if we can make use
> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
> > of protobuf will also be more easier.
> >
> > However i think its better to do this effort in trunk itself.
> > In offline talks, few members were interested to start 3.3.0 release. And
> > given that happens soon, I feel its better
> > we do this task in trunk itself as branch diverge is very much possible.
> > And to bring to call a merge on such a big branch will be even more tough
> > task.
> >
> > my 2 cents.
> >
> > Thanks
> > Sunil
> >
> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> >
> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
> >> generate
> >> the protobuf code. It will download the protoc binary from the maven
> >> central so we do not need to install protoc on the build machine any
> more.
> >>
> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
> >>
> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
> >> for
> >> > over 2 month related to the Protobuf problem .
> >> > According to the latest successful build log:
> >> >
> >>
> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
> >> > the
> >> > os version was ubuntu 14.04 and for the jobs that are failling now
> such
> >> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
> >> > the os version is 18.04. I'm not very familiar with the version
> changing
> >> > for the jobs but I did a little search, according to:
> >> >
> >> >
> >>
> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
> >> > &
> >> >
> >> >
> >>
> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
> >> > it both said that the version of libprotc-dev and protobuf-compiler
> >> > available for ubuntu 18.04 is 3.0.0
> >> >
> >> >
> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
> wrote:
> >> >
> >> >> Thanx Vinay for the initiative, Makes sense to add support for
> >> different
> >> >> architectures.
> >> >>
> >> >> +1, for the branch idea.
> >> >> Good Luck!!!
> >> >>
> >> >> -Ayush
> >> >>
> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> > For HBase, we purged all the protobuf related things from the
> public
> >> >> API,
> >> >> > and then upgraded to a shaded and relocated version of protobuf. We
> >> have
> >> >> > created a repo for this:
> >> >> >
> >> >> > https://github.com/apache/hbase-thirdparty
> >> >> >
> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
> >> jars,
> >> >> our
> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
> >> >> discuss
> >> >> > on how to deal with the upgrading of coprocessor. Glad to see that
> >> the
> >> >> > hadoop community is also willing to solve the problem.
> >> >> >
> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
> 上午1:23写道：
> >> >> >
> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
> >> that
> >> >> >> Hadoop and the downstream projects work correctly after you
> upgrade
> >> >> core
> >> >> >> components like Protobuf.
> >> >> >> So while branching and working on a branch is easy, merging back
> >> after
> >> >> you
> >> >> >> upgrade some of these core components is insanely hard. You might
> >> want
> >> >> to
> >> >> >> make sure that community buys into upgrading these components in
> the
> >> >> trunk.
> >> >> >> That way we will get testing and downstream components will notice
> >> when
> >> >> >> things break.
> >> >> >>
> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
> >> long
> >> >> >> time; I have argued that 2.5 is out of support and we cannot stay
> on
> >> >> that
> >> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
> >> code
> >> >> base.
> >> >> >> It has been rightly pointed to me that while all the arguments I
> >> make
> >> >> is
> >> >> >> correct; it is a very complicated task to upgrade Protobuf, and
> the
> >> >> worst
> >> >> >> part is we will not even know what breaks until downstream
> projects
> >> >> pick up
> >> >> >> these changes and work against us.
> >> >> >>
> >> >> >> If we work off the Hadoop version 3 — and assume that we have
> >> >> "shading" in
> >> >> >> place for all deployments; it might be possible to get there;
> still
> >> a
> >> >> >> daunting task.
> >> >> >>
> >> >> >> So best of luck with the branch approach — But please remember,
> >> Merging
> >> >> >> back will be hard, Just my 2 cents.
> >> >> >>
> >> >> >> — Anu
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
> >> zhengzhenyulixi@gmail.com
> >> >> >
> >> >> >> wrote:
> >> >> >>
> >> >> >>> Hi,
> >> >> >>>
> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
> >> >> separate
> >> >> >>> branch with it's own ARM CI seems a really good idea.
> >> >> >>> By doing this we won't break any of the undergoing development in
> >> >> trunk
> >> >> >> and
> >> >> >>> a CI can be a very good way to show what are the
> >> >> >>> current problems and what have been fixed, it will also provide a
> >> very
> >> >> >> good
> >> >> >>> view for contributors that are intrested to working on
> >> >> >>> this. We can finally merge back the branch to trunk until the
> >> >> community
> >> >> >>> thinks it is good enough and stable enough. We can donate
> >> >> >>> ARM machines to the existing CI system for the job.
> >> >> >>>
> >> >> >>> I wonder if this approch possible?
> >> >> >>>
> >> >> >>> BR,
> >> >> >>>
> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
> >> liusheng2048@gmail.com>
> >> >> >>> wrote:
> >> >> >>>
> >> >> >>>> Hi,
> >> >> >>>>
> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
> >> community
> >> >> >>>> mentioned by Vinay. I am working on building and
> >> >> >>>> testing Hadoop components on aarch64 server these days, besides
> >> the
> >> >> >>> missing
> >> >> >>>> dependices of ARM platform issues #1 #2 #3
> >> >> >>>> mentioned by Vinay, other similar issue has also be found, such
> as
> >> >> the
> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
> >> >> >>>>
> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
> >> hoped to
> >> >> >> add
> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
> >> >> >>>> sure about if there is any potential effect or confilict on the
> >> trunk
> >> >> >>>> branch, so maybe creating a ARM specific branch for doing these
> >> stuff
> >> >> >>>> is a better choice, what do you think?
> >> >> >>>>
> >> >> >>>> Hope to hear thoughts from you :)
> >> >> >>>>
> >> >> >>>> BR,
> >> >> >>>> Liu sheng
> >> >> >>>>
> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> >> >> >>>>
> >> >> >>>>> Hi Folks,
> >> >> >>>>>
> >> >> >>>>> ARM is becoming famous lately in its processing capability and
> >> has
> >> >> >> got
> >> >> >>>> the
> >> >> >>>>> potential to run Bigdata workloads.
> >> >> >>>>> Many users have been moving to ARM machines due to its low
> cost.
> >> >> >>>>>
> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
> >> (Rasberry
> >> >> >> PI)
> >> >> >>>> for
> >> >> >>>>> experimental purposes. Today ARM architecture is taking some of
> >> the
> >> >> >>>>> serverside processing as well. So there will be/is a real need
> of
> >> >> >>> Hadoop
> >> >> >>>> to
> >> >> >>>>> support ARM architecture as well.
> >> >> >>>>>
> >> >> >>>>> There are bunch of users who are trying out building Hadoop on
> >> ARM,
> >> >> >>>> trying
> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
> >> >> >>>>>
> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
> issues,
> >> >> >> found
> >> >> >>>> from
> >> >> >>>>> testing done in openlab in [2].
> >> >> >>>>>
> >> >> >>>>> 1. Protobuf :
> >> >> >>>>> -------------------
> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
> >> protobuf
> >> >> >>>> 2.5.0
> >> >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0
> is
> >> >> not
> >> >> >>>> being
> >> >> >>>>> maintained in the community. While protobuf 3.x is being
> actively
> >> >> >>> adopted
> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
> proto2
> >> >> >>>> messages.
> >> >> >>>>> Due to some compilation issues in the generated java code,
> which
> >> can
> >> >> >>>> induce
> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
> from
> >> >> >> 2.5.0
> >> >> >>>> was
> >> >> >>>>> not taken up.
> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> >> >> >>> classpath
> >> >> >>>>> problem in downstream projects.
> >> >> >>>>>    There are patches available to fix compilation in Hadoop.
> But
> >> >> >> need
> >> >> >>> to
> >> >> >>>>> find a way to upgrade protobuf to latest version and still
> >> maintain
> >> >> >> the
> >> >> >>>>> downstream's classpath using shading feature of Hadoop build.
> >> >> >>>>>
> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even before
> >> >> >> shade
> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
> >> >> >> continue
> >> >> >>>>> explore possibilities.
> >> >> >>>>>
> >> >> >>>>> 2. leveldbjni:
> >> >> >>>>> ---------------
> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
> >> architecture,
> >> >> >>>> need
> >> >> >>>>> to check whether any of the future versions support ARM and can
> >> >> >> hadoop
> >> >> >>>>> upgrade to that version.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> >> >> >>>>> -------------------------
> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable
> by
> >> >> >>> default
> >> >> >>>> in
> >> >> >>>>> the maven repository. Workaround is to build it locally and
> keep
> >> in
> >> >> >>> local
> >> >> >>>>> maven repository.
> >> >> >>>>> Need to check whether any future versions of
> >> 'protoc-gen-grpc-java'
> >> >> >> is
> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade
> it?
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> Once the compilation issues are solved, then there might be
> many
> >> >> >> native
> >> >> >>>>> code related issues due to different architectures.
> >> >> >>>>> So to explore everything, need to join hands together and
> >> proceed.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> Let us discuss and check, whether any body else out there who
> >> also
> >> >> >> need
> >> >> >>>> the
> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
> >> hands
> >> >> >>> and
> >> >> >>>>> time in this work.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
> >> >> >>>>> [2]
> >> >> >>
> >> >>
> >>
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
> >> >> >>>>>
> >> >> >>>>> -Vinay
> >> >> >>
> >> >>
> >> >
> >>
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Anu Engineer <ae...@cloudera.com.INVALID>]

Yes, I think that is what Sunil and I are trying to suggest; the complex
dependencies like Protobuf, if you do it in the trunk you have a better
change of getting it done. Otherwise, at merge point random downstream
applications which you have never heard of will object, and Hadoop
compatibility rules are very clear so you cannot fix it.

With that said, even doing this in the trunk is complex; It might be good
for you to host a meeting and get some feedback. I have openly said it is a
great idea like "belling the cat", but the effort is in getting the
community to agree and align. Solve that, most of your technical problems
will be easier to solve.

If you go into a branch, it might be that the community might forget about
your work; and when you come in to merge you will see issues which you did
not think about.

So, Here is what would be great if you can make this happen; for ARM work,
get a list of dependencies that needed to be upgraded; see if you can get
the community aligned with this goal; since ARM might not be in the target
for many users. You need to convince users that even without ARM, this is a
great idea.

If you like we can get together during one of the HDFS meetups hosted by
Wei-chiu.

Thanks
Anu



On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
wrote:

> Hi all,
>
> Thanks for the response.
> As I see, protobuf upgrade is long pending and most awaited one.
>
> @Sunil
> Protobuf upgrade looks to be a non-trivial task.
> Thanks @Duo Zhang for the suggestion of
> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the problem
> of dependency on build environment.
> However more problem lies in upgrade protobuf without breaking the
> downstream builds.
> Reason is many downstream projects directly refers server specific jars and
> they expect protobuf-2.5.0 jar to get added to classpath by transitive
> dependency.
>
> There are some historical discussions and suggestions on
> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
> upgrade.
> Recommended path for solution is, try to upgrade protobuf using shading of
> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
> dependencies for downstreams.
> I am trying out ideas on this and if it gets completed within time, may be
> we can target trunk itself for the protobuf upgrade.
>
> separate branch idea is suggested for the overall ARM support including
> protobuf upgrade and other problems mentioned in the discussion above.
>
> I dont expect separate branch to have a huge changes, apart from bug fixes,
> since there are no separate features specific to ARM is being planned.
> So timely rebase of separate branch would reduce the overhead on branch
> review/merge task.
>
> Still, if the solution to protobuf upgrade winds up early, without any side
> effects, I am more than happy to land it in trunk itself.
>
> Thanks,
> Vinay
> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:
>
> > Thanks Vinay for starting the thread.
> >
> > I agree to Anu's view point related to protobuf. And with the suggestion
> > pointed out by Duo Zhang, if we can make use
> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
> > of protobuf will also be more easier.
> >
> > However i think its better to do this effort in trunk itself.
> > In offline talks, few members were interested to start 3.3.0 release. And
> > given that happens soon, I feel its better
> > we do this task in trunk itself as branch diverge is very much possible.
> > And to bring to call a merge on such a big branch will be even more tough
> > task.
> >
> > my 2 cents.
> >
> > Thanks
> > Sunil
> >
> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> >
> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
> >> generate
> >> the protobuf code. It will download the protoc binary from the maven
> >> central so we do not need to install protoc on the build machine any
> more.
> >>
> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
> >>
> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
> >> for
> >> > over 2 month related to the Protobuf problem .
> >> > According to the latest successful build log:
> >> >
> >>
> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
> >> > the
> >> > os version was ubuntu 14.04 and for the jobs that are failling now
> such
> >> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
> >> > the os version is 18.04. I'm not very familiar with the version
> changing
> >> > for the jobs but I did a little search, according to:
> >> >
> >> >
> >>
> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
> >> > &
> >> >
> >> >
> >>
> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
> >> > it both said that the version of libprotc-dev and protobuf-compiler
> >> > available for ubuntu 18.04 is 3.0.0
> >> >
> >> >
> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
> wrote:
> >> >
> >> >> Thanx Vinay for the initiative, Makes sense to add support for
> >> different
> >> >> architectures.
> >> >>
> >> >> +1, for the branch idea.
> >> >> Good Luck!!!
> >> >>
> >> >> -Ayush
> >> >>
> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> > For HBase, we purged all the protobuf related things from the
> public
> >> >> API,
> >> >> > and then upgraded to a shaded and relocated version of protobuf. We
> >> have
> >> >> > created a repo for this:
> >> >> >
> >> >> > https://github.com/apache/hbase-thirdparty
> >> >> >
> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
> >> jars,
> >> >> our
> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
> >> >> discuss
> >> >> > on how to deal with the upgrading of coprocessor. Glad to see that
> >> the
> >> >> > hadoop community is also willing to solve the problem.
> >> >> >
> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
> 上午1:23写道：
> >> >> >
> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
> >> that
> >> >> >> Hadoop and the downstream projects work correctly after you
> upgrade
> >> >> core
> >> >> >> components like Protobuf.
> >> >> >> So while branching and working on a branch is easy, merging back
> >> after
> >> >> you
> >> >> >> upgrade some of these core components is insanely hard. You might
> >> want
> >> >> to
> >> >> >> make sure that community buys into upgrading these components in
> the
> >> >> trunk.
> >> >> >> That way we will get testing and downstream components will notice
> >> when
> >> >> >> things break.
> >> >> >>
> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
> >> long
> >> >> >> time; I have argued that 2.5 is out of support and we cannot stay
> on
> >> >> that
> >> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
> >> code
> >> >> base.
> >> >> >> It has been rightly pointed to me that while all the arguments I
> >> make
> >> >> is
> >> >> >> correct; it is a very complicated task to upgrade Protobuf, and
> the
> >> >> worst
> >> >> >> part is we will not even know what breaks until downstream
> projects
> >> >> pick up
> >> >> >> these changes and work against us.
> >> >> >>
> >> >> >> If we work off the Hadoop version 3 — and assume that we have
> >> >> "shading" in
> >> >> >> place for all deployments; it might be possible to get there;
> still
> >> a
> >> >> >> daunting task.
> >> >> >>
> >> >> >> So best of luck with the branch approach — But please remember,
> >> Merging
> >> >> >> back will be hard, Just my 2 cents.
> >> >> >>
> >> >> >> — Anu
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
> >> zhengzhenyulixi@gmail.com
> >> >> >
> >> >> >> wrote:
> >> >> >>
> >> >> >>> Hi,
> >> >> >>>
> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
> >> >> separate
> >> >> >>> branch with it's own ARM CI seems a really good idea.
> >> >> >>> By doing this we won't break any of the undergoing development in
> >> >> trunk
> >> >> >> and
> >> >> >>> a CI can be a very good way to show what are the
> >> >> >>> current problems and what have been fixed, it will also provide a
> >> very
> >> >> >> good
> >> >> >>> view for contributors that are intrested to working on
> >> >> >>> this. We can finally merge back the branch to trunk until the
> >> >> community
> >> >> >>> thinks it is good enough and stable enough. We can donate
> >> >> >>> ARM machines to the existing CI system for the job.
> >> >> >>>
> >> >> >>> I wonder if this approch possible?
> >> >> >>>
> >> >> >>> BR,
> >> >> >>>
> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
> >> liusheng2048@gmail.com>
> >> >> >>> wrote:
> >> >> >>>
> >> >> >>>> Hi,
> >> >> >>>>
> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
> >> community
> >> >> >>>> mentioned by Vinay. I am working on building and
> >> >> >>>> testing Hadoop components on aarch64 server these days, besides
> >> the
> >> >> >>> missing
> >> >> >>>> dependices of ARM platform issues #1 #2 #3
> >> >> >>>> mentioned by Vinay, other similar issue has also be found, such
> as
> >> >> the
> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
> >> >> >>>>
> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
> >> hoped to
> >> >> >> add
> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
> >> >> >>>> sure about if there is any potential effect or confilict on the
> >> trunk
> >> >> >>>> branch, so maybe creating a ARM specific branch for doing these
> >> stuff
> >> >> >>>> is a better choice, what do you think?
> >> >> >>>>
> >> >> >>>> Hope to hear thoughts from you :)
> >> >> >>>>
> >> >> >>>> BR,
> >> >> >>>> Liu sheng
> >> >> >>>>
> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> >> >> >>>>
> >> >> >>>>> Hi Folks,
> >> >> >>>>>
> >> >> >>>>> ARM is becoming famous lately in its processing capability and
> >> has
> >> >> >> got
> >> >> >>>> the
> >> >> >>>>> potential to run Bigdata workloads.
> >> >> >>>>> Many users have been moving to ARM machines due to its low
> cost.
> >> >> >>>>>
> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
> >> (Rasberry
> >> >> >> PI)
> >> >> >>>> for
> >> >> >>>>> experimental purposes. Today ARM architecture is taking some of
> >> the
> >> >> >>>>> serverside processing as well. So there will be/is a real need
> of
> >> >> >>> Hadoop
> >> >> >>>> to
> >> >> >>>>> support ARM architecture as well.
> >> >> >>>>>
> >> >> >>>>> There are bunch of users who are trying out building Hadoop on
> >> ARM,
> >> >> >>>> trying
> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
> >> >> >>>>>
> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
> issues,
> >> >> >> found
> >> >> >>>> from
> >> >> >>>>> testing done in openlab in [2].
> >> >> >>>>>
> >> >> >>>>> 1. Protobuf :
> >> >> >>>>> -------------------
> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
> >> protobuf
> >> >> >>>> 2.5.0
> >> >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0
> is
> >> >> not
> >> >> >>>> being
> >> >> >>>>> maintained in the community. While protobuf 3.x is being
> actively
> >> >> >>> adopted
> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
> proto2
> >> >> >>>> messages.
> >> >> >>>>> Due to some compilation issues in the generated java code,
> which
> >> can
> >> >> >>>> induce
> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
> from
> >> >> >> 2.5.0
> >> >> >>>> was
> >> >> >>>>> not taken up.
> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> >> >> >>> classpath
> >> >> >>>>> problem in downstream projects.
> >> >> >>>>>    There are patches available to fix compilation in Hadoop.
> But
> >> >> >> need
> >> >> >>> to
> >> >> >>>>> find a way to upgrade protobuf to latest version and still
> >> maintain
> >> >> >> the
> >> >> >>>>> downstream's classpath using shading feature of Hadoop build.
> >> >> >>>>>
> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even before
> >> >> >> shade
> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
> >> >> >> continue
> >> >> >>>>> explore possibilities.
> >> >> >>>>>
> >> >> >>>>> 2. leveldbjni:
> >> >> >>>>> ---------------
> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
> >> architecture,
> >> >> >>>> need
> >> >> >>>>> to check whether any of the future versions support ARM and can
> >> >> >> hadoop
> >> >> >>>>> upgrade to that version.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> >> >> >>>>> -------------------------
> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable
> by
> >> >> >>> default
> >> >> >>>> in
> >> >> >>>>> the maven repository. Workaround is to build it locally and
> keep
> >> in
> >> >> >>> local
> >> >> >>>>> maven repository.
> >> >> >>>>> Need to check whether any future versions of
> >> 'protoc-gen-grpc-java'
> >> >> >> is
> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade
> it?
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> Once the compilation issues are solved, then there might be
> many
> >> >> >> native
> >> >> >>>>> code related issues due to different architectures.
> >> >> >>>>> So to explore everything, need to join hands together and
> >> proceed.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> Let us discuss and check, whether any body else out there who
> >> also
> >> >> >> need
> >> >> >>>> the
> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
> >> hands
> >> >> >>> and
> >> >> >>>>> time in this work.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
> >> >> >>>>> [2]
> >> >> >>
> >> >>
> >>
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
> >> >> >>>>>
> >> >> >>>>> -Vinay
> >> >> >>
> >> >>
> >> >
> >>
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Anu Engineer <ae...@cloudera.com.INVALID>]

Yes, I think that is what Sunil and I are trying to suggest; the complex
dependencies like Protobuf, if you do it in the trunk you have a better
change of getting it done. Otherwise, at merge point random downstream
applications which you have never heard of will object, and Hadoop
compatibility rules are very clear so you cannot fix it.

With that said, even doing this in the trunk is complex; It might be good
for you to host a meeting and get some feedback. I have openly said it is a
great idea like "belling the cat", but the effort is in getting the
community to agree and align. Solve that, most of your technical problems
will be easier to solve.

If you go into a branch, it might be that the community might forget about
your work; and when you come in to merge you will see issues which you did
not think about.

So, Here is what would be great if you can make this happen; for ARM work,
get a list of dependencies that needed to be upgraded; see if you can get
the community aligned with this goal; since ARM might not be in the target
for many users. You need to convince users that even without ARM, this is a
great idea.

If you like we can get together during one of the HDFS meetups hosted by
Wei-chiu.

Thanks
Anu



On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vi...@apache.org>
wrote:

> Hi all,
>
> Thanks for the response.
> As I see, protobuf upgrade is long pending and most awaited one.
>
> @Sunil
> Protobuf upgrade looks to be a non-trivial task.
> Thanks @Duo Zhang for the suggestion of
> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the problem
> of dependency on build environment.
> However more problem lies in upgrade protobuf without breaking the
> downstream builds.
> Reason is many downstream projects directly refers server specific jars and
> they expect protobuf-2.5.0 jar to get added to classpath by transitive
> dependency.
>
> There are some historical discussions and suggestions on
> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
> upgrade.
> Recommended path for solution is, try to upgrade protobuf using shading of
> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
> dependencies for downstreams.
> I am trying out ideas on this and if it gets completed within time, may be
> we can target trunk itself for the protobuf upgrade.
>
> separate branch idea is suggested for the overall ARM support including
> protobuf upgrade and other problems mentioned in the discussion above.
>
> I dont expect separate branch to have a huge changes, apart from bug fixes,
> since there are no separate features specific to ARM is being planned.
> So timely rebase of separate branch would reduce the overhead on branch
> review/merge task.
>
> Still, if the solution to protobuf upgrade winds up early, without any side
> effects, I am more than happy to land it in trunk itself.
>
> Thanks,
> Vinay
> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:
>
> > Thanks Vinay for starting the thread.
> >
> > I agree to Anu's view point related to protobuf. And with the suggestion
> > pointed out by Duo Zhang, if we can make use
> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
> > of protobuf will also be more easier.
> >
> > However i think its better to do this effort in trunk itself.
> > In offline talks, few members were interested to start 3.3.0 release. And
> > given that happens soon, I feel its better
> > we do this task in trunk itself as branch diverge is very much possible.
> > And to bring to call a merge on such a big branch will be even more tough
> > task.
> >
> > my 2 cents.
> >
> > Thanks
> > Sunil
> >
> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> >
> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
> >> generate
> >> the protobuf code. It will download the protoc binary from the maven
> >> central so we do not need to install protoc on the build machine any
> more.
> >>
> >> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
> >>
> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
> >> for
> >> > over 2 month related to the Protobuf problem .
> >> > According to the latest successful build log:
> >> >
> >>
> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
> >> > the
> >> > os version was ubuntu 14.04 and for the jobs that are failling now
> such
> >> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
> >> > the os version is 18.04. I'm not very familiar with the version
> changing
> >> > for the jobs but I did a little search, according to:
> >> >
> >> >
> >>
> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
> >> > &
> >> >
> >> >
> >>
> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
> >> > it both said that the version of libprotc-dev and protobuf-compiler
> >> > available for ubuntu 18.04 is 3.0.0
> >> >
> >> >
> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com>
> wrote:
> >> >
> >> >> Thanx Vinay for the initiative, Makes sense to add support for
> >> different
> >> >> architectures.
> >> >>
> >> >> +1, for the branch idea.
> >> >> Good Luck!!!
> >> >>
> >> >> -Ayush
> >> >>
> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> > For HBase, we purged all the protobuf related things from the
> public
> >> >> API,
> >> >> > and then upgraded to a shaded and relocated version of protobuf. We
> >> have
> >> >> > created a repo for this:
> >> >> >
> >> >> > https://github.com/apache/hbase-thirdparty
> >> >> >
> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5
> >> jars,
> >> >> our
> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
> >> >> discuss
> >> >> > on how to deal with the upgrading of coprocessor. Glad to see that
> >> the
> >> >> > hadoop community is also willing to solve the problem.
> >> >> >
> >> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二
> 上午1:23写道：
> >> >> >
> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
> >> that
> >> >> >> Hadoop and the downstream projects work correctly after you
> upgrade
> >> >> core
> >> >> >> components like Protobuf.
> >> >> >> So while branching and working on a branch is easy, merging back
> >> after
> >> >> you
> >> >> >> upgrade some of these core components is insanely hard. You might
> >> want
> >> >> to
> >> >> >> make sure that community buys into upgrading these components in
> the
> >> >> trunk.
> >> >> >> That way we will get testing and downstream components will notice
> >> when
> >> >> >> things break.
> >> >> >>
> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
> >> long
> >> >> >> time; I have argued that 2.5 is out of support and we cannot stay
> on
> >> >> that
> >> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
> >> code
> >> >> base.
> >> >> >> It has been rightly pointed to me that while all the arguments I
> >> make
> >> >> is
> >> >> >> correct; it is a very complicated task to upgrade Protobuf, and
> the
> >> >> worst
> >> >> >> part is we will not even know what breaks until downstream
> projects
> >> >> pick up
> >> >> >> these changes and work against us.
> >> >> >>
> >> >> >> If we work off the Hadoop version 3 — and assume that we have
> >> >> "shading" in
> >> >> >> place for all deployments; it might be possible to get there;
> still
> >> a
> >> >> >> daunting task.
> >> >> >>
> >> >> >> So best of luck with the branch approach — But please remember,
> >> Merging
> >> >> >> back will be hard, Just my 2 cents.
> >> >> >>
> >> >> >> — Anu
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
> >> zhengzhenyulixi@gmail.com
> >> >> >
> >> >> >> wrote:
> >> >> >>
> >> >> >>> Hi,
> >> >> >>>
> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
> >> >> separate
> >> >> >>> branch with it's own ARM CI seems a really good idea.
> >> >> >>> By doing this we won't break any of the undergoing development in
> >> >> trunk
> >> >> >> and
> >> >> >>> a CI can be a very good way to show what are the
> >> >> >>> current problems and what have been fixed, it will also provide a
> >> very
> >> >> >> good
> >> >> >>> view for contributors that are intrested to working on
> >> >> >>> this. We can finally merge back the branch to trunk until the
> >> >> community
> >> >> >>> thinks it is good enough and stable enough. We can donate
> >> >> >>> ARM machines to the existing CI system for the job.
> >> >> >>>
> >> >> >>> I wonder if this approch possible?
> >> >> >>>
> >> >> >>> BR,
> >> >> >>>
> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
> >> liusheng2048@gmail.com>
> >> >> >>> wrote:
> >> >> >>>
> >> >> >>>> Hi,
> >> >> >>>>
> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
> >> community
> >> >> >>>> mentioned by Vinay. I am working on building and
> >> >> >>>> testing Hadoop components on aarch64 server these days, besides
> >> the
> >> >> >>> missing
> >> >> >>>> dependices of ARM platform issues #1 #2 #3
> >> >> >>>> mentioned by Vinay, other similar issue has also be found, such
> as
> >> >> the
> >> >> >>>> "PhantomJS" dependent package also missing for aarch64.
> >> >> >>>>
> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and
> >> hoped to
> >> >> >> add
> >> >> >>>> an ARM specific CI to Hadoop repo. we are not
> >> >> >>>> sure about if there is any potential effect or confilict on the
> >> trunk
> >> >> >>>> branch, so maybe creating a ARM specific branch for doing these
> >> stuff
> >> >> >>>> is a better choice, what do you think?
> >> >> >>>>
> >> >> >>>> Hope to hear thoughts from you :)
> >> >> >>>>
> >> >> >>>> BR,
> >> >> >>>> Liu sheng
> >> >> >>>>
> >> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> >> >> >>>>
> >> >> >>>>> Hi Folks,
> >> >> >>>>>
> >> >> >>>>> ARM is becoming famous lately in its processing capability and
> >> has
> >> >> >> got
> >> >> >>>> the
> >> >> >>>>> potential to run Bigdata workloads.
> >> >> >>>>> Many users have been moving to ARM machines due to its low
> cost.
> >> >> >>>>>
> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM
> >> (Rasberry
> >> >> >> PI)
> >> >> >>>> for
> >> >> >>>>> experimental purposes. Today ARM architecture is taking some of
> >> the
> >> >> >>>>> serverside processing as well. So there will be/is a real need
> of
> >> >> >>> Hadoop
> >> >> >>>> to
> >> >> >>>>> support ARM architecture as well.
> >> >> >>>>>
> >> >> >>>>> There are bunch of users who are trying out building Hadoop on
> >> ARM,
> >> >> >>>> trying
> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
> >> >> >>>>>
> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below
> issues,
> >> >> >> found
> >> >> >>>> from
> >> >> >>>>> testing done in openlab in [2].
> >> >> >>>>>
> >> >> >>>>> 1. Protobuf :
> >> >> >>>>> -------------------
> >> >> >>>>>     Hadoop project (also some downstream projects) stuck to
> >> protobuf
> >> >> >>>> 2.5.0
> >> >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0
> is
> >> >> not
> >> >> >>>> being
> >> >> >>>>> maintained in the community. While protobuf 3.x is being
> actively
> >> >> >>> adopted
> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for
> proto2
> >> >> >>>> messages.
> >> >> >>>>> Due to some compilation issues in the generated java code,
> which
> >> can
> >> >> >>>> induce
> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
> from
> >> >> >> 2.5.0
> >> >> >>>> was
> >> >> >>>>> not taken up.
> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> >> >> >>> classpath
> >> >> >>>>> problem in downstream projects.
> >> >> >>>>>    There are patches available to fix compilation in Hadoop.
> But
> >> >> >> need
> >> >> >>> to
> >> >> >>>>> find a way to upgrade protobuf to latest version and still
> >> maintain
> >> >> >> the
> >> >> >>>>> downstream's classpath using shading feature of Hadoop build.
> >> >> >>>>>
> >> >> >>>>>     There is a Jira for protobuf upgrade[3] created even before
> >> >> >> shade
> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
> >> >> >> continue
> >> >> >>>>> explore possibilities.
> >> >> >>>>>
> >> >> >>>>> 2. leveldbjni:
> >> >> >>>>> ---------------
> >> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
> >> architecture,
> >> >> >>>> need
> >> >> >>>>> to check whether any of the future versions support ARM and can
> >> >> >> hadoop
> >> >> >>>>> upgrade to that version.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> >> >> >>>>> -------------------------
> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable
> by
> >> >> >>> default
> >> >> >>>> in
> >> >> >>>>> the maven repository. Workaround is to build it locally and
> keep
> >> in
> >> >> >>> local
> >> >> >>>>> maven repository.
> >> >> >>>>> Need to check whether any future versions of
> >> 'protoc-gen-grpc-java'
> >> >> >> is
> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade
> it?
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> Once the compilation issues are solved, then there might be
> many
> >> >> >> native
> >> >> >>>>> code related issues due to different architectures.
> >> >> >>>>> So to explore everything, need to join hands together and
> >> proceed.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> Let us discuss and check, whether any body else out there who
> >> also
> >> >> >> need
> >> >> >>>> the
> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
> >> hands
> >> >> >>> and
> >> >> >>>>> time in this work.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
> >> >> >>>>> [2]
> >> >> >>
> >> >>
> >>
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
> >> >> >>>>>
> >> >> >>>>> -Vinay
> >> >> >>
> >> >>
> >> >
> >>
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B <vi...@apache.org>]

Hi all,

Thanks for the response.
As I see, protobuf upgrade is long pending and most awaited one.

@Sunil
Protobuf upgrade looks to be a non-trivial task.
Thanks @Duo Zhang for the suggestion of
'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the problem
of dependency on build environment.
However more problem lies in upgrade protobuf without breaking the
downstream builds.
Reason is many downstream projects directly refers server specific jars and
they expect protobuf-2.5.0 jar to get added to classpath by transitive
dependency.

There are some historical discussions and suggestions on
https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
upgrade.
Recommended path for solution is, try to upgrade protobuf using shading of
latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
dependencies for downstreams.
I am trying out ideas on this and if it gets completed within time, may be
we can target trunk itself for the protobuf upgrade.

separate branch idea is suggested for the overall ARM support including
protobuf upgrade and other problems mentioned in the discussion above.

I dont expect separate branch to have a huge changes, apart from bug fixes,
since there are no separate features specific to ARM is being planned.
So timely rebase of separate branch would reduce the overhead on branch
review/merge task.

Still, if the solution to protobuf upgrade winds up early, without any side
effects, I am more than happy to land it in trunk itself.

Thanks,
Vinay
On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:

> Thanks Vinay for starting the thread.
>
> I agree to Anu's view point related to protobuf. And with the suggestion
> pointed out by Duo Zhang, if we can make use
> of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
> of protobuf will also be more easier.
>
> However i think its better to do this effort in trunk itself.
> In offline talks, few members were interested to start 3.3.0 release. And
> given that happens soon, I feel its better
> we do this task in trunk itself as branch diverge is very much possible.
> And to bring to call a merge on such a big branch will be even more tough
> task.
>
> my 2 cents.
>
> Thanks
> Sunil
>
> On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
>> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>> generate
>> the protobuf code. It will download the protoc binary from the maven
>> central so we do not need to install protoc on the build machine any more.
>>
>> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>>
>> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
>> for
>> > over 2 month related to the Protobuf problem .
>> > According to the latest successful build log:
>> >
>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>> > the
>> > os version was ubuntu 14.04 and for the jobs that are failling now such
>> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
>> > the os version is 18.04. I'm not very familiar with the version changing
>> > for the jobs but I did a little search, according to:
>> >
>> >
>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>> > &
>> >
>> >
>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>> > it both said that the version of libprotc-dev and protobuf-compiler
>> > available for ubuntu 18.04 is 3.0.0
>> >
>> >
>> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com> wrote:
>> >
>> >> Thanx Vinay for the initiative, Makes sense to add support for
>> different
>> >> architectures.
>> >>
>> >> +1, for the branch idea.
>> >> Good Luck!!!
>> >>
>> >> -Ayush
>> >>
>> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
>> >> wrote:
>> >> >
>> >> > For HBase, we purged all the protobuf related things from the public
>> >> API,
>> >> > and then upgraded to a shaded and relocated version of protobuf. We
>> have
>> >> > created a repo for this:
>> >> >
>> >> > https://github.com/apache/hbase-thirdparty
>> >> >
>> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>> jars,
>> >> our
>> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
>> >> discuss
>> >> > on how to deal with the upgrading of coprocessor. Glad to see that
>> the
>> >> > hadoop community is also willing to solve the problem.
>> >> >
>> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
>> >> >
>> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
>> that
>> >> >> Hadoop and the downstream projects work correctly after you upgrade
>> >> core
>> >> >> components like Protobuf.
>> >> >> So while branching and working on a branch is easy, merging back
>> after
>> >> you
>> >> >> upgrade some of these core components is insanely hard. You might
>> want
>> >> to
>> >> >> make sure that community buys into upgrading these components in the
>> >> trunk.
>> >> >> That way we will get testing and downstream components will notice
>> when
>> >> >> things break.
>> >> >>
>> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
>> long
>> >> >> time; I have argued that 2.5 is out of support and we cannot stay on
>> >> that
>> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>> code
>> >> base.
>> >> >> It has been rightly pointed to me that while all the arguments I
>> make
>> >> is
>> >> >> correct; it is a very complicated task to upgrade Protobuf, and the
>> >> worst
>> >> >> part is we will not even know what breaks until downstream projects
>> >> pick up
>> >> >> these changes and work against us.
>> >> >>
>> >> >> If we work off the Hadoop version 3 — and assume that we have
>> >> "shading" in
>> >> >> place for all deployments; it might be possible to get there; still
>> a
>> >> >> daunting task.
>> >> >>
>> >> >> So best of luck with the branch approach — But please remember,
>> Merging
>> >> >> back will be hard, Just my 2 cents.
>> >> >>
>> >> >> — Anu
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>> zhengzhenyulixi@gmail.com
>> >> >
>> >> >> wrote:
>> >> >>
>> >> >>> Hi,
>> >> >>>
>> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
>> >> separate
>> >> >>> branch with it's own ARM CI seems a really good idea.
>> >> >>> By doing this we won't break any of the undergoing development in
>> >> trunk
>> >> >> and
>> >> >>> a CI can be a very good way to show what are the
>> >> >>> current problems and what have been fixed, it will also provide a
>> very
>> >> >> good
>> >> >>> view for contributors that are intrested to working on
>> >> >>> this. We can finally merge back the branch to trunk until the
>> >> community
>> >> >>> thinks it is good enough and stable enough. We can donate
>> >> >>> ARM machines to the existing CI system for the job.
>> >> >>>
>> >> >>> I wonder if this approch possible?
>> >> >>>
>> >> >>> BR,
>> >> >>>
>> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>> liusheng2048@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>>> Hi,
>> >> >>>>
>> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>> community
>> >> >>>> mentioned by Vinay. I am working on building and
>> >> >>>> testing Hadoop components on aarch64 server these days, besides
>> the
>> >> >>> missing
>> >> >>>> dependices of ARM platform issues #1 #2 #3
>> >> >>>> mentioned by Vinay, other similar issue has also be found, such as
>> >> the
>> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >> >>>>
>> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>> hoped to
>> >> >> add
>> >> >>>> an ARM specific CI to Hadoop repo. we are not
>> >> >>>> sure about if there is any potential effect or confilict on the
>> trunk
>> >> >>>> branch, so maybe creating a ARM specific branch for doing these
>> stuff
>> >> >>>> is a better choice, what do you think?
>> >> >>>>
>> >> >>>> Hope to hear thoughts from you :)
>> >> >>>>
>> >> >>>> BR,
>> >> >>>> Liu sheng
>> >> >>>>
>> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
>> >> >>>>
>> >> >>>>> Hi Folks,
>> >> >>>>>
>> >> >>>>> ARM is becoming famous lately in its processing capability and
>> has
>> >> >> got
>> >> >>>> the
>> >> >>>>> potential to run Bigdata workloads.
>> >> >>>>> Many users have been moving to ARM machines due to its low cost.
>> >> >>>>>
>> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>> (Rasberry
>> >> >> PI)
>> >> >>>> for
>> >> >>>>> experimental purposes. Today ARM architecture is taking some of
>> the
>> >> >>>>> serverside processing as well. So there will be/is a real need of
>> >> >>> Hadoop
>> >> >>>> to
>> >> >>>>> support ARM architecture as well.
>> >> >>>>>
>> >> >>>>> There are bunch of users who are trying out building Hadoop on
>> ARM,
>> >> >>>> trying
>> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>> >> >>>>>
>> >> >>>>> As of today, Hadoop does not compile on ARM due to below issues,
>> >> >> found
>> >> >>>> from
>> >> >>>>> testing done in openlab in [2].
>> >> >>>>>
>> >> >>>>> 1. Protobuf :
>> >> >>>>> -------------------
>> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>> protobuf
>> >> >>>> 2.5.0
>> >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is
>> >> not
>> >> >>>> being
>> >> >>>>> maintained in the community. While protobuf 3.x is being actively
>> >> >>> adopted
>> >> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2
>> >> >>>> messages.
>> >> >>>>> Due to some compilation issues in the generated java code, which
>> can
>> >> >>>> induce
>> >> >>>>> problems in downstream. Due to this reason protobuf upgrade from
>> >> >> 2.5.0
>> >> >>>> was
>> >> >>>>> not taken up.
>> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
>> >> >>> classpath
>> >> >>>>> problem in downstream projects.
>> >> >>>>>    There are patches available to fix compilation in Hadoop. But
>> >> >> need
>> >> >>> to
>> >> >>>>> find a way to upgrade protobuf to latest version and still
>> maintain
>> >> >> the
>> >> >>>>> downstream's classpath using shading feature of Hadoop build.
>> >> >>>>>
>> >> >>>>>     There is a Jira for protobuf upgrade[3] created even before
>> >> >> shade
>> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>> >> >> continue
>> >> >>>>> explore possibilities.
>> >> >>>>>
>> >> >>>>> 2. leveldbjni:
>> >> >>>>> ---------------
>> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>> architecture,
>> >> >>>> need
>> >> >>>>> to check whether any of the future versions support ARM and can
>> >> >> hadoop
>> >> >>>>> upgrade to that version.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>> >> >>>>> -------------------------
>> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
>> >> >>> default
>> >> >>>> in
>> >> >>>>> the maven repository. Workaround is to build it locally and keep
>> in
>> >> >>> local
>> >> >>>>> maven repository.
>> >> >>>>> Need to check whether any future versions of
>> 'protoc-gen-grpc-java'
>> >> >> is
>> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Once the compilation issues are solved, then there might be many
>> >> >> native
>> >> >>>>> code related issues due to different architectures.
>> >> >>>>> So to explore everything, need to join hands together and
>> proceed.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Let us discuss and check, whether any body else out there who
>> also
>> >> >> need
>> >> >>>> the
>> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
>> hands
>> >> >>> and
>> >> >>>>> time in this work.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>> >> >>>>> [2]
>> >> >>
>> >>
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>> >> >>>>>
>> >> >>>>> -Vinay
>> >> >>
>> >>
>> >
>>
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B <vi...@apache.org>]

Hi all,

Thanks for the response.
As I see, protobuf upgrade is long pending and most awaited one.

@Sunil
Protobuf upgrade looks to be a non-trivial task.
Thanks @Duo Zhang for the suggestion of
'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the problem
of dependency on build environment.
However more problem lies in upgrade protobuf without breaking the
downstream builds.
Reason is many downstream projects directly refers server specific jars and
they expect protobuf-2.5.0 jar to get added to classpath by transitive
dependency.

There are some historical discussions and suggestions on
https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
upgrade.
Recommended path for solution is, try to upgrade protobuf using shading of
latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
dependencies for downstreams.
I am trying out ideas on this and if it gets completed within time, may be
we can target trunk itself for the protobuf upgrade.

separate branch idea is suggested for the overall ARM support including
protobuf upgrade and other problems mentioned in the discussion above.

I dont expect separate branch to have a huge changes, apart from bug fixes,
since there are no separate features specific to ARM is being planned.
So timely rebase of separate branch would reduce the overhead on branch
review/merge task.

Still, if the solution to protobuf upgrade winds up early, without any side
effects, I am more than happy to land it in trunk itself.

Thanks,
Vinay
On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:

> Thanks Vinay for starting the thread.
>
> I agree to Anu's view point related to protobuf. And with the suggestion
> pointed out by Duo Zhang, if we can make use
> of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
> of protobuf will also be more easier.
>
> However i think its better to do this effort in trunk itself.
> In offline talks, few members were interested to start 3.3.0 release. And
> given that happens soon, I feel its better
> we do this task in trunk itself as branch diverge is very much possible.
> And to bring to call a merge on such a big branch will be even more tough
> task.
>
> my 2 cents.
>
> Thanks
> Sunil
>
> On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
>> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>> generate
>> the protobuf code. It will download the protoc binary from the maven
>> central so we do not need to install protoc on the build machine any more.
>>
>> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>>
>> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
>> for
>> > over 2 month related to the Protobuf problem .
>> > According to the latest successful build log:
>> >
>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>> > the
>> > os version was ubuntu 14.04 and for the jobs that are failling now such
>> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
>> > the os version is 18.04. I'm not very familiar with the version changing
>> > for the jobs but I did a little search, according to:
>> >
>> >
>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>> > &
>> >
>> >
>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>> > it both said that the version of libprotc-dev and protobuf-compiler
>> > available for ubuntu 18.04 is 3.0.0
>> >
>> >
>> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com> wrote:
>> >
>> >> Thanx Vinay for the initiative, Makes sense to add support for
>> different
>> >> architectures.
>> >>
>> >> +1, for the branch idea.
>> >> Good Luck!!!
>> >>
>> >> -Ayush
>> >>
>> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
>> >> wrote:
>> >> >
>> >> > For HBase, we purged all the protobuf related things from the public
>> >> API,
>> >> > and then upgraded to a shaded and relocated version of protobuf. We
>> have
>> >> > created a repo for this:
>> >> >
>> >> > https://github.com/apache/hbase-thirdparty
>> >> >
>> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>> jars,
>> >> our
>> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
>> >> discuss
>> >> > on how to deal with the upgrading of coprocessor. Glad to see that
>> the
>> >> > hadoop community is also willing to solve the problem.
>> >> >
>> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
>> >> >
>> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
>> that
>> >> >> Hadoop and the downstream projects work correctly after you upgrade
>> >> core
>> >> >> components like Protobuf.
>> >> >> So while branching and working on a branch is easy, merging back
>> after
>> >> you
>> >> >> upgrade some of these core components is insanely hard. You might
>> want
>> >> to
>> >> >> make sure that community buys into upgrading these components in the
>> >> trunk.
>> >> >> That way we will get testing and downstream components will notice
>> when
>> >> >> things break.
>> >> >>
>> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
>> long
>> >> >> time; I have argued that 2.5 is out of support and we cannot stay on
>> >> that
>> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>> code
>> >> base.
>> >> >> It has been rightly pointed to me that while all the arguments I
>> make
>> >> is
>> >> >> correct; it is a very complicated task to upgrade Protobuf, and the
>> >> worst
>> >> >> part is we will not even know what breaks until downstream projects
>> >> pick up
>> >> >> these changes and work against us.
>> >> >>
>> >> >> If we work off the Hadoop version 3 — and assume that we have
>> >> "shading" in
>> >> >> place for all deployments; it might be possible to get there; still
>> a
>> >> >> daunting task.
>> >> >>
>> >> >> So best of luck with the branch approach — But please remember,
>> Merging
>> >> >> back will be hard, Just my 2 cents.
>> >> >>
>> >> >> — Anu
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>> zhengzhenyulixi@gmail.com
>> >> >
>> >> >> wrote:
>> >> >>
>> >> >>> Hi,
>> >> >>>
>> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
>> >> separate
>> >> >>> branch with it's own ARM CI seems a really good idea.
>> >> >>> By doing this we won't break any of the undergoing development in
>> >> trunk
>> >> >> and
>> >> >>> a CI can be a very good way to show what are the
>> >> >>> current problems and what have been fixed, it will also provide a
>> very
>> >> >> good
>> >> >>> view for contributors that are intrested to working on
>> >> >>> this. We can finally merge back the branch to trunk until the
>> >> community
>> >> >>> thinks it is good enough and stable enough. We can donate
>> >> >>> ARM machines to the existing CI system for the job.
>> >> >>>
>> >> >>> I wonder if this approch possible?
>> >> >>>
>> >> >>> BR,
>> >> >>>
>> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>> liusheng2048@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>>> Hi,
>> >> >>>>
>> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>> community
>> >> >>>> mentioned by Vinay. I am working on building and
>> >> >>>> testing Hadoop components on aarch64 server these days, besides
>> the
>> >> >>> missing
>> >> >>>> dependices of ARM platform issues #1 #2 #3
>> >> >>>> mentioned by Vinay, other similar issue has also be found, such as
>> >> the
>> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >> >>>>
>> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>> hoped to
>> >> >> add
>> >> >>>> an ARM specific CI to Hadoop repo. we are not
>> >> >>>> sure about if there is any potential effect or confilict on the
>> trunk
>> >> >>>> branch, so maybe creating a ARM specific branch for doing these
>> stuff
>> >> >>>> is a better choice, what do you think?
>> >> >>>>
>> >> >>>> Hope to hear thoughts from you :)
>> >> >>>>
>> >> >>>> BR,
>> >> >>>> Liu sheng
>> >> >>>>
>> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
>> >> >>>>
>> >> >>>>> Hi Folks,
>> >> >>>>>
>> >> >>>>> ARM is becoming famous lately in its processing capability and
>> has
>> >> >> got
>> >> >>>> the
>> >> >>>>> potential to run Bigdata workloads.
>> >> >>>>> Many users have been moving to ARM machines due to its low cost.
>> >> >>>>>
>> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>> (Rasberry
>> >> >> PI)
>> >> >>>> for
>> >> >>>>> experimental purposes. Today ARM architecture is taking some of
>> the
>> >> >>>>> serverside processing as well. So there will be/is a real need of
>> >> >>> Hadoop
>> >> >>>> to
>> >> >>>>> support ARM architecture as well.
>> >> >>>>>
>> >> >>>>> There are bunch of users who are trying out building Hadoop on
>> ARM,
>> >> >>>> trying
>> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>> >> >>>>>
>> >> >>>>> As of today, Hadoop does not compile on ARM due to below issues,
>> >> >> found
>> >> >>>> from
>> >> >>>>> testing done in openlab in [2].
>> >> >>>>>
>> >> >>>>> 1. Protobuf :
>> >> >>>>> -------------------
>> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>> protobuf
>> >> >>>> 2.5.0
>> >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is
>> >> not
>> >> >>>> being
>> >> >>>>> maintained in the community. While protobuf 3.x is being actively
>> >> >>> adopted
>> >> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2
>> >> >>>> messages.
>> >> >>>>> Due to some compilation issues in the generated java code, which
>> can
>> >> >>>> induce
>> >> >>>>> problems in downstream. Due to this reason protobuf upgrade from
>> >> >> 2.5.0
>> >> >>>> was
>> >> >>>>> not taken up.
>> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
>> >> >>> classpath
>> >> >>>>> problem in downstream projects.
>> >> >>>>>    There are patches available to fix compilation in Hadoop. But
>> >> >> need
>> >> >>> to
>> >> >>>>> find a way to upgrade protobuf to latest version and still
>> maintain
>> >> >> the
>> >> >>>>> downstream's classpath using shading feature of Hadoop build.
>> >> >>>>>
>> >> >>>>>     There is a Jira for protobuf upgrade[3] created even before
>> >> >> shade
>> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>> >> >> continue
>> >> >>>>> explore possibilities.
>> >> >>>>>
>> >> >>>>> 2. leveldbjni:
>> >> >>>>> ---------------
>> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>> architecture,
>> >> >>>> need
>> >> >>>>> to check whether any of the future versions support ARM and can
>> >> >> hadoop
>> >> >>>>> upgrade to that version.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>> >> >>>>> -------------------------
>> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
>> >> >>> default
>> >> >>>> in
>> >> >>>>> the maven repository. Workaround is to build it locally and keep
>> in
>> >> >>> local
>> >> >>>>> maven repository.
>> >> >>>>> Need to check whether any future versions of
>> 'protoc-gen-grpc-java'
>> >> >> is
>> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Once the compilation issues are solved, then there might be many
>> >> >> native
>> >> >>>>> code related issues due to different architectures.
>> >> >>>>> So to explore everything, need to join hands together and
>> proceed.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Let us discuss and check, whether any body else out there who
>> also
>> >> >> need
>> >> >>>> the
>> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
>> hands
>> >> >>> and
>> >> >>>>> time in this work.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>> >> >>>>> [2]
>> >> >>
>> >>
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>> >> >>>>>
>> >> >>>>> -Vinay
>> >> >>
>> >>
>> >
>>
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B <vi...@apache.org>]

Hi all,

Thanks for the response.
As I see, protobuf upgrade is long pending and most awaited one.

@Sunil
Protobuf upgrade looks to be a non-trivial task.
Thanks @Duo Zhang for the suggestion of
'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the problem
of dependency on build environment.
However more problem lies in upgrade protobuf without breaking the
downstream builds.
Reason is many downstream projects directly refers server specific jars and
they expect protobuf-2.5.0 jar to get added to classpath by transitive
dependency.

There are some historical discussions and suggestions on
https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
upgrade.
Recommended path for solution is, try to upgrade protobuf using shading of
latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
dependencies for downstreams.
I am trying out ideas on this and if it gets completed within time, may be
we can target trunk itself for the protobuf upgrade.

separate branch idea is suggested for the overall ARM support including
protobuf upgrade and other problems mentioned in the discussion above.

I dont expect separate branch to have a huge changes, apart from bug fixes,
since there are no separate features specific to ARM is being planned.
So timely rebase of separate branch would reduce the overhead on branch
review/merge task.

Still, if the solution to protobuf upgrade winds up early, without any side
effects, I am more than happy to land it in trunk itself.

Thanks,
Vinay
On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:

> Thanks Vinay for starting the thread.
>
> I agree to Anu's view point related to protobuf. And with the suggestion
> pointed out by Duo Zhang, if we can make use
> of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
> of protobuf will also be more easier.
>
> However i think its better to do this effort in trunk itself.
> In offline talks, few members were interested to start 3.3.0 release. And
> given that happens soon, I feel its better
> we do this task in trunk itself as branch diverge is very much possible.
> And to bring to call a merge on such a big branch will be even more tough
> task.
>
> my 2 cents.
>
> Thanks
> Sunil
>
> On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
>> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>> generate
>> the protobuf code. It will download the protoc binary from the maven
>> central so we do not need to install protoc on the build machine any more.
>>
>> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>>
>> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
>> for
>> > over 2 month related to the Protobuf problem .
>> > According to the latest successful build log:
>> >
>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>> > the
>> > os version was ubuntu 14.04 and for the jobs that are failling now such
>> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
>> > the os version is 18.04. I'm not very familiar with the version changing
>> > for the jobs but I did a little search, according to:
>> >
>> >
>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>> > &
>> >
>> >
>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>> > it both said that the version of libprotc-dev and protobuf-compiler
>> > available for ubuntu 18.04 is 3.0.0
>> >
>> >
>> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com> wrote:
>> >
>> >> Thanx Vinay for the initiative, Makes sense to add support for
>> different
>> >> architectures.
>> >>
>> >> +1, for the branch idea.
>> >> Good Luck!!!
>> >>
>> >> -Ayush
>> >>
>> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
>> >> wrote:
>> >> >
>> >> > For HBase, we purged all the protobuf related things from the public
>> >> API,
>> >> > and then upgraded to a shaded and relocated version of protobuf. We
>> have
>> >> > created a repo for this:
>> >> >
>> >> > https://github.com/apache/hbase-thirdparty
>> >> >
>> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>> jars,
>> >> our
>> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
>> >> discuss
>> >> > on how to deal with the upgrading of coprocessor. Glad to see that
>> the
>> >> > hadoop community is also willing to solve the problem.
>> >> >
>> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
>> >> >
>> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
>> that
>> >> >> Hadoop and the downstream projects work correctly after you upgrade
>> >> core
>> >> >> components like Protobuf.
>> >> >> So while branching and working on a branch is easy, merging back
>> after
>> >> you
>> >> >> upgrade some of these core components is insanely hard. You might
>> want
>> >> to
>> >> >> make sure that community buys into upgrading these components in the
>> >> trunk.
>> >> >> That way we will get testing and downstream components will notice
>> when
>> >> >> things break.
>> >> >>
>> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
>> long
>> >> >> time; I have argued that 2.5 is out of support and we cannot stay on
>> >> that
>> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>> code
>> >> base.
>> >> >> It has been rightly pointed to me that while all the arguments I
>> make
>> >> is
>> >> >> correct; it is a very complicated task to upgrade Protobuf, and the
>> >> worst
>> >> >> part is we will not even know what breaks until downstream projects
>> >> pick up
>> >> >> these changes and work against us.
>> >> >>
>> >> >> If we work off the Hadoop version 3 — and assume that we have
>> >> "shading" in
>> >> >> place for all deployments; it might be possible to get there; still
>> a
>> >> >> daunting task.
>> >> >>
>> >> >> So best of luck with the branch approach — But please remember,
>> Merging
>> >> >> back will be hard, Just my 2 cents.
>> >> >>
>> >> >> — Anu
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>> zhengzhenyulixi@gmail.com
>> >> >
>> >> >> wrote:
>> >> >>
>> >> >>> Hi,
>> >> >>>
>> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
>> >> separate
>> >> >>> branch with it's own ARM CI seems a really good idea.
>> >> >>> By doing this we won't break any of the undergoing development in
>> >> trunk
>> >> >> and
>> >> >>> a CI can be a very good way to show what are the
>> >> >>> current problems and what have been fixed, it will also provide a
>> very
>> >> >> good
>> >> >>> view for contributors that are intrested to working on
>> >> >>> this. We can finally merge back the branch to trunk until the
>> >> community
>> >> >>> thinks it is good enough and stable enough. We can donate
>> >> >>> ARM machines to the existing CI system for the job.
>> >> >>>
>> >> >>> I wonder if this approch possible?
>> >> >>>
>> >> >>> BR,
>> >> >>>
>> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>> liusheng2048@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>>> Hi,
>> >> >>>>
>> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>> community
>> >> >>>> mentioned by Vinay. I am working on building and
>> >> >>>> testing Hadoop components on aarch64 server these days, besides
>> the
>> >> >>> missing
>> >> >>>> dependices of ARM platform issues #1 #2 #3
>> >> >>>> mentioned by Vinay, other similar issue has also be found, such as
>> >> the
>> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >> >>>>
>> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>> hoped to
>> >> >> add
>> >> >>>> an ARM specific CI to Hadoop repo. we are not
>> >> >>>> sure about if there is any potential effect or confilict on the
>> trunk
>> >> >>>> branch, so maybe creating a ARM specific branch for doing these
>> stuff
>> >> >>>> is a better choice, what do you think?
>> >> >>>>
>> >> >>>> Hope to hear thoughts from you :)
>> >> >>>>
>> >> >>>> BR,
>> >> >>>> Liu sheng
>> >> >>>>
>> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
>> >> >>>>
>> >> >>>>> Hi Folks,
>> >> >>>>>
>> >> >>>>> ARM is becoming famous lately in its processing capability and
>> has
>> >> >> got
>> >> >>>> the
>> >> >>>>> potential to run Bigdata workloads.
>> >> >>>>> Many users have been moving to ARM machines due to its low cost.
>> >> >>>>>
>> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>> (Rasberry
>> >> >> PI)
>> >> >>>> for
>> >> >>>>> experimental purposes. Today ARM architecture is taking some of
>> the
>> >> >>>>> serverside processing as well. So there will be/is a real need of
>> >> >>> Hadoop
>> >> >>>> to
>> >> >>>>> support ARM architecture as well.
>> >> >>>>>
>> >> >>>>> There are bunch of users who are trying out building Hadoop on
>> ARM,
>> >> >>>> trying
>> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>> >> >>>>>
>> >> >>>>> As of today, Hadoop does not compile on ARM due to below issues,
>> >> >> found
>> >> >>>> from
>> >> >>>>> testing done in openlab in [2].
>> >> >>>>>
>> >> >>>>> 1. Protobuf :
>> >> >>>>> -------------------
>> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>> protobuf
>> >> >>>> 2.5.0
>> >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is
>> >> not
>> >> >>>> being
>> >> >>>>> maintained in the community. While protobuf 3.x is being actively
>> >> >>> adopted
>> >> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2
>> >> >>>> messages.
>> >> >>>>> Due to some compilation issues in the generated java code, which
>> can
>> >> >>>> induce
>> >> >>>>> problems in downstream. Due to this reason protobuf upgrade from
>> >> >> 2.5.0
>> >> >>>> was
>> >> >>>>> not taken up.
>> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
>> >> >>> classpath
>> >> >>>>> problem in downstream projects.
>> >> >>>>>    There are patches available to fix compilation in Hadoop. But
>> >> >> need
>> >> >>> to
>> >> >>>>> find a way to upgrade protobuf to latest version and still
>> maintain
>> >> >> the
>> >> >>>>> downstream's classpath using shading feature of Hadoop build.
>> >> >>>>>
>> >> >>>>>     There is a Jira for protobuf upgrade[3] created even before
>> >> >> shade
>> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>> >> >> continue
>> >> >>>>> explore possibilities.
>> >> >>>>>
>> >> >>>>> 2. leveldbjni:
>> >> >>>>> ---------------
>> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>> architecture,
>> >> >>>> need
>> >> >>>>> to check whether any of the future versions support ARM and can
>> >> >> hadoop
>> >> >>>>> upgrade to that version.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>> >> >>>>> -------------------------
>> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
>> >> >>> default
>> >> >>>> in
>> >> >>>>> the maven repository. Workaround is to build it locally and keep
>> in
>> >> >>> local
>> >> >>>>> maven repository.
>> >> >>>>> Need to check whether any future versions of
>> 'protoc-gen-grpc-java'
>> >> >> is
>> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Once the compilation issues are solved, then there might be many
>> >> >> native
>> >> >>>>> code related issues due to different architectures.
>> >> >>>>> So to explore everything, need to join hands together and
>> proceed.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Let us discuss and check, whether any body else out there who
>> also
>> >> >> need
>> >> >>>> the
>> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
>> hands
>> >> >>> and
>> >> >>>>> time in this work.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>> >> >>>>> [2]
>> >> >>
>> >>
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>> >> >>>>>
>> >> >>>>> -Vinay
>> >> >>
>> >>
>> >
>>
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B <vi...@apache.org>]

Hi all,

Thanks for the response.
As I see, protobuf upgrade is long pending and most awaited one.

@Sunil
Protobuf upgrade looks to be a non-trivial task.
Thanks @Duo Zhang for the suggestion of
'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the problem
of dependency on build environment.
However more problem lies in upgrade protobuf without breaking the
downstream builds.
Reason is many downstream projects directly refers server specific jars and
they expect protobuf-2.5.0 jar to get added to classpath by transitive
dependency.

There are some historical discussions and suggestions on
https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
upgrade.
Recommended path for solution is, try to upgrade protobuf using shading of
latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
dependencies for downstreams.
I am trying out ideas on this and if it gets completed within time, may be
we can target trunk itself for the protobuf upgrade.

separate branch idea is suggested for the overall ARM support including
protobuf upgrade and other problems mentioned in the discussion above.

I dont expect separate branch to have a huge changes, apart from bug fixes,
since there are no separate features specific to ARM is being planned.
So timely rebase of separate branch would reduce the overhead on branch
review/merge task.

Still, if the solution to protobuf upgrade winds up early, without any side
effects, I am more than happy to land it in trunk itself.

Thanks,
Vinay
On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <su...@apache.org> wrote:

> Thanks Vinay for starting the thread.
>
> I agree to Anu's view point related to protobuf. And with the suggestion
> pointed out by Duo Zhang, if we can make use
> of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
> of protobuf will also be more easier.
>
> However i think its better to do this effort in trunk itself.
> In offline talks, few members were interested to start 3.3.0 release. And
> given that happens soon, I feel its better
> we do this task in trunk itself as branch diverge is very much possible.
> And to bring to call a merge on such a big branch will be even more tough
> task.
>
> my 2 cents.
>
> Thanks
> Sunil
>
> On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
>> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>> generate
>> the protobuf code. It will download the protoc binary from the maven
>> central so we do not need to install protoc on the build machine any more.
>>
>> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>>
>> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
>> for
>> > over 2 month related to the Protobuf problem .
>> > According to the latest successful build log:
>> >
>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>> > the
>> > os version was ubuntu 14.04 and for the jobs that are failling now such
>> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
>> > the os version is 18.04. I'm not very familiar with the version changing
>> > for the jobs but I did a little search, according to:
>> >
>> >
>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>> > &
>> >
>> >
>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>> > it both said that the version of libprotc-dev and protobuf-compiler
>> > available for ubuntu 18.04 is 3.0.0
>> >
>> >
>> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com> wrote:
>> >
>> >> Thanx Vinay for the initiative, Makes sense to add support for
>> different
>> >> architectures.
>> >>
>> >> +1, for the branch idea.
>> >> Good Luck!!!
>> >>
>> >> -Ayush
>> >>
>> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
>> >> wrote:
>> >> >
>> >> > For HBase, we purged all the protobuf related things from the public
>> >> API,
>> >> > and then upgraded to a shaded and relocated version of protobuf. We
>> have
>> >> > created a repo for this:
>> >> >
>> >> > https://github.com/apache/hbase-thirdparty
>> >> >
>> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>> jars,
>> >> our
>> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
>> >> discuss
>> >> > on how to deal with the upgrading of coprocessor. Glad to see that
>> the
>> >> > hadoop community is also willing to solve the problem.
>> >> >
>> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
>> >> >
>> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
>> that
>> >> >> Hadoop and the downstream projects work correctly after you upgrade
>> >> core
>> >> >> components like Protobuf.
>> >> >> So while branching and working on a branch is easy, merging back
>> after
>> >> you
>> >> >> upgrade some of these core components is insanely hard. You might
>> want
>> >> to
>> >> >> make sure that community buys into upgrading these components in the
>> >> trunk.
>> >> >> That way we will get testing and downstream components will notice
>> when
>> >> >> things break.
>> >> >>
>> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
>> long
>> >> >> time; I have argued that 2.5 is out of support and we cannot stay on
>> >> that
>> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>> code
>> >> base.
>> >> >> It has been rightly pointed to me that while all the arguments I
>> make
>> >> is
>> >> >> correct; it is a very complicated task to upgrade Protobuf, and the
>> >> worst
>> >> >> part is we will not even know what breaks until downstream projects
>> >> pick up
>> >> >> these changes and work against us.
>> >> >>
>> >> >> If we work off the Hadoop version 3 — and assume that we have
>> >> "shading" in
>> >> >> place for all deployments; it might be possible to get there; still
>> a
>> >> >> daunting task.
>> >> >>
>> >> >> So best of luck with the branch approach — But please remember,
>> Merging
>> >> >> back will be hard, Just my 2 cents.
>> >> >>
>> >> >> — Anu
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>> zhengzhenyulixi@gmail.com
>> >> >
>> >> >> wrote:
>> >> >>
>> >> >>> Hi,
>> >> >>>
>> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
>> >> separate
>> >> >>> branch with it's own ARM CI seems a really good idea.
>> >> >>> By doing this we won't break any of the undergoing development in
>> >> trunk
>> >> >> and
>> >> >>> a CI can be a very good way to show what are the
>> >> >>> current problems and what have been fixed, it will also provide a
>> very
>> >> >> good
>> >> >>> view for contributors that are intrested to working on
>> >> >>> this. We can finally merge back the branch to trunk until the
>> >> community
>> >> >>> thinks it is good enough and stable enough. We can donate
>> >> >>> ARM machines to the existing CI system for the job.
>> >> >>>
>> >> >>> I wonder if this approch possible?
>> >> >>>
>> >> >>> BR,
>> >> >>>
>> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>> liusheng2048@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>>> Hi,
>> >> >>>>
>> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>> community
>> >> >>>> mentioned by Vinay. I am working on building and
>> >> >>>> testing Hadoop components on aarch64 server these days, besides
>> the
>> >> >>> missing
>> >> >>>> dependices of ARM platform issues #1 #2 #3
>> >> >>>> mentioned by Vinay, other similar issue has also be found, such as
>> >> the
>> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >> >>>>
>> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>> hoped to
>> >> >> add
>> >> >>>> an ARM specific CI to Hadoop repo. we are not
>> >> >>>> sure about if there is any potential effect or confilict on the
>> trunk
>> >> >>>> branch, so maybe creating a ARM specific branch for doing these
>> stuff
>> >> >>>> is a better choice, what do you think?
>> >> >>>>
>> >> >>>> Hope to hear thoughts from you :)
>> >> >>>>
>> >> >>>> BR,
>> >> >>>> Liu sheng
>> >> >>>>
>> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
>> >> >>>>
>> >> >>>>> Hi Folks,
>> >> >>>>>
>> >> >>>>> ARM is becoming famous lately in its processing capability and
>> has
>> >> >> got
>> >> >>>> the
>> >> >>>>> potential to run Bigdata workloads.
>> >> >>>>> Many users have been moving to ARM machines due to its low cost.
>> >> >>>>>
>> >> >>>>> In the past there were attempts to compile Hadoop on ARM
>> (Rasberry
>> >> >> PI)
>> >> >>>> for
>> >> >>>>> experimental purposes. Today ARM architecture is taking some of
>> the
>> >> >>>>> serverside processing as well. So there will be/is a real need of
>> >> >>> Hadoop
>> >> >>>> to
>> >> >>>>> support ARM architecture as well.
>> >> >>>>>
>> >> >>>>> There are bunch of users who are trying out building Hadoop on
>> ARM,
>> >> >>>> trying
>> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>> >> >>>>>
>> >> >>>>> As of today, Hadoop does not compile on ARM due to below issues,
>> >> >> found
>> >> >>>> from
>> >> >>>>> testing done in openlab in [2].
>> >> >>>>>
>> >> >>>>> 1. Protobuf :
>> >> >>>>> -------------------
>> >> >>>>>     Hadoop project (also some downstream projects) stuck to
>> protobuf
>> >> >>>> 2.5.0
>> >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is
>> >> not
>> >> >>>> being
>> >> >>>>> maintained in the community. While protobuf 3.x is being actively
>> >> >>> adopted
>> >> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2
>> >> >>>> messages.
>> >> >>>>> Due to some compilation issues in the generated java code, which
>> can
>> >> >>>> induce
>> >> >>>>> problems in downstream. Due to this reason protobuf upgrade from
>> >> >> 2.5.0
>> >> >>>> was
>> >> >>>>> not taken up.
>> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
>> >> >>> classpath
>> >> >>>>> problem in downstream projects.
>> >> >>>>>    There are patches available to fix compilation in Hadoop. But
>> >> >> need
>> >> >>> to
>> >> >>>>> find a way to upgrade protobuf to latest version and still
>> maintain
>> >> >> the
>> >> >>>>> downstream's classpath using shading feature of Hadoop build.
>> >> >>>>>
>> >> >>>>>     There is a Jira for protobuf upgrade[3] created even before
>> >> >> shade
>> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>> >> >> continue
>> >> >>>>> explore possibilities.
>> >> >>>>>
>> >> >>>>> 2. leveldbjni:
>> >> >>>>> ---------------
>> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
>> architecture,
>> >> >>>> need
>> >> >>>>> to check whether any of the future versions support ARM and can
>> >> >> hadoop
>> >> >>>>> upgrade to that version.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>> >> >>>>> -------------------------
>> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
>> >> >>> default
>> >> >>>> in
>> >> >>>>> the maven repository. Workaround is to build it locally and keep
>> in
>> >> >>> local
>> >> >>>>> maven repository.
>> >> >>>>> Need to check whether any future versions of
>> 'protoc-gen-grpc-java'
>> >> >> is
>> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Once the compilation issues are solved, then there might be many
>> >> >> native
>> >> >>>>> code related issues due to different architectures.
>> >> >>>>> So to explore everything, need to join hands together and
>> proceed.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Let us discuss and check, whether any body else out there who
>> also
>> >> >> need
>> >> >>>> the
>> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
>> hands
>> >> >>> and
>> >> >>>>> time in this work.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>> >> >>>>> [2]
>> >> >>
>> >>
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>> >> >>>>>
>> >> >>>>> -Vinay
>> >> >>
>> >>
>> >
>>
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Sunil Govindan <su...@apache.org>]

Thanks Vinay for starting the thread.

I agree to Anu's view point related to protobuf. And with the suggestion
pointed out by Duo Zhang, if we can make use
of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
of protobuf will also be more easier.

However i think its better to do this effort in trunk itself.
In offline talks, few members were interested to start 3.3.0 release. And
given that happens soon, I feel its better
we do this task in trunk itself as branch diverge is very much possible.
And to bring to call a merge on such a big branch will be even more tough
task.

my 2 cents.

Thanks
Sunil

On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com> wrote:

> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to generate
> the protobuf code. It will download the protoc binary from the maven
> central so we do not need to install protoc on the build machine any more.
>
> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>
> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
> for
> > over 2 month related to the Protobuf problem .
> > According to the latest successful build log:
> >
> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
> > the
> > os version was ubuntu 14.04 and for the jobs that are failling now such
> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
> > the os version is 18.04. I'm not very familiar with the version changing
> > for the jobs but I did a little search, according to:
> >
> >
> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
> > &
> >
> >
> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
> > it both said that the version of libprotc-dev and protobuf-compiler
> > available for ubuntu 18.04 is 3.0.0
> >
> >
> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com> wrote:
> >
> >> Thanx Vinay for the initiative, Makes sense to add support for different
> >> architectures.
> >>
> >> +1, for the branch idea.
> >> Good Luck!!!
> >>
> >> -Ayush
> >>
> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> >> wrote:
> >> >
> >> > For HBase, we purged all the protobuf related things from the public
> >> API,
> >> > and then upgraded to a shaded and relocated version of protobuf. We
> have
> >> > created a repo for this:
> >> >
> >> > https://github.com/apache/hbase-thirdparty
> >> >
> >> > But since the hadoop dependencies still pull in the protobuf 2.5 jars,
> >> our
> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
> >> discuss
> >> > on how to deal with the upgrading of coprocessor. Glad to see that the
> >> > hadoop community is also willing to solve the problem.
> >> >
> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
> >> >
> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
> that
> >> >> Hadoop and the downstream projects work correctly after you upgrade
> >> core
> >> >> components like Protobuf.
> >> >> So while branching and working on a branch is easy, merging back
> after
> >> you
> >> >> upgrade some of these core components is insanely hard. You might
> want
> >> to
> >> >> make sure that community buys into upgrading these components in the
> >> trunk.
> >> >> That way we will get testing and downstream components will notice
> when
> >> >> things break.
> >> >>
> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
> long
> >> >> time; I have argued that 2.5 is out of support and we cannot stay on
> >> that
> >> >> branch forever; or we need to take ownership of the Protobuf 2.5 code
> >> base.
> >> >> It has been rightly pointed to me that while all the arguments I make
> >> is
> >> >> correct; it is a very complicated task to upgrade Protobuf, and the
> >> worst
> >> >> part is we will not even know what breaks until downstream projects
> >> pick up
> >> >> these changes and work against us.
> >> >>
> >> >> If we work off the Hadoop version 3 — and assume that we have
> >> "shading" in
> >> >> place for all deployments; it might be possible to get there; still a
> >> >> daunting task.
> >> >>
> >> >> So best of luck with the branch approach — But please remember,
> Merging
> >> >> back will be hard, Just my 2 cents.
> >> >>
> >> >> — Anu
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
> zhengzhenyulixi@gmail.com
> >> >
> >> >> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
> >> separate
> >> >>> branch with it's own ARM CI seems a really good idea.
> >> >>> By doing this we won't break any of the undergoing development in
> >> trunk
> >> >> and
> >> >>> a CI can be a very good way to show what are the
> >> >>> current problems and what have been fixed, it will also provide a
> very
> >> >> good
> >> >>> view for contributors that are intrested to working on
> >> >>> this. We can finally merge back the branch to trunk until the
> >> community
> >> >>> thinks it is good enough and stable enough. We can donate
> >> >>> ARM machines to the existing CI system for the job.
> >> >>>
> >> >>> I wonder if this approch possible?
> >> >>>
> >> >>> BR,
> >> >>>
> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <liusheng2048@gmail.com
> >
> >> >>> wrote:
> >> >>>
> >> >>>> Hi,
> >> >>>>
> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
> community
> >> >>>> mentioned by Vinay. I am working on building and
> >> >>>> testing Hadoop components on aarch64 server these days, besides the
> >> >>> missing
> >> >>>> dependices of ARM platform issues #1 #2 #3
> >> >>>> mentioned by Vinay, other similar issue has also be found, such as
> >> the
> >> >>>> "PhantomJS" dependent package also missing for aarch64.
> >> >>>>
> >> >>>> To promote the ARM support for Hadoop, we have discussed and hoped
> to
> >> >> add
> >> >>>> an ARM specific CI to Hadoop repo. we are not
> >> >>>> sure about if there is any potential effect or confilict on the
> trunk
> >> >>>> branch, so maybe creating a ARM specific branch for doing these
> stuff
> >> >>>> is a better choice, what do you think?
> >> >>>>
> >> >>>> Hope to hear thoughts from you :)
> >> >>>>
> >> >>>> BR,
> >> >>>> Liu sheng
> >> >>>>
> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> >> >>>>
> >> >>>>> Hi Folks,
> >> >>>>>
> >> >>>>> ARM is becoming famous lately in its processing capability and has
> >> >> got
> >> >>>> the
> >> >>>>> potential to run Bigdata workloads.
> >> >>>>> Many users have been moving to ARM machines due to its low cost.
> >> >>>>>
> >> >>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry
> >> >> PI)
> >> >>>> for
> >> >>>>> experimental purposes. Today ARM architecture is taking some of
> the
> >> >>>>> serverside processing as well. So there will be/is a real need of
> >> >>> Hadoop
> >> >>>> to
> >> >>>>> support ARM architecture as well.
> >> >>>>>
> >> >>>>> There are bunch of users who are trying out building Hadoop on
> ARM,
> >> >>>> trying
> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
> >> >>>>>
> >> >>>>> As of today, Hadoop does not compile on ARM due to below issues,
> >> >> found
> >> >>>> from
> >> >>>>> testing done in openlab in [2].
> >> >>>>>
> >> >>>>> 1. Protobuf :
> >> >>>>> -------------------
> >> >>>>>     Hadoop project (also some downstream projects) stuck to
> protobuf
> >> >>>> 2.5.0
> >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is
> >> not
> >> >>>> being
> >> >>>>> maintained in the community. While protobuf 3.x is being actively
> >> >>> adopted
> >> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2
> >> >>>> messages.
> >> >>>>> Due to some compilation issues in the generated java code, which
> can
> >> >>>> induce
> >> >>>>> problems in downstream. Due to this reason protobuf upgrade from
> >> >> 2.5.0
> >> >>>> was
> >> >>>>> not taken up.
> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> >> >>> classpath
> >> >>>>> problem in downstream projects.
> >> >>>>>    There are patches available to fix compilation in Hadoop. But
> >> >> need
> >> >>> to
> >> >>>>> find a way to upgrade protobuf to latest version and still
> maintain
> >> >> the
> >> >>>>> downstream's classpath using shading feature of Hadoop build.
> >> >>>>>
> >> >>>>>     There is a Jira for protobuf upgrade[3] created even before
> >> >> shade
> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
> >> >> continue
> >> >>>>> explore possibilities.
> >> >>>>>
> >> >>>>> 2. leveldbjni:
> >> >>>>> ---------------
> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
> architecture,
> >> >>>> need
> >> >>>>> to check whether any of the future versions support ARM and can
> >> >> hadoop
> >> >>>>> upgrade to that version.
> >> >>>>>
> >> >>>>>
> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> >> >>>>> -------------------------
> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
> >> >>> default
> >> >>>> in
> >> >>>>> the maven repository. Workaround is to build it locally and keep
> in
> >> >>> local
> >> >>>>> maven repository.
> >> >>>>> Need to check whether any future versions of
> 'protoc-gen-grpc-java'
> >> >> is
> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
> >> >>>>>
> >> >>>>>
> >> >>>>> Once the compilation issues are solved, then there might be many
> >> >> native
> >> >>>>> code related issues due to different architectures.
> >> >>>>> So to explore everything, need to join hands together and proceed.
> >> >>>>>
> >> >>>>>
> >> >>>>> Let us discuss and check, whether any body else out there who also
> >> >> need
> >> >>>> the
> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
> hands
> >> >>> and
> >> >>>>> time in this work.
> >> >>>>>
> >> >>>>>
> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
> >> >>>>> [2]
> >> >>
> >>
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
> >> >>>>>
> >> >>>>> -Vinay
> >> >>
> >>
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Sunil Govindan <su...@apache.org>]

Thanks Vinay for starting the thread.

I agree to Anu's view point related to protobuf. And with the suggestion
pointed out by Duo Zhang, if we can make use
of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
of protobuf will also be more easier.

However i think its better to do this effort in trunk itself.
In offline talks, few members were interested to start 3.3.0 release. And
given that happens soon, I feel its better
we do this task in trunk itself as branch diverge is very much possible.
And to bring to call a merge on such a big branch will be even more tough
task.

my 2 cents.

Thanks
Sunil

On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com> wrote:

> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to generate
> the protobuf code. It will download the protoc binary from the maven
> central so we do not need to install protoc on the build machine any more.
>
> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>
> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
> for
> > over 2 month related to the Protobuf problem .
> > According to the latest successful build log:
> >
> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
> > the
> > os version was ubuntu 14.04 and for the jobs that are failling now such
> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
> > the os version is 18.04. I'm not very familiar with the version changing
> > for the jobs but I did a little search, according to:
> >
> >
> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
> > &
> >
> >
> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
> > it both said that the version of libprotc-dev and protobuf-compiler
> > available for ubuntu 18.04 is 3.0.0
> >
> >
> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com> wrote:
> >
> >> Thanx Vinay for the initiative, Makes sense to add support for different
> >> architectures.
> >>
> >> +1, for the branch idea.
> >> Good Luck!!!
> >>
> >> -Ayush
> >>
> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> >> wrote:
> >> >
> >> > For HBase, we purged all the protobuf related things from the public
> >> API,
> >> > and then upgraded to a shaded and relocated version of protobuf. We
> have
> >> > created a repo for this:
> >> >
> >> > https://github.com/apache/hbase-thirdparty
> >> >
> >> > But since the hadoop dependencies still pull in the protobuf 2.5 jars,
> >> our
> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
> >> discuss
> >> > on how to deal with the upgrading of coprocessor. Glad to see that the
> >> > hadoop community is also willing to solve the problem.
> >> >
> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
> >> >
> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
> that
> >> >> Hadoop and the downstream projects work correctly after you upgrade
> >> core
> >> >> components like Protobuf.
> >> >> So while branching and working on a branch is easy, merging back
> after
> >> you
> >> >> upgrade some of these core components is insanely hard. You might
> want
> >> to
> >> >> make sure that community buys into upgrading these components in the
> >> trunk.
> >> >> That way we will get testing and downstream components will notice
> when
> >> >> things break.
> >> >>
> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
> long
> >> >> time; I have argued that 2.5 is out of support and we cannot stay on
> >> that
> >> >> branch forever; or we need to take ownership of the Protobuf 2.5 code
> >> base.
> >> >> It has been rightly pointed to me that while all the arguments I make
> >> is
> >> >> correct; it is a very complicated task to upgrade Protobuf, and the
> >> worst
> >> >> part is we will not even know what breaks until downstream projects
> >> pick up
> >> >> these changes and work against us.
> >> >>
> >> >> If we work off the Hadoop version 3 — and assume that we have
> >> "shading" in
> >> >> place for all deployments; it might be possible to get there; still a
> >> >> daunting task.
> >> >>
> >> >> So best of luck with the branch approach — But please remember,
> Merging
> >> >> back will be hard, Just my 2 cents.
> >> >>
> >> >> — Anu
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
> zhengzhenyulixi@gmail.com
> >> >
> >> >> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
> >> separate
> >> >>> branch with it's own ARM CI seems a really good idea.
> >> >>> By doing this we won't break any of the undergoing development in
> >> trunk
> >> >> and
> >> >>> a CI can be a very good way to show what are the
> >> >>> current problems and what have been fixed, it will also provide a
> very
> >> >> good
> >> >>> view for contributors that are intrested to working on
> >> >>> this. We can finally merge back the branch to trunk until the
> >> community
> >> >>> thinks it is good enough and stable enough. We can donate
> >> >>> ARM machines to the existing CI system for the job.
> >> >>>
> >> >>> I wonder if this approch possible?
> >> >>>
> >> >>> BR,
> >> >>>
> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <liusheng2048@gmail.com
> >
> >> >>> wrote:
> >> >>>
> >> >>>> Hi,
> >> >>>>
> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
> community
> >> >>>> mentioned by Vinay. I am working on building and
> >> >>>> testing Hadoop components on aarch64 server these days, besides the
> >> >>> missing
> >> >>>> dependices of ARM platform issues #1 #2 #3
> >> >>>> mentioned by Vinay, other similar issue has also be found, such as
> >> the
> >> >>>> "PhantomJS" dependent package also missing for aarch64.
> >> >>>>
> >> >>>> To promote the ARM support for Hadoop, we have discussed and hoped
> to
> >> >> add
> >> >>>> an ARM specific CI to Hadoop repo. we are not
> >> >>>> sure about if there is any potential effect or confilict on the
> trunk
> >> >>>> branch, so maybe creating a ARM specific branch for doing these
> stuff
> >> >>>> is a better choice, what do you think?
> >> >>>>
> >> >>>> Hope to hear thoughts from you :)
> >> >>>>
> >> >>>> BR,
> >> >>>> Liu sheng
> >> >>>>
> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> >> >>>>
> >> >>>>> Hi Folks,
> >> >>>>>
> >> >>>>> ARM is becoming famous lately in its processing capability and has
> >> >> got
> >> >>>> the
> >> >>>>> potential to run Bigdata workloads.
> >> >>>>> Many users have been moving to ARM machines due to its low cost.
> >> >>>>>
> >> >>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry
> >> >> PI)
> >> >>>> for
> >> >>>>> experimental purposes. Today ARM architecture is taking some of
> the
> >> >>>>> serverside processing as well. So there will be/is a real need of
> >> >>> Hadoop
> >> >>>> to
> >> >>>>> support ARM architecture as well.
> >> >>>>>
> >> >>>>> There are bunch of users who are trying out building Hadoop on
> ARM,
> >> >>>> trying
> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
> >> >>>>>
> >> >>>>> As of today, Hadoop does not compile on ARM due to below issues,
> >> >> found
> >> >>>> from
> >> >>>>> testing done in openlab in [2].
> >> >>>>>
> >> >>>>> 1. Protobuf :
> >> >>>>> -------------------
> >> >>>>>     Hadoop project (also some downstream projects) stuck to
> protobuf
> >> >>>> 2.5.0
> >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is
> >> not
> >> >>>> being
> >> >>>>> maintained in the community. While protobuf 3.x is being actively
> >> >>> adopted
> >> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2
> >> >>>> messages.
> >> >>>>> Due to some compilation issues in the generated java code, which
> can
> >> >>>> induce
> >> >>>>> problems in downstream. Due to this reason protobuf upgrade from
> >> >> 2.5.0
> >> >>>> was
> >> >>>>> not taken up.
> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> >> >>> classpath
> >> >>>>> problem in downstream projects.
> >> >>>>>    There are patches available to fix compilation in Hadoop. But
> >> >> need
> >> >>> to
> >> >>>>> find a way to upgrade protobuf to latest version and still
> maintain
> >> >> the
> >> >>>>> downstream's classpath using shading feature of Hadoop build.
> >> >>>>>
> >> >>>>>     There is a Jira for protobuf upgrade[3] created even before
> >> >> shade
> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
> >> >> continue
> >> >>>>> explore possibilities.
> >> >>>>>
> >> >>>>> 2. leveldbjni:
> >> >>>>> ---------------
> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
> architecture,
> >> >>>> need
> >> >>>>> to check whether any of the future versions support ARM and can
> >> >> hadoop
> >> >>>>> upgrade to that version.
> >> >>>>>
> >> >>>>>
> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> >> >>>>> -------------------------
> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
> >> >>> default
> >> >>>> in
> >> >>>>> the maven repository. Workaround is to build it locally and keep
> in
> >> >>> local
> >> >>>>> maven repository.
> >> >>>>> Need to check whether any future versions of
> 'protoc-gen-grpc-java'
> >> >> is
> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
> >> >>>>>
> >> >>>>>
> >> >>>>> Once the compilation issues are solved, then there might be many
> >> >> native
> >> >>>>> code related issues due to different architectures.
> >> >>>>> So to explore everything, need to join hands together and proceed.
> >> >>>>>
> >> >>>>>
> >> >>>>> Let us discuss and check, whether any body else out there who also
> >> >> need
> >> >>>> the
> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
> hands
> >> >>> and
> >> >>>>> time in this work.
> >> >>>>>
> >> >>>>>
> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
> >> >>>>> [2]
> >> >>
> >>
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
> >> >>>>>
> >> >>>>> -Vinay
> >> >>
> >>
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Sunil Govindan <su...@apache.org>]

Thanks Vinay for starting the thread.

I agree to Anu's view point related to protobuf. And with the suggestion
pointed out by Duo Zhang, if we can make use
of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
of protobuf will also be more easier.

However i think its better to do this effort in trunk itself.
In offline talks, few members were interested to start 3.3.0 release. And
given that happens soon, I feel its better
we do this task in trunk itself as branch diverge is very much possible.
And to bring to call a merge on such a big branch will be even more tough
task.

my 2 cents.

Thanks
Sunil

On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com> wrote:

> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to generate
> the protobuf code. It will download the protoc binary from the maven
> central so we do not need to install protoc on the build machine any more.
>
> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>
> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
> for
> > over 2 month related to the Protobuf problem .
> > According to the latest successful build log:
> >
> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
> > the
> > os version was ubuntu 14.04 and for the jobs that are failling now such
> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
> > the os version is 18.04. I'm not very familiar with the version changing
> > for the jobs but I did a little search, according to:
> >
> >
> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
> > &
> >
> >
> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
> > it both said that the version of libprotc-dev and protobuf-compiler
> > available for ubuntu 18.04 is 3.0.0
> >
> >
> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com> wrote:
> >
> >> Thanx Vinay for the initiative, Makes sense to add support for different
> >> architectures.
> >>
> >> +1, for the branch idea.
> >> Good Luck!!!
> >>
> >> -Ayush
> >>
> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> >> wrote:
> >> >
> >> > For HBase, we purged all the protobuf related things from the public
> >> API,
> >> > and then upgraded to a shaded and relocated version of protobuf. We
> have
> >> > created a repo for this:
> >> >
> >> > https://github.com/apache/hbase-thirdparty
> >> >
> >> > But since the hadoop dependencies still pull in the protobuf 2.5 jars,
> >> our
> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
> >> discuss
> >> > on how to deal with the upgrading of coprocessor. Glad to see that the
> >> > hadoop community is also willing to solve the problem.
> >> >
> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
> >> >
> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
> that
> >> >> Hadoop and the downstream projects work correctly after you upgrade
> >> core
> >> >> components like Protobuf.
> >> >> So while branching and working on a branch is easy, merging back
> after
> >> you
> >> >> upgrade some of these core components is insanely hard. You might
> want
> >> to
> >> >> make sure that community buys into upgrading these components in the
> >> trunk.
> >> >> That way we will get testing and downstream components will notice
> when
> >> >> things break.
> >> >>
> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
> long
> >> >> time; I have argued that 2.5 is out of support and we cannot stay on
> >> that
> >> >> branch forever; or we need to take ownership of the Protobuf 2.5 code
> >> base.
> >> >> It has been rightly pointed to me that while all the arguments I make
> >> is
> >> >> correct; it is a very complicated task to upgrade Protobuf, and the
> >> worst
> >> >> part is we will not even know what breaks until downstream projects
> >> pick up
> >> >> these changes and work against us.
> >> >>
> >> >> If we work off the Hadoop version 3 — and assume that we have
> >> "shading" in
> >> >> place for all deployments; it might be possible to get there; still a
> >> >> daunting task.
> >> >>
> >> >> So best of luck with the branch approach — But please remember,
> Merging
> >> >> back will be hard, Just my 2 cents.
> >> >>
> >> >> — Anu
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
> zhengzhenyulixi@gmail.com
> >> >
> >> >> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
> >> separate
> >> >>> branch with it's own ARM CI seems a really good idea.
> >> >>> By doing this we won't break any of the undergoing development in
> >> trunk
> >> >> and
> >> >>> a CI can be a very good way to show what are the
> >> >>> current problems and what have been fixed, it will also provide a
> very
> >> >> good
> >> >>> view for contributors that are intrested to working on
> >> >>> this. We can finally merge back the branch to trunk until the
> >> community
> >> >>> thinks it is good enough and stable enough. We can donate
> >> >>> ARM machines to the existing CI system for the job.
> >> >>>
> >> >>> I wonder if this approch possible?
> >> >>>
> >> >>> BR,
> >> >>>
> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <liusheng2048@gmail.com
> >
> >> >>> wrote:
> >> >>>
> >> >>>> Hi,
> >> >>>>
> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
> community
> >> >>>> mentioned by Vinay. I am working on building and
> >> >>>> testing Hadoop components on aarch64 server these days, besides the
> >> >>> missing
> >> >>>> dependices of ARM platform issues #1 #2 #3
> >> >>>> mentioned by Vinay, other similar issue has also be found, such as
> >> the
> >> >>>> "PhantomJS" dependent package also missing for aarch64.
> >> >>>>
> >> >>>> To promote the ARM support for Hadoop, we have discussed and hoped
> to
> >> >> add
> >> >>>> an ARM specific CI to Hadoop repo. we are not
> >> >>>> sure about if there is any potential effect or confilict on the
> trunk
> >> >>>> branch, so maybe creating a ARM specific branch for doing these
> stuff
> >> >>>> is a better choice, what do you think?
> >> >>>>
> >> >>>> Hope to hear thoughts from you :)
> >> >>>>
> >> >>>> BR,
> >> >>>> Liu sheng
> >> >>>>
> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> >> >>>>
> >> >>>>> Hi Folks,
> >> >>>>>
> >> >>>>> ARM is becoming famous lately in its processing capability and has
> >> >> got
> >> >>>> the
> >> >>>>> potential to run Bigdata workloads.
> >> >>>>> Many users have been moving to ARM machines due to its low cost.
> >> >>>>>
> >> >>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry
> >> >> PI)
> >> >>>> for
> >> >>>>> experimental purposes. Today ARM architecture is taking some of
> the
> >> >>>>> serverside processing as well. So there will be/is a real need of
> >> >>> Hadoop
> >> >>>> to
> >> >>>>> support ARM architecture as well.
> >> >>>>>
> >> >>>>> There are bunch of users who are trying out building Hadoop on
> ARM,
> >> >>>> trying
> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
> >> >>>>>
> >> >>>>> As of today, Hadoop does not compile on ARM due to below issues,
> >> >> found
> >> >>>> from
> >> >>>>> testing done in openlab in [2].
> >> >>>>>
> >> >>>>> 1. Protobuf :
> >> >>>>> -------------------
> >> >>>>>     Hadoop project (also some downstream projects) stuck to
> protobuf
> >> >>>> 2.5.0
> >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is
> >> not
> >> >>>> being
> >> >>>>> maintained in the community. While protobuf 3.x is being actively
> >> >>> adopted
> >> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2
> >> >>>> messages.
> >> >>>>> Due to some compilation issues in the generated java code, which
> can
> >> >>>> induce
> >> >>>>> problems in downstream. Due to this reason protobuf upgrade from
> >> >> 2.5.0
> >> >>>> was
> >> >>>>> not taken up.
> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> >> >>> classpath
> >> >>>>> problem in downstream projects.
> >> >>>>>    There are patches available to fix compilation in Hadoop. But
> >> >> need
> >> >>> to
> >> >>>>> find a way to upgrade protobuf to latest version and still
> maintain
> >> >> the
> >> >>>>> downstream's classpath using shading feature of Hadoop build.
> >> >>>>>
> >> >>>>>     There is a Jira for protobuf upgrade[3] created even before
> >> >> shade
> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
> >> >> continue
> >> >>>>> explore possibilities.
> >> >>>>>
> >> >>>>> 2. leveldbjni:
> >> >>>>> ---------------
> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
> architecture,
> >> >>>> need
> >> >>>>> to check whether any of the future versions support ARM and can
> >> >> hadoop
> >> >>>>> upgrade to that version.
> >> >>>>>
> >> >>>>>
> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> >> >>>>> -------------------------
> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
> >> >>> default
> >> >>>> in
> >> >>>>> the maven repository. Workaround is to build it locally and keep
> in
> >> >>> local
> >> >>>>> maven repository.
> >> >>>>> Need to check whether any future versions of
> 'protoc-gen-grpc-java'
> >> >> is
> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
> >> >>>>>
> >> >>>>>
> >> >>>>> Once the compilation issues are solved, then there might be many
> >> >> native
> >> >>>>> code related issues due to different architectures.
> >> >>>>> So to explore everything, need to join hands together and proceed.
> >> >>>>>
> >> >>>>>
> >> >>>>> Let us discuss and check, whether any body else out there who also
> >> >> need
> >> >>>> the
> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
> hands
> >> >>> and
> >> >>>>> time in this work.
> >> >>>>>
> >> >>>>>
> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
> >> >>>>> [2]
> >> >>
> >>
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
> >> >>>>>
> >> >>>>> -Vinay
> >> >>
> >>
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Sunil Govindan <su...@apache.org>]

Thanks Vinay for starting the thread.

I agree to Anu's view point related to protobuf. And with the suggestion
pointed out by Duo Zhang, if we can make use
of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
of protobuf will also be more easier.

However i think its better to do this effort in trunk itself.
In offline talks, few members were interested to start 3.3.0 release. And
given that happens soon, I feel its better
we do this task in trunk itself as branch diverge is very much possible.
And to bring to call a merge on such a big branch will be even more tough
task.

my 2 cents.

Thanks
Sunil

On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <pa...@gmail.com> wrote:

> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to generate
> the protobuf code. It will download the protoc binary from the maven
> central so we do not need to install protoc on the build machine any more.
>
> Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：
>
> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
> for
> > over 2 month related to the Protobuf problem .
> > According to the latest successful build log:
> >
> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
> > the
> > os version was ubuntu 14.04 and for the jobs that are failling now such
> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
> > the os version is 18.04. I'm not very familiar with the version changing
> > for the jobs but I did a little search, according to:
> >
> >
> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
> > &
> >
> >
> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
> > it both said that the version of libprotc-dev and protobuf-compiler
> > available for ubuntu 18.04 is 3.0.0
> >
> >
> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com> wrote:
> >
> >> Thanx Vinay for the initiative, Makes sense to add support for different
> >> architectures.
> >>
> >> +1, for the branch idea.
> >> Good Luck!!!
> >>
> >> -Ayush
> >>
> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> >> wrote:
> >> >
> >> > For HBase, we purged all the protobuf related things from the public
> >> API,
> >> > and then upgraded to a shaded and relocated version of protobuf. We
> have
> >> > created a repo for this:
> >> >
> >> > https://github.com/apache/hbase-thirdparty
> >> >
> >> > But since the hadoop dependencies still pull in the protobuf 2.5 jars,
> >> our
> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
> >> discuss
> >> > on how to deal with the upgrading of coprocessor. Glad to see that the
> >> > hadoop community is also willing to solve the problem.
> >> >
> >> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
> >> >
> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
> that
> >> >> Hadoop and the downstream projects work correctly after you upgrade
> >> core
> >> >> components like Protobuf.
> >> >> So while branching and working on a branch is easy, merging back
> after
> >> you
> >> >> upgrade some of these core components is insanely hard. You might
> want
> >> to
> >> >> make sure that community buys into upgrading these components in the
> >> trunk.
> >> >> That way we will get testing and downstream components will notice
> when
> >> >> things break.
> >> >>
> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
> long
> >> >> time; I have argued that 2.5 is out of support and we cannot stay on
> >> that
> >> >> branch forever; or we need to take ownership of the Protobuf 2.5 code
> >> base.
> >> >> It has been rightly pointed to me that while all the arguments I make
> >> is
> >> >> correct; it is a very complicated task to upgrade Protobuf, and the
> >> worst
> >> >> part is we will not even know what breaks until downstream projects
> >> pick up
> >> >> these changes and work against us.
> >> >>
> >> >> If we work off the Hadoop version 3 — and assume that we have
> >> "shading" in
> >> >> place for all deployments; it might be possible to get there; still a
> >> >> daunting task.
> >> >>
> >> >> So best of luck with the branch approach — But please remember,
> Merging
> >> >> back will be hard, Just my 2 cents.
> >> >>
> >> >> — Anu
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
> zhengzhenyulixi@gmail.com
> >> >
> >> >> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
> >> separate
> >> >>> branch with it's own ARM CI seems a really good idea.
> >> >>> By doing this we won't break any of the undergoing development in
> >> trunk
> >> >> and
> >> >>> a CI can be a very good way to show what are the
> >> >>> current problems and what have been fixed, it will also provide a
> very
> >> >> good
> >> >>> view for contributors that are intrested to working on
> >> >>> this. We can finally merge back the branch to trunk until the
> >> community
> >> >>> thinks it is good enough and stable enough. We can donate
> >> >>> ARM machines to the existing CI system for the job.
> >> >>>
> >> >>> I wonder if this approch possible?
> >> >>>
> >> >>> BR,
> >> >>>
> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <liusheng2048@gmail.com
> >
> >> >>> wrote:
> >> >>>
> >> >>>> Hi,
> >> >>>>
> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
> community
> >> >>>> mentioned by Vinay. I am working on building and
> >> >>>> testing Hadoop components on aarch64 server these days, besides the
> >> >>> missing
> >> >>>> dependices of ARM platform issues #1 #2 #3
> >> >>>> mentioned by Vinay, other similar issue has also be found, such as
> >> the
> >> >>>> "PhantomJS" dependent package also missing for aarch64.
> >> >>>>
> >> >>>> To promote the ARM support for Hadoop, we have discussed and hoped
> to
> >> >> add
> >> >>>> an ARM specific CI to Hadoop repo. we are not
> >> >>>> sure about if there is any potential effect or confilict on the
> trunk
> >> >>>> branch, so maybe creating a ARM specific branch for doing these
> stuff
> >> >>>> is a better choice, what do you think?
> >> >>>>
> >> >>>> Hope to hear thoughts from you :)
> >> >>>>
> >> >>>> BR,
> >> >>>> Liu sheng
> >> >>>>
> >> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> >> >>>>
> >> >>>>> Hi Folks,
> >> >>>>>
> >> >>>>> ARM is becoming famous lately in its processing capability and has
> >> >> got
> >> >>>> the
> >> >>>>> potential to run Bigdata workloads.
> >> >>>>> Many users have been moving to ARM machines due to its low cost.
> >> >>>>>
> >> >>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry
> >> >> PI)
> >> >>>> for
> >> >>>>> experimental purposes. Today ARM architecture is taking some of
> the
> >> >>>>> serverside processing as well. So there will be/is a real need of
> >> >>> Hadoop
> >> >>>> to
> >> >>>>> support ARM architecture as well.
> >> >>>>>
> >> >>>>> There are bunch of users who are trying out building Hadoop on
> ARM,
> >> >>>> trying
> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
> >> >>>>>
> >> >>>>> As of today, Hadoop does not compile on ARM due to below issues,
> >> >> found
> >> >>>> from
> >> >>>>> testing done in openlab in [2].
> >> >>>>>
> >> >>>>> 1. Protobuf :
> >> >>>>> -------------------
> >> >>>>>     Hadoop project (also some downstream projects) stuck to
> protobuf
> >> >>>> 2.5.0
> >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is
> >> not
> >> >>>> being
> >> >>>>> maintained in the community. While protobuf 3.x is being actively
> >> >>> adopted
> >> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2
> >> >>>> messages.
> >> >>>>> Due to some compilation issues in the generated java code, which
> can
> >> >>>> induce
> >> >>>>> problems in downstream. Due to this reason protobuf upgrade from
> >> >> 2.5.0
> >> >>>> was
> >> >>>>> not taken up.
> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> >> >>> classpath
> >> >>>>> problem in downstream projects.
> >> >>>>>    There are patches available to fix compilation in Hadoop. But
> >> >> need
> >> >>> to
> >> >>>>> find a way to upgrade protobuf to latest version and still
> maintain
> >> >> the
> >> >>>>> downstream's classpath using shading feature of Hadoop build.
> >> >>>>>
> >> >>>>>     There is a Jira for protobuf upgrade[3] created even before
> >> >> shade
> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and
> >> >> continue
> >> >>>>> explore possibilities.
> >> >>>>>
> >> >>>>> 2. leveldbjni:
> >> >>>>> ---------------
> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
> architecture,
> >> >>>> need
> >> >>>>> to check whether any of the future versions support ARM and can
> >> >> hadoop
> >> >>>>> upgrade to that version.
> >> >>>>>
> >> >>>>>
> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> >> >>>>> -------------------------
> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
> >> >>> default
> >> >>>> in
> >> >>>>> the maven repository. Workaround is to build it locally and keep
> in
> >> >>> local
> >> >>>>> maven repository.
> >> >>>>> Need to check whether any future versions of
> 'protoc-gen-grpc-java'
> >> >> is
> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
> >> >>>>>
> >> >>>>>
> >> >>>>> Once the compilation issues are solved, then there might be many
> >> >> native
> >> >>>>> code related issues due to different architectures.
> >> >>>>> So to explore everything, need to join hands together and proceed.
> >> >>>>>
> >> >>>>>
> >> >>>>> Let us discuss and check, whether any body else out there who also
> >> >> need
> >> >>>> the
> >> >>>>> support of Hadoop on ARM architectures and ready to lend their
> hands
> >> >>> and
> >> >>>>> time in this work.
> >> >>>>>
> >> >>>>>
> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
> >> >>>>> [2]
> >> >>
> >>
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
> >> >>>>>
> >> >>>>> -Vinay
> >> >>
> >>
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

["张铎 (Duo Zhang)" <pa...@gmail.com>]

Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to generate
the protobuf code. It will download the protoc binary from the maven
central so we do not need to install protoc on the build machine any more.

Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：

> BTW, I also noticed that the Hadoop-trunk-Commit job has been failling for
> over 2 month related to the Protobuf problem .
> According to the latest successful build log:
> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
> the
> os version was ubuntu 14.04 and for the jobs that are failling now such
> as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
> the os version is 18.04. I'm not very familiar with the version changing
> for the jobs but I did a little search, according to:
>
> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
> &
>
> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
> it both said that the version of libprotc-dev and protobuf-compiler
> available for ubuntu 18.04 is 3.0.0
>
>
> On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com> wrote:
>
>> Thanx Vinay for the initiative, Makes sense to add support for different
>> architectures.
>>
>> +1, for the branch idea.
>> Good Luck!!!
>>
>> -Ayush
>>
>> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
>> wrote:
>> >
>> > For HBase, we purged all the protobuf related things from the public
>> API,
>> > and then upgraded to a shaded and relocated version of protobuf. We have
>> > created a repo for this:
>> >
>> > https://github.com/apache/hbase-thirdparty
>> >
>> > But since the hadoop dependencies still pull in the protobuf 2.5 jars,
>> our
>> > coprocessors are still on protobuf 2.5. Recently we have opened a
>> discuss
>> > on how to deal with the upgrading of coprocessor. Glad to see that the
>> > hadoop community is also willing to solve the problem.
>> >
>> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
>> >
>> >> +1, for the branch idea. Just FYI, Your biggest problem is proving that
>> >> Hadoop and the downstream projects work correctly after you upgrade
>> core
>> >> components like Protobuf.
>> >> So while branching and working on a branch is easy, merging back after
>> you
>> >> upgrade some of these core components is insanely hard. You might want
>> to
>> >> make sure that community buys into upgrading these components in the
>> trunk.
>> >> That way we will get testing and downstream components will notice when
>> >> things break.
>> >>
>> >> That said, I have lobbied for the upgrade of Protobuf for a really long
>> >> time; I have argued that 2.5 is out of support and we cannot stay on
>> that
>> >> branch forever; or we need to take ownership of the Protobuf 2.5 code
>> base.
>> >> It has been rightly pointed to me that while all the arguments I make
>> is
>> >> correct; it is a very complicated task to upgrade Protobuf, and the
>> worst
>> >> part is we will not even know what breaks until downstream projects
>> pick up
>> >> these changes and work against us.
>> >>
>> >> If we work off the Hadoop version 3 — and assume that we have
>> "shading" in
>> >> place for all deployments; it might be possible to get there; still a
>> >> daunting task.
>> >>
>> >> So best of luck with the branch approach — But please remember, Merging
>> >> back will be hard, Just my 2 cents.
>> >>
>> >> — Anu
>> >>
>> >>
>> >>
>> >>
>> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zhengzhenyulixi@gmail.com
>> >
>> >> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
>> separate
>> >>> branch with it's own ARM CI seems a really good idea.
>> >>> By doing this we won't break any of the undergoing development in
>> trunk
>> >> and
>> >>> a CI can be a very good way to show what are the
>> >>> current problems and what have been fixed, it will also provide a very
>> >> good
>> >>> view for contributors that are intrested to working on
>> >>> this. We can finally merge back the branch to trunk until the
>> community
>> >>> thinks it is good enough and stable enough. We can donate
>> >>> ARM machines to the existing CI system for the job.
>> >>>
>> >>> I wonder if this approch possible?
>> >>>
>> >>> BR,
>> >>>
>> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com>
>> >>> wrote:
>> >>>
>> >>>> Hi,
>> >>>>
>> >>>> Thanks Vinay for bring this up, I am a member of "Openlab" community
>> >>>> mentioned by Vinay. I am working on building and
>> >>>> testing Hadoop components on aarch64 server these days, besides the
>> >>> missing
>> >>>> dependices of ARM platform issues #1 #2 #3
>> >>>> mentioned by Vinay, other similar issue has also be found, such as
>> the
>> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >>>>
>> >>>> To promote the ARM support for Hadoop, we have discussed and hoped to
>> >> add
>> >>>> an ARM specific CI to Hadoop repo. we are not
>> >>>> sure about if there is any potential effect or confilict on the trunk
>> >>>> branch, so maybe creating a ARM specific branch for doing these stuff
>> >>>> is a better choice, what do you think?
>> >>>>
>> >>>> Hope to hear thoughts from you :)
>> >>>>
>> >>>> BR,
>> >>>> Liu sheng
>> >>>>
>> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
>> >>>>
>> >>>>> Hi Folks,
>> >>>>>
>> >>>>> ARM is becoming famous lately in its processing capability and has
>> >> got
>> >>>> the
>> >>>>> potential to run Bigdata workloads.
>> >>>>> Many users have been moving to ARM machines due to its low cost.
>> >>>>>
>> >>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry
>> >> PI)
>> >>>> for
>> >>>>> experimental purposes. Today ARM architecture is taking some of the
>> >>>>> serverside processing as well. So there will be/is a real need of
>> >>> Hadoop
>> >>>> to
>> >>>>> support ARM architecture as well.
>> >>>>>
>> >>>>> There are bunch of users who are trying out building Hadoop on ARM,
>> >>>> trying
>> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>> >>>>>
>> >>>>> As of today, Hadoop does not compile on ARM due to below issues,
>> >> found
>> >>>> from
>> >>>>> testing done in openlab in [2].
>> >>>>>
>> >>>>> 1. Protobuf :
>> >>>>> -------------------
>> >>>>>     Hadoop project (also some downstream projects) stuck to protobuf
>> >>>> 2.5.0
>> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is
>> not
>> >>>> being
>> >>>>> maintained in the community. While protobuf 3.x is being actively
>> >>> adopted
>> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2
>> >>>> messages.
>> >>>>> Due to some compilation issues in the generated java code, which can
>> >>>> induce
>> >>>>> problems in downstream. Due to this reason protobuf upgrade from
>> >> 2.5.0
>> >>>> was
>> >>>>> not taken up.
>> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
>> >>> classpath
>> >>>>> problem in downstream projects.
>> >>>>>    There are patches available to fix compilation in Hadoop. But
>> >> need
>> >>> to
>> >>>>> find a way to upgrade protobuf to latest version and still maintain
>> >> the
>> >>>>> downstream's classpath using shading feature of Hadoop build.
>> >>>>>
>> >>>>>     There is a Jira for protobuf upgrade[3] created even before
>> >> shade
>> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>> >> continue
>> >>>>> explore possibilities.
>> >>>>>
>> >>>>> 2. leveldbjni:
>> >>>>> ---------------
>> >>>>>    Current leveldbjni used in YARN doesnot support ARM architecture,
>> >>>> need
>> >>>>> to check whether any of the future versions support ARM and can
>> >> hadoop
>> >>>>> upgrade to that version.
>> >>>>>
>> >>>>>
>> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>> >>>>> -------------------------
>> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
>> >>> default
>> >>>> in
>> >>>>> the maven repository. Workaround is to build it locally and keep in
>> >>> local
>> >>>>> maven repository.
>> >>>>> Need to check whether any future versions of 'protoc-gen-grpc-java'
>> >> is
>> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
>> >>>>>
>> >>>>>
>> >>>>> Once the compilation issues are solved, then there might be many
>> >> native
>> >>>>> code related issues due to different architectures.
>> >>>>> So to explore everything, need to join hands together and proceed.
>> >>>>>
>> >>>>>
>> >>>>> Let us discuss and check, whether any body else out there who also
>> >> need
>> >>>> the
>> >>>>> support of Hadoop on ARM architectures and ready to lend their hands
>> >>> and
>> >>>>> time in this work.
>> >>>>>
>> >>>>>
>> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>> >>>>> [2]
>> >>
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>> >>>>>
>> >>>>> -Vinay
>> >>
>>
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

["张铎 (Duo Zhang)" <pa...@gmail.com>]

Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to generate
the protobuf code. It will download the protoc binary from the maven
central so we do not need to install protoc on the build machine any more.

Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：

> BTW, I also noticed that the Hadoop-trunk-Commit job has been failling for
> over 2 month related to the Protobuf problem .
> According to the latest successful build log:
> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
> the
> os version was ubuntu 14.04 and for the jobs that are failling now such
> as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
> the os version is 18.04. I'm not very familiar with the version changing
> for the jobs but I did a little search, according to:
>
> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
> &
>
> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
> it both said that the version of libprotc-dev and protobuf-compiler
> available for ubuntu 18.04 is 3.0.0
>
>
> On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com> wrote:
>
>> Thanx Vinay for the initiative, Makes sense to add support for different
>> architectures.
>>
>> +1, for the branch idea.
>> Good Luck!!!
>>
>> -Ayush
>>
>> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
>> wrote:
>> >
>> > For HBase, we purged all the protobuf related things from the public
>> API,
>> > and then upgraded to a shaded and relocated version of protobuf. We have
>> > created a repo for this:
>> >
>> > https://github.com/apache/hbase-thirdparty
>> >
>> > But since the hadoop dependencies still pull in the protobuf 2.5 jars,
>> our
>> > coprocessors are still on protobuf 2.5. Recently we have opened a
>> discuss
>> > on how to deal with the upgrading of coprocessor. Glad to see that the
>> > hadoop community is also willing to solve the problem.
>> >
>> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
>> >
>> >> +1, for the branch idea. Just FYI, Your biggest problem is proving that
>> >> Hadoop and the downstream projects work correctly after you upgrade
>> core
>> >> components like Protobuf.
>> >> So while branching and working on a branch is easy, merging back after
>> you
>> >> upgrade some of these core components is insanely hard. You might want
>> to
>> >> make sure that community buys into upgrading these components in the
>> trunk.
>> >> That way we will get testing and downstream components will notice when
>> >> things break.
>> >>
>> >> That said, I have lobbied for the upgrade of Protobuf for a really long
>> >> time; I have argued that 2.5 is out of support and we cannot stay on
>> that
>> >> branch forever; or we need to take ownership of the Protobuf 2.5 code
>> base.
>> >> It has been rightly pointed to me that while all the arguments I make
>> is
>> >> correct; it is a very complicated task to upgrade Protobuf, and the
>> worst
>> >> part is we will not even know what breaks until downstream projects
>> pick up
>> >> these changes and work against us.
>> >>
>> >> If we work off the Hadoop version 3 — and assume that we have
>> "shading" in
>> >> place for all deployments; it might be possible to get there; still a
>> >> daunting task.
>> >>
>> >> So best of luck with the branch approach — But please remember, Merging
>> >> back will be hard, Just my 2 cents.
>> >>
>> >> — Anu
>> >>
>> >>
>> >>
>> >>
>> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zhengzhenyulixi@gmail.com
>> >
>> >> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
>> separate
>> >>> branch with it's own ARM CI seems a really good idea.
>> >>> By doing this we won't break any of the undergoing development in
>> trunk
>> >> and
>> >>> a CI can be a very good way to show what are the
>> >>> current problems and what have been fixed, it will also provide a very
>> >> good
>> >>> view for contributors that are intrested to working on
>> >>> this. We can finally merge back the branch to trunk until the
>> community
>> >>> thinks it is good enough and stable enough. We can donate
>> >>> ARM machines to the existing CI system for the job.
>> >>>
>> >>> I wonder if this approch possible?
>> >>>
>> >>> BR,
>> >>>
>> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com>
>> >>> wrote:
>> >>>
>> >>>> Hi,
>> >>>>
>> >>>> Thanks Vinay for bring this up, I am a member of "Openlab" community
>> >>>> mentioned by Vinay. I am working on building and
>> >>>> testing Hadoop components on aarch64 server these days, besides the
>> >>> missing
>> >>>> dependices of ARM platform issues #1 #2 #3
>> >>>> mentioned by Vinay, other similar issue has also be found, such as
>> the
>> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >>>>
>> >>>> To promote the ARM support for Hadoop, we have discussed and hoped to
>> >> add
>> >>>> an ARM specific CI to Hadoop repo. we are not
>> >>>> sure about if there is any potential effect or confilict on the trunk
>> >>>> branch, so maybe creating a ARM specific branch for doing these stuff
>> >>>> is a better choice, what do you think?
>> >>>>
>> >>>> Hope to hear thoughts from you :)
>> >>>>
>> >>>> BR,
>> >>>> Liu sheng
>> >>>>
>> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
>> >>>>
>> >>>>> Hi Folks,
>> >>>>>
>> >>>>> ARM is becoming famous lately in its processing capability and has
>> >> got
>> >>>> the
>> >>>>> potential to run Bigdata workloads.
>> >>>>> Many users have been moving to ARM machines due to its low cost.
>> >>>>>
>> >>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry
>> >> PI)
>> >>>> for
>> >>>>> experimental purposes. Today ARM architecture is taking some of the
>> >>>>> serverside processing as well. So there will be/is a real need of
>> >>> Hadoop
>> >>>> to
>> >>>>> support ARM architecture as well.
>> >>>>>
>> >>>>> There are bunch of users who are trying out building Hadoop on ARM,
>> >>>> trying
>> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>> >>>>>
>> >>>>> As of today, Hadoop does not compile on ARM due to below issues,
>> >> found
>> >>>> from
>> >>>>> testing done in openlab in [2].
>> >>>>>
>> >>>>> 1. Protobuf :
>> >>>>> -------------------
>> >>>>>     Hadoop project (also some downstream projects) stuck to protobuf
>> >>>> 2.5.0
>> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is
>> not
>> >>>> being
>> >>>>> maintained in the community. While protobuf 3.x is being actively
>> >>> adopted
>> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2
>> >>>> messages.
>> >>>>> Due to some compilation issues in the generated java code, which can
>> >>>> induce
>> >>>>> problems in downstream. Due to this reason protobuf upgrade from
>> >> 2.5.0
>> >>>> was
>> >>>>> not taken up.
>> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
>> >>> classpath
>> >>>>> problem in downstream projects.
>> >>>>>    There are patches available to fix compilation in Hadoop. But
>> >> need
>> >>> to
>> >>>>> find a way to upgrade protobuf to latest version and still maintain
>> >> the
>> >>>>> downstream's classpath using shading feature of Hadoop build.
>> >>>>>
>> >>>>>     There is a Jira for protobuf upgrade[3] created even before
>> >> shade
>> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>> >> continue
>> >>>>> explore possibilities.
>> >>>>>
>> >>>>> 2. leveldbjni:
>> >>>>> ---------------
>> >>>>>    Current leveldbjni used in YARN doesnot support ARM architecture,
>> >>>> need
>> >>>>> to check whether any of the future versions support ARM and can
>> >> hadoop
>> >>>>> upgrade to that version.
>> >>>>>
>> >>>>>
>> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>> >>>>> -------------------------
>> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
>> >>> default
>> >>>> in
>> >>>>> the maven repository. Workaround is to build it locally and keep in
>> >>> local
>> >>>>> maven repository.
>> >>>>> Need to check whether any future versions of 'protoc-gen-grpc-java'
>> >> is
>> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
>> >>>>>
>> >>>>>
>> >>>>> Once the compilation issues are solved, then there might be many
>> >> native
>> >>>>> code related issues due to different architectures.
>> >>>>> So to explore everything, need to join hands together and proceed.
>> >>>>>
>> >>>>>
>> >>>>> Let us discuss and check, whether any body else out there who also
>> >> need
>> >>>> the
>> >>>>> support of Hadoop on ARM architectures and ready to lend their hands
>> >>> and
>> >>>>> time in this work.
>> >>>>>
>> >>>>>
>> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>> >>>>> [2]
>> >>
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>> >>>>>
>> >>>>> -Vinay
>> >>
>>
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

["张铎 (Duo Zhang)" <pa...@gmail.com>]

Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to generate
the protobuf code. It will download the protoc binary from the maven
central so we do not need to install protoc on the build machine any more.

Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：

> BTW, I also noticed that the Hadoop-trunk-Commit job has been failling for
> over 2 month related to the Protobuf problem .
> According to the latest successful build log:
> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
> the
> os version was ubuntu 14.04 and for the jobs that are failling now such
> as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
> the os version is 18.04. I'm not very familiar with the version changing
> for the jobs but I did a little search, according to:
>
> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
> &
>
> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
> it both said that the version of libprotc-dev and protobuf-compiler
> available for ubuntu 18.04 is 3.0.0
>
>
> On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com> wrote:
>
>> Thanx Vinay for the initiative, Makes sense to add support for different
>> architectures.
>>
>> +1, for the branch idea.
>> Good Luck!!!
>>
>> -Ayush
>>
>> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
>> wrote:
>> >
>> > For HBase, we purged all the protobuf related things from the public
>> API,
>> > and then upgraded to a shaded and relocated version of protobuf. We have
>> > created a repo for this:
>> >
>> > https://github.com/apache/hbase-thirdparty
>> >
>> > But since the hadoop dependencies still pull in the protobuf 2.5 jars,
>> our
>> > coprocessors are still on protobuf 2.5. Recently we have opened a
>> discuss
>> > on how to deal with the upgrading of coprocessor. Glad to see that the
>> > hadoop community is also willing to solve the problem.
>> >
>> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
>> >
>> >> +1, for the branch idea. Just FYI, Your biggest problem is proving that
>> >> Hadoop and the downstream projects work correctly after you upgrade
>> core
>> >> components like Protobuf.
>> >> So while branching and working on a branch is easy, merging back after
>> you
>> >> upgrade some of these core components is insanely hard. You might want
>> to
>> >> make sure that community buys into upgrading these components in the
>> trunk.
>> >> That way we will get testing and downstream components will notice when
>> >> things break.
>> >>
>> >> That said, I have lobbied for the upgrade of Protobuf for a really long
>> >> time; I have argued that 2.5 is out of support and we cannot stay on
>> that
>> >> branch forever; or we need to take ownership of the Protobuf 2.5 code
>> base.
>> >> It has been rightly pointed to me that while all the arguments I make
>> is
>> >> correct; it is a very complicated task to upgrade Protobuf, and the
>> worst
>> >> part is we will not even know what breaks until downstream projects
>> pick up
>> >> these changes and work against us.
>> >>
>> >> If we work off the Hadoop version 3 — and assume that we have
>> "shading" in
>> >> place for all deployments; it might be possible to get there; still a
>> >> daunting task.
>> >>
>> >> So best of luck with the branch approach — But please remember, Merging
>> >> back will be hard, Just my 2 cents.
>> >>
>> >> — Anu
>> >>
>> >>
>> >>
>> >>
>> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zhengzhenyulixi@gmail.com
>> >
>> >> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
>> separate
>> >>> branch with it's own ARM CI seems a really good idea.
>> >>> By doing this we won't break any of the undergoing development in
>> trunk
>> >> and
>> >>> a CI can be a very good way to show what are the
>> >>> current problems and what have been fixed, it will also provide a very
>> >> good
>> >>> view for contributors that are intrested to working on
>> >>> this. We can finally merge back the branch to trunk until the
>> community
>> >>> thinks it is good enough and stable enough. We can donate
>> >>> ARM machines to the existing CI system for the job.
>> >>>
>> >>> I wonder if this approch possible?
>> >>>
>> >>> BR,
>> >>>
>> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com>
>> >>> wrote:
>> >>>
>> >>>> Hi,
>> >>>>
>> >>>> Thanks Vinay for bring this up, I am a member of "Openlab" community
>> >>>> mentioned by Vinay. I am working on building and
>> >>>> testing Hadoop components on aarch64 server these days, besides the
>> >>> missing
>> >>>> dependices of ARM platform issues #1 #2 #3
>> >>>> mentioned by Vinay, other similar issue has also be found, such as
>> the
>> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >>>>
>> >>>> To promote the ARM support for Hadoop, we have discussed and hoped to
>> >> add
>> >>>> an ARM specific CI to Hadoop repo. we are not
>> >>>> sure about if there is any potential effect or confilict on the trunk
>> >>>> branch, so maybe creating a ARM specific branch for doing these stuff
>> >>>> is a better choice, what do you think?
>> >>>>
>> >>>> Hope to hear thoughts from you :)
>> >>>>
>> >>>> BR,
>> >>>> Liu sheng
>> >>>>
>> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
>> >>>>
>> >>>>> Hi Folks,
>> >>>>>
>> >>>>> ARM is becoming famous lately in its processing capability and has
>> >> got
>> >>>> the
>> >>>>> potential to run Bigdata workloads.
>> >>>>> Many users have been moving to ARM machines due to its low cost.
>> >>>>>
>> >>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry
>> >> PI)
>> >>>> for
>> >>>>> experimental purposes. Today ARM architecture is taking some of the
>> >>>>> serverside processing as well. So there will be/is a real need of
>> >>> Hadoop
>> >>>> to
>> >>>>> support ARM architecture as well.
>> >>>>>
>> >>>>> There are bunch of users who are trying out building Hadoop on ARM,
>> >>>> trying
>> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>> >>>>>
>> >>>>> As of today, Hadoop does not compile on ARM due to below issues,
>> >> found
>> >>>> from
>> >>>>> testing done in openlab in [2].
>> >>>>>
>> >>>>> 1. Protobuf :
>> >>>>> -------------------
>> >>>>>     Hadoop project (also some downstream projects) stuck to protobuf
>> >>>> 2.5.0
>> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is
>> not
>> >>>> being
>> >>>>> maintained in the community. While protobuf 3.x is being actively
>> >>> adopted
>> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2
>> >>>> messages.
>> >>>>> Due to some compilation issues in the generated java code, which can
>> >>>> induce
>> >>>>> problems in downstream. Due to this reason protobuf upgrade from
>> >> 2.5.0
>> >>>> was
>> >>>>> not taken up.
>> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
>> >>> classpath
>> >>>>> problem in downstream projects.
>> >>>>>    There are patches available to fix compilation in Hadoop. But
>> >> need
>> >>> to
>> >>>>> find a way to upgrade protobuf to latest version and still maintain
>> >> the
>> >>>>> downstream's classpath using shading feature of Hadoop build.
>> >>>>>
>> >>>>>     There is a Jira for protobuf upgrade[3] created even before
>> >> shade
>> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>> >> continue
>> >>>>> explore possibilities.
>> >>>>>
>> >>>>> 2. leveldbjni:
>> >>>>> ---------------
>> >>>>>    Current leveldbjni used in YARN doesnot support ARM architecture,
>> >>>> need
>> >>>>> to check whether any of the future versions support ARM and can
>> >> hadoop
>> >>>>> upgrade to that version.
>> >>>>>
>> >>>>>
>> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>> >>>>> -------------------------
>> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
>> >>> default
>> >>>> in
>> >>>>> the maven repository. Workaround is to build it locally and keep in
>> >>> local
>> >>>>> maven repository.
>> >>>>> Need to check whether any future versions of 'protoc-gen-grpc-java'
>> >> is
>> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
>> >>>>>
>> >>>>>
>> >>>>> Once the compilation issues are solved, then there might be many
>> >> native
>> >>>>> code related issues due to different architectures.
>> >>>>> So to explore everything, need to join hands together and proceed.
>> >>>>>
>> >>>>>
>> >>>>> Let us discuss and check, whether any body else out there who also
>> >> need
>> >>>> the
>> >>>>> support of Hadoop on ARM architectures and ready to lend their hands
>> >>> and
>> >>>>> time in this work.
>> >>>>>
>> >>>>>
>> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>> >>>>> [2]
>> >>
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>> >>>>>
>> >>>>> -Vinay
>> >>
>>
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

["张铎 (Duo Zhang)" <pa...@gmail.com>]

Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to generate
the protobuf code. It will download the protoc binary from the maven
central so we do not need to install protoc on the build machine any more.

Zhenyu Zheng <zh...@gmail.com> 于2019年9月4日周三 下午5:27写道：

> BTW, I also noticed that the Hadoop-trunk-Commit job has been failling for
> over 2 month related to the Protobuf problem .
> According to the latest successful build log:
> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
> the
> os version was ubuntu 14.04 and for the jobs that are failling now such
> as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
> the os version is 18.04. I'm not very familiar with the version changing
> for the jobs but I did a little search, according to:
>
> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
> &
>
> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
> it both said that the version of libprotc-dev and protobuf-compiler
> available for ubuntu 18.04 is 3.0.0
>
>
> On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com> wrote:
>
>> Thanx Vinay for the initiative, Makes sense to add support for different
>> architectures.
>>
>> +1, for the branch idea.
>> Good Luck!!!
>>
>> -Ayush
>>
>> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com>
>> wrote:
>> >
>> > For HBase, we purged all the protobuf related things from the public
>> API,
>> > and then upgraded to a shaded and relocated version of protobuf. We have
>> > created a repo for this:
>> >
>> > https://github.com/apache/hbase-thirdparty
>> >
>> > But since the hadoop dependencies still pull in the protobuf 2.5 jars,
>> our
>> > coprocessors are still on protobuf 2.5. Recently we have opened a
>> discuss
>> > on how to deal with the upgrading of coprocessor. Glad to see that the
>> > hadoop community is also willing to solve the problem.
>> >
>> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
>> >
>> >> +1, for the branch idea. Just FYI, Your biggest problem is proving that
>> >> Hadoop and the downstream projects work correctly after you upgrade
>> core
>> >> components like Protobuf.
>> >> So while branching and working on a branch is easy, merging back after
>> you
>> >> upgrade some of these core components is insanely hard. You might want
>> to
>> >> make sure that community buys into upgrading these components in the
>> trunk.
>> >> That way we will get testing and downstream components will notice when
>> >> things break.
>> >>
>> >> That said, I have lobbied for the upgrade of Protobuf for a really long
>> >> time; I have argued that 2.5 is out of support and we cannot stay on
>> that
>> >> branch forever; or we need to take ownership of the Protobuf 2.5 code
>> base.
>> >> It has been rightly pointed to me that while all the arguments I make
>> is
>> >> correct; it is a very complicated task to upgrade Protobuf, and the
>> worst
>> >> part is we will not even know what breaks until downstream projects
>> pick up
>> >> these changes and work against us.
>> >>
>> >> If we work off the Hadoop version 3 — and assume that we have
>> "shading" in
>> >> place for all deployments; it might be possible to get there; still a
>> >> daunting task.
>> >>
>> >> So best of luck with the branch approach — But please remember, Merging
>> >> back will be hard, Just my 2 cents.
>> >>
>> >> — Anu
>> >>
>> >>
>> >>
>> >>
>> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zhengzhenyulixi@gmail.com
>> >
>> >> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
>> separate
>> >>> branch with it's own ARM CI seems a really good idea.
>> >>> By doing this we won't break any of the undergoing development in
>> trunk
>> >> and
>> >>> a CI can be a very good way to show what are the
>> >>> current problems and what have been fixed, it will also provide a very
>> >> good
>> >>> view for contributors that are intrested to working on
>> >>> this. We can finally merge back the branch to trunk until the
>> community
>> >>> thinks it is good enough and stable enough. We can donate
>> >>> ARM machines to the existing CI system for the job.
>> >>>
>> >>> I wonder if this approch possible?
>> >>>
>> >>> BR,
>> >>>
>> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com>
>> >>> wrote:
>> >>>
>> >>>> Hi,
>> >>>>
>> >>>> Thanks Vinay for bring this up, I am a member of "Openlab" community
>> >>>> mentioned by Vinay. I am working on building and
>> >>>> testing Hadoop components on aarch64 server these days, besides the
>> >>> missing
>> >>>> dependices of ARM platform issues #1 #2 #3
>> >>>> mentioned by Vinay, other similar issue has also be found, such as
>> the
>> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >>>>
>> >>>> To promote the ARM support for Hadoop, we have discussed and hoped to
>> >> add
>> >>>> an ARM specific CI to Hadoop repo. we are not
>> >>>> sure about if there is any potential effect or confilict on the trunk
>> >>>> branch, so maybe creating a ARM specific branch for doing these stuff
>> >>>> is a better choice, what do you think?
>> >>>>
>> >>>> Hope to hear thoughts from you :)
>> >>>>
>> >>>> BR,
>> >>>> Liu sheng
>> >>>>
>> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
>> >>>>
>> >>>>> Hi Folks,
>> >>>>>
>> >>>>> ARM is becoming famous lately in its processing capability and has
>> >> got
>> >>>> the
>> >>>>> potential to run Bigdata workloads.
>> >>>>> Many users have been moving to ARM machines due to its low cost.
>> >>>>>
>> >>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry
>> >> PI)
>> >>>> for
>> >>>>> experimental purposes. Today ARM architecture is taking some of the
>> >>>>> serverside processing as well. So there will be/is a real need of
>> >>> Hadoop
>> >>>> to
>> >>>>> support ARM architecture as well.
>> >>>>>
>> >>>>> There are bunch of users who are trying out building Hadoop on ARM,
>> >>>> trying
>> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>> >>>>>
>> >>>>> As of today, Hadoop does not compile on ARM due to below issues,
>> >> found
>> >>>> from
>> >>>>> testing done in openlab in [2].
>> >>>>>
>> >>>>> 1. Protobuf :
>> >>>>> -------------------
>> >>>>>     Hadoop project (also some downstream projects) stuck to protobuf
>> >>>> 2.5.0
>> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is
>> not
>> >>>> being
>> >>>>> maintained in the community. While protobuf 3.x is being actively
>> >>> adopted
>> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2
>> >>>> messages.
>> >>>>> Due to some compilation issues in the generated java code, which can
>> >>>> induce
>> >>>>> problems in downstream. Due to this reason protobuf upgrade from
>> >> 2.5.0
>> >>>> was
>> >>>>> not taken up.
>> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
>> >>> classpath
>> >>>>> problem in downstream projects.
>> >>>>>    There are patches available to fix compilation in Hadoop. But
>> >> need
>> >>> to
>> >>>>> find a way to upgrade protobuf to latest version and still maintain
>> >> the
>> >>>>> downstream's classpath using shading feature of Hadoop build.
>> >>>>>
>> >>>>>     There is a Jira for protobuf upgrade[3] created even before
>> >> shade
>> >>>>> support was added to Hadoop. Now need to revisit the Jira and
>> >> continue
>> >>>>> explore possibilities.
>> >>>>>
>> >>>>> 2. leveldbjni:
>> >>>>> ---------------
>> >>>>>    Current leveldbjni used in YARN doesnot support ARM architecture,
>> >>>> need
>> >>>>> to check whether any of the future versions support ARM and can
>> >> hadoop
>> >>>>> upgrade to that version.
>> >>>>>
>> >>>>>
>> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>> >>>>> -------------------------
>> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
>> >>> default
>> >>>> in
>> >>>>> the maven repository. Workaround is to build it locally and keep in
>> >>> local
>> >>>>> maven repository.
>> >>>>> Need to check whether any future versions of 'protoc-gen-grpc-java'
>> >> is
>> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
>> >>>>>
>> >>>>>
>> >>>>> Once the compilation issues are solved, then there might be many
>> >> native
>> >>>>> code related issues due to different architectures.
>> >>>>> So to explore everything, need to join hands together and proceed.
>> >>>>>
>> >>>>>
>> >>>>> Let us discuss and check, whether any body else out there who also
>> >> need
>> >>>> the
>> >>>>> support of Hadoop on ARM architectures and ready to lend their hands
>> >>> and
>> >>>>> time in this work.
>> >>>>>
>> >>>>>
>> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>> >>>>> [2]
>> >>
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>> >>>>>
>> >>>>> -Vinay
>> >>
>>
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Zhenyu Zheng <zh...@gmail.com>]

BTW, I also noticed that the Hadoop-trunk-Commit job has been failling for
over 2 month related to the Protobuf problem .
According to the latest successful build log:
https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
the
os version was ubuntu 14.04 and for the jobs that are failling now such as:
https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
the os version is 18.04. I'm not very familiar with the version changing
for the jobs but I did a little search, according to:
https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
&
https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
it both said that the version of libprotc-dev and protobuf-compiler
available for ubuntu 18.04 is 3.0.0


On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com> wrote:

> Thanx Vinay for the initiative, Makes sense to add support for different
> architectures.
>
> +1, for the branch idea.
> Good Luck!!!
>
> -Ayush
>
> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> >
> > For HBase, we purged all the protobuf related things from the public API,
> > and then upgraded to a shaded and relocated version of protobuf. We have
> > created a repo for this:
> >
> > https://github.com/apache/hbase-thirdparty
> >
> > But since the hadoop dependencies still pull in the protobuf 2.5 jars,
> our
> > coprocessors are still on protobuf 2.5. Recently we have opened a discuss
> > on how to deal with the upgrading of coprocessor. Glad to see that the
> > hadoop community is also willing to solve the problem.
> >
> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
> >
> >> +1, for the branch idea. Just FYI, Your biggest problem is proving that
> >> Hadoop and the downstream projects work correctly after you upgrade core
> >> components like Protobuf.
> >> So while branching and working on a branch is easy, merging back after
> you
> >> upgrade some of these core components is insanely hard. You might want
> to
> >> make sure that community buys into upgrading these components in the
> trunk.
> >> That way we will get testing and downstream components will notice when
> >> things break.
> >>
> >> That said, I have lobbied for the upgrade of Protobuf for a really long
> >> time; I have argued that 2.5 is out of support and we cannot stay on
> that
> >> branch forever; or we need to take ownership of the Protobuf 2.5 code
> base.
> >> It has been rightly pointed to me that while all the arguments I make is
> >> correct; it is a very complicated task to upgrade Protobuf, and the
> worst
> >> part is we will not even know what breaks until downstream projects
> pick up
> >> these changes and work against us.
> >>
> >> If we work off the Hadoop version 3 — and assume that we have "shading"
> in
> >> place for all deployments; it might be possible to get there; still a
> >> daunting task.
> >>
> >> So best of luck with the branch approach — But please remember, Merging
> >> back will be hard, Just my 2 cents.
> >>
> >> — Anu
> >>
> >>
> >>
> >>
> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zh...@gmail.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
> separate
> >>> branch with it's own ARM CI seems a really good idea.
> >>> By doing this we won't break any of the undergoing development in trunk
> >> and
> >>> a CI can be a very good way to show what are the
> >>> current problems and what have been fixed, it will also provide a very
> >> good
> >>> view for contributors that are intrested to working on
> >>> this. We can finally merge back the branch to trunk until the community
> >>> thinks it is good enough and stable enough. We can donate
> >>> ARM machines to the existing CI system for the job.
> >>>
> >>> I wonder if this approch possible?
> >>>
> >>> BR,
> >>>
> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> Thanks Vinay for bring this up, I am a member of "Openlab" community
> >>>> mentioned by Vinay. I am working on building and
> >>>> testing Hadoop components on aarch64 server these days, besides the
> >>> missing
> >>>> dependices of ARM platform issues #1 #2 #3
> >>>> mentioned by Vinay, other similar issue has also be found, such as the
> >>>> "PhantomJS" dependent package also missing for aarch64.
> >>>>
> >>>> To promote the ARM support for Hadoop, we have discussed and hoped to
> >> add
> >>>> an ARM specific CI to Hadoop repo. we are not
> >>>> sure about if there is any potential effect or confilict on the trunk
> >>>> branch, so maybe creating a ARM specific branch for doing these stuff
> >>>> is a better choice, what do you think?
> >>>>
> >>>> Hope to hear thoughts from you :)
> >>>>
> >>>> BR,
> >>>> Liu sheng
> >>>>
> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> >>>>
> >>>>> Hi Folks,
> >>>>>
> >>>>> ARM is becoming famous lately in its processing capability and has
> >> got
> >>>> the
> >>>>> potential to run Bigdata workloads.
> >>>>> Many users have been moving to ARM machines due to its low cost.
> >>>>>
> >>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry
> >> PI)
> >>>> for
> >>>>> experimental purposes. Today ARM architecture is taking some of the
> >>>>> serverside processing as well. So there will be/is a real need of
> >>> Hadoop
> >>>> to
> >>>>> support ARM architecture as well.
> >>>>>
> >>>>> There are bunch of users who are trying out building Hadoop on ARM,
> >>>> trying
> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
> >>>>>
> >>>>> As of today, Hadoop does not compile on ARM due to below issues,
> >> found
> >>>> from
> >>>>> testing done in openlab in [2].
> >>>>>
> >>>>> 1. Protobuf :
> >>>>> -------------------
> >>>>>     Hadoop project (also some downstream projects) stuck to protobuf
> >>>> 2.5.0
> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is not
> >>>> being
> >>>>> maintained in the community. While protobuf 3.x is being actively
> >>> adopted
> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2
> >>>> messages.
> >>>>> Due to some compilation issues in the generated java code, which can
> >>>> induce
> >>>>> problems in downstream. Due to this reason protobuf upgrade from
> >> 2.5.0
> >>>> was
> >>>>> not taken up.
> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> >>> classpath
> >>>>> problem in downstream projects.
> >>>>>    There are patches available to fix compilation in Hadoop. But
> >> need
> >>> to
> >>>>> find a way to upgrade protobuf to latest version and still maintain
> >> the
> >>>>> downstream's classpath using shading feature of Hadoop build.
> >>>>>
> >>>>>     There is a Jira for protobuf upgrade[3] created even before
> >> shade
> >>>>> support was added to Hadoop. Now need to revisit the Jira and
> >> continue
> >>>>> explore possibilities.
> >>>>>
> >>>>> 2. leveldbjni:
> >>>>> ---------------
> >>>>>    Current leveldbjni used in YARN doesnot support ARM architecture,
> >>>> need
> >>>>> to check whether any of the future versions support ARM and can
> >> hadoop
> >>>>> upgrade to that version.
> >>>>>
> >>>>>
> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> >>>>> -------------------------
> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
> >>> default
> >>>> in
> >>>>> the maven repository. Workaround is to build it locally and keep in
> >>> local
> >>>>> maven repository.
> >>>>> Need to check whether any future versions of 'protoc-gen-grpc-java'
> >> is
> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
> >>>>>
> >>>>>
> >>>>> Once the compilation issues are solved, then there might be many
> >> native
> >>>>> code related issues due to different architectures.
> >>>>> So to explore everything, need to join hands together and proceed.
> >>>>>
> >>>>>
> >>>>> Let us discuss and check, whether any body else out there who also
> >> need
> >>>> the
> >>>>> support of Hadoop on ARM architectures and ready to lend their hands
> >>> and
> >>>>> time in this work.
> >>>>>
> >>>>>
> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
> >>>>> [2]
> >>
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
> >>>>>
> >>>>> -Vinay
> >>
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Zhenyu Zheng <zh...@gmail.com>]

BTW, I also noticed that the Hadoop-trunk-Commit job has been failling for
over 2 month related to the Protobuf problem .
According to the latest successful build log:
https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
the
os version was ubuntu 14.04 and for the jobs that are failling now such as:
https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
the os version is 18.04. I'm not very familiar with the version changing
for the jobs but I did a little search, according to:
https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
&
https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
it both said that the version of libprotc-dev and protobuf-compiler
available for ubuntu 18.04 is 3.0.0


On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ay...@gmail.com> wrote:

> Thanx Vinay for the initiative, Makes sense to add support for different
> architectures.
>
> +1, for the branch idea.
> Good Luck!!!
>
> -Ayush
>
> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> >
> > For HBase, we purged all the protobuf related things from the public API,
> > and then upgraded to a shaded and relocated version of protobuf. We have
> > created a repo for this:
> >
> > https://github.com/apache/hbase-thirdparty
> >
> > But since the hadoop dependencies still pull in the protobuf 2.5 jars,
> our
> > coprocessors are still on protobuf 2.5. Recently we have opened a discuss
> > on how to deal with the upgrading of coprocessor. Glad to see that the
> > hadoop community is also willing to solve the problem.
> >
> > Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
> >
> >> +1, for the branch idea. Just FYI, Your biggest problem is proving that
> >> Hadoop and the downstream projects work correctly after you upgrade core
> >> components like Protobuf.
> >> So while branching and working on a branch is easy, merging back after
> you
> >> upgrade some of these core components is insanely hard. You might want
> to
> >> make sure that community buys into upgrading these components in the
> trunk.
> >> That way we will get testing and downstream components will notice when
> >> things break.
> >>
> >> That said, I have lobbied for the upgrade of Protobuf for a really long
> >> time; I have argued that 2.5 is out of support and we cannot stay on
> that
> >> branch forever; or we need to take ownership of the Protobuf 2.5 code
> base.
> >> It has been rightly pointed to me that while all the arguments I make is
> >> correct; it is a very complicated task to upgrade Protobuf, and the
> worst
> >> part is we will not even know what breaks until downstream projects
> pick up
> >> these changes and work against us.
> >>
> >> If we work off the Hadoop version 3 — and assume that we have "shading"
> in
> >> place for all deployments; it might be possible to get there; still a
> >> daunting task.
> >>
> >> So best of luck with the branch approach — But please remember, Merging
> >> back will be hard, Just my 2 cents.
> >>
> >> — Anu
> >>
> >>
> >>
> >>
> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zh...@gmail.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
> separate
> >>> branch with it's own ARM CI seems a really good idea.
> >>> By doing this we won't break any of the undergoing development in trunk
> >> and
> >>> a CI can be a very good way to show what are the
> >>> current problems and what have been fixed, it will also provide a very
> >> good
> >>> view for contributors that are intrested to working on
> >>> this. We can finally merge back the branch to trunk until the community
> >>> thinks it is good enough and stable enough. We can donate
> >>> ARM machines to the existing CI system for the job.
> >>>
> >>> I wonder if this approch possible?
> >>>
> >>> BR,
> >>>
> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> Thanks Vinay for bring this up, I am a member of "Openlab" community
> >>>> mentioned by Vinay. I am working on building and
> >>>> testing Hadoop components on aarch64 server these days, besides the
> >>> missing
> >>>> dependices of ARM platform issues #1 #2 #3
> >>>> mentioned by Vinay, other similar issue has also be found, such as the
> >>>> "PhantomJS" dependent package also missing for aarch64.
> >>>>
> >>>> To promote the ARM support for Hadoop, we have discussed and hoped to
> >> add
> >>>> an ARM specific CI to Hadoop repo. we are not
> >>>> sure about if there is any potential effect or confilict on the trunk
> >>>> branch, so maybe creating a ARM specific branch for doing these stuff
> >>>> is a better choice, what do you think?
> >>>>
> >>>> Hope to hear thoughts from you :)
> >>>>
> >>>> BR,
> >>>> Liu sheng
> >>>>
> >>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> >>>>
> >>>>> Hi Folks,
> >>>>>
> >>>>> ARM is becoming famous lately in its processing capability and has
> >> got
> >>>> the
> >>>>> potential to run Bigdata workloads.
> >>>>> Many users have been moving to ARM machines due to its low cost.
> >>>>>
> >>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry
> >> PI)
> >>>> for
> >>>>> experimental purposes. Today ARM architecture is taking some of the
> >>>>> serverside processing as well. So there will be/is a real need of
> >>> Hadoop
> >>>> to
> >>>>> support ARM architecture as well.
> >>>>>
> >>>>> There are bunch of users who are trying out building Hadoop on ARM,
> >>>> trying
> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
> >>>>>
> >>>>> As of today, Hadoop does not compile on ARM due to below issues,
> >> found
> >>>> from
> >>>>> testing done in openlab in [2].
> >>>>>
> >>>>> 1. Protobuf :
> >>>>> -------------------
> >>>>>     Hadoop project (also some downstream projects) stuck to protobuf
> >>>> 2.5.0
> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is not
> >>>> being
> >>>>> maintained in the community. While protobuf 3.x is being actively
> >>> adopted
> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2
> >>>> messages.
> >>>>> Due to some compilation issues in the generated java code, which can
> >>>> induce
> >>>>> problems in downstream. Due to this reason protobuf upgrade from
> >> 2.5.0
> >>>> was
> >>>>> not taken up.
> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> >>> classpath
> >>>>> problem in downstream projects.
> >>>>>    There are patches available to fix compilation in Hadoop. But
> >> need
> >>> to
> >>>>> find a way to upgrade protobuf to latest version and still maintain
> >> the
> >>>>> downstream's classpath using shading feature of Hadoop build.
> >>>>>
> >>>>>     There is a Jira for protobuf upgrade[3] created even before
> >> shade
> >>>>> support was added to Hadoop. Now need to revisit the Jira and
> >> continue
> >>>>> explore possibilities.
> >>>>>
> >>>>> 2. leveldbjni:
> >>>>> ---------------
> >>>>>    Current leveldbjni used in YARN doesnot support ARM architecture,
> >>>> need
> >>>>> to check whether any of the future versions support ARM and can
> >> hadoop
> >>>>> upgrade to that version.
> >>>>>
> >>>>>
> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> >>>>> -------------------------
> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
> >>> default
> >>>> in
> >>>>> the maven repository. Workaround is to build it locally and keep in
> >>> local
> >>>>> maven repository.
> >>>>> Need to check whether any future versions of 'protoc-gen-grpc-java'
> >> is
> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
> >>>>>
> >>>>>
> >>>>> Once the compilation issues are solved, then there might be many
> >> native
> >>>>> code related issues due to different architectures.
> >>>>> So to explore everything, need to join hands together and proceed.
> >>>>>
> >>>>>
> >>>>> Let us discuss and check, whether any body else out there who also
> >> need
> >>>> the
> >>>>> support of Hadoop on ARM architectures and ready to lend their hands
> >>> and
> >>>>> time in this work.
> >>>>>
> >>>>>
> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
> >>>>> [2]
> >>
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
> >>>>>
> >>>>> -Vinay
> >>
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Ayush Saxena <ay...@gmail.com>]

Thanx Vinay for the initiative, Makes sense to add support for different architectures.

+1, for the branch idea.
Good Luck!!!

-Ayush

> On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> 
> For HBase, we purged all the protobuf related things from the public API,
> and then upgraded to a shaded and relocated version of protobuf. We have
> created a repo for this:
> 
> https://github.com/apache/hbase-thirdparty
> 
> But since the hadoop dependencies still pull in the protobuf 2.5 jars, our
> coprocessors are still on protobuf 2.5. Recently we have opened a discuss
> on how to deal with the upgrading of coprocessor. Glad to see that the
> hadoop community is also willing to solve the problem.
> 
> Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
> 
>> +1, for the branch idea. Just FYI, Your biggest problem is proving that
>> Hadoop and the downstream projects work correctly after you upgrade core
>> components like Protobuf.
>> So while branching and working on a branch is easy, merging back after you
>> upgrade some of these core components is insanely hard. You might want to
>> make sure that community buys into upgrading these components in the trunk.
>> That way we will get testing and downstream components will notice when
>> things break.
>> 
>> That said, I have lobbied for the upgrade of Protobuf for a really long
>> time; I have argued that 2.5 is out of support and we cannot stay on that
>> branch forever; or we need to take ownership of the Protobuf 2.5 code base.
>> It has been rightly pointed to me that while all the arguments I make is
>> correct; it is a very complicated task to upgrade Protobuf, and the worst
>> part is we will not even know what breaks until downstream projects pick up
>> these changes and work against us.
>> 
>> If we work off the Hadoop version 3 — and assume that we have "shading" in
>> place for all deployments; it might be possible to get there; still a
>> daunting task.
>> 
>> So best of luck with the branch approach — But please remember, Merging
>> back will be hard, Just my 2 cents.
>> 
>> — Anu
>> 
>> 
>> 
>> 
>> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zh...@gmail.com>
>> wrote:
>> 
>>> Hi,
>>> 
>>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate
>>> branch with it's own ARM CI seems a really good idea.
>>> By doing this we won't break any of the undergoing development in trunk
>> and
>>> a CI can be a very good way to show what are the
>>> current problems and what have been fixed, it will also provide a very
>> good
>>> view for contributors that are intrested to working on
>>> this. We can finally merge back the branch to trunk until the community
>>> thinks it is good enough and stable enough. We can donate
>>> ARM machines to the existing CI system for the job.
>>> 
>>> I wonder if this approch possible?
>>> 
>>> BR,
>>> 
>>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com>
>>> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> Thanks Vinay for bring this up, I am a member of "Openlab" community
>>>> mentioned by Vinay. I am working on building and
>>>> testing Hadoop components on aarch64 server these days, besides the
>>> missing
>>>> dependices of ARM platform issues #1 #2 #3
>>>> mentioned by Vinay, other similar issue has also be found, such as the
>>>> "PhantomJS" dependent package also missing for aarch64.
>>>> 
>>>> To promote the ARM support for Hadoop, we have discussed and hoped to
>> add
>>>> an ARM specific CI to Hadoop repo. we are not
>>>> sure about if there is any potential effect or confilict on the trunk
>>>> branch, so maybe creating a ARM specific branch for doing these stuff
>>>> is a better choice, what do you think?
>>>> 
>>>> Hope to hear thoughts from you :)
>>>> 
>>>> BR,
>>>> Liu sheng
>>>> 
>>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
>>>> 
>>>>> Hi Folks,
>>>>> 
>>>>> ARM is becoming famous lately in its processing capability and has
>> got
>>>> the
>>>>> potential to run Bigdata workloads.
>>>>> Many users have been moving to ARM machines due to its low cost.
>>>>> 
>>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry
>> PI)
>>>> for
>>>>> experimental purposes. Today ARM architecture is taking some of the
>>>>> serverside processing as well. So there will be/is a real need of
>>> Hadoop
>>>> to
>>>>> support ARM architecture as well.
>>>>> 
>>>>> There are bunch of users who are trying out building Hadoop on ARM,
>>>> trying
>>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>>>>> 
>>>>> As of today, Hadoop does not compile on ARM due to below issues,
>> found
>>>> from
>>>>> testing done in openlab in [2].
>>>>> 
>>>>> 1. Protobuf :
>>>>> -------------------
>>>>>     Hadoop project (also some downstream projects) stuck to protobuf
>>>> 2.5.0
>>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is not
>>>> being
>>>>> maintained in the community. While protobuf 3.x is being actively
>>> adopted
>>>>> widely, still protobuf 3.x provides wire compatibility for proto2
>>>> messages.
>>>>> Due to some compilation issues in the generated java code, which can
>>>> induce
>>>>> problems in downstream. Due to this reason protobuf upgrade from
>> 2.5.0
>>>> was
>>>>> not taken up.
>>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
>>> classpath
>>>>> problem in downstream projects.
>>>>>    There are patches available to fix compilation in Hadoop. But
>> need
>>> to
>>>>> find a way to upgrade protobuf to latest version and still maintain
>> the
>>>>> downstream's classpath using shading feature of Hadoop build.
>>>>> 
>>>>>     There is a Jira for protobuf upgrade[3] created even before
>> shade
>>>>> support was added to Hadoop. Now need to revisit the Jira and
>> continue
>>>>> explore possibilities.
>>>>> 
>>>>> 2. leveldbjni:
>>>>> ---------------
>>>>>    Current leveldbjni used in YARN doesnot support ARM architecture,
>>>> need
>>>>> to check whether any of the future versions support ARM and can
>> hadoop
>>>>> upgrade to that version.
>>>>> 
>>>>> 
>>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>>>>> -------------------------
>>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
>>> default
>>>> in
>>>>> the maven repository. Workaround is to build it locally and keep in
>>> local
>>>>> maven repository.
>>>>> Need to check whether any future versions of 'protoc-gen-grpc-java'
>> is
>>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
>>>>> 
>>>>> 
>>>>> Once the compilation issues are solved, then there might be many
>> native
>>>>> code related issues due to different architectures.
>>>>> So to explore everything, need to join hands together and proceed.
>>>>> 
>>>>> 
>>>>> Let us discuss and check, whether any body else out there who also
>> need
>>>> the
>>>>> support of Hadoop on ARM architectures and ready to lend their hands
>>> and
>>>>> time in this work.
>>>>> 
>>>>> 
>>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>>>>> [2]
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>>>>> 
>>>>> -Vinay
>> 

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Ayush Saxena <ay...@gmail.com>]

Thanx Vinay for the initiative, Makes sense to add support for different architectures.

+1, for the branch idea.
Good Luck!!!

-Ayush

> On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> 
> For HBase, we purged all the protobuf related things from the public API,
> and then upgraded to a shaded and relocated version of protobuf. We have
> created a repo for this:
> 
> https://github.com/apache/hbase-thirdparty
> 
> But since the hadoop dependencies still pull in the protobuf 2.5 jars, our
> coprocessors are still on protobuf 2.5. Recently we have opened a discuss
> on how to deal with the upgrading of coprocessor. Glad to see that the
> hadoop community is also willing to solve the problem.
> 
> Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
> 
>> +1, for the branch idea. Just FYI, Your biggest problem is proving that
>> Hadoop and the downstream projects work correctly after you upgrade core
>> components like Protobuf.
>> So while branching and working on a branch is easy, merging back after you
>> upgrade some of these core components is insanely hard. You might want to
>> make sure that community buys into upgrading these components in the trunk.
>> That way we will get testing and downstream components will notice when
>> things break.
>> 
>> That said, I have lobbied for the upgrade of Protobuf for a really long
>> time; I have argued that 2.5 is out of support and we cannot stay on that
>> branch forever; or we need to take ownership of the Protobuf 2.5 code base.
>> It has been rightly pointed to me that while all the arguments I make is
>> correct; it is a very complicated task to upgrade Protobuf, and the worst
>> part is we will not even know what breaks until downstream projects pick up
>> these changes and work against us.
>> 
>> If we work off the Hadoop version 3 — and assume that we have "shading" in
>> place for all deployments; it might be possible to get there; still a
>> daunting task.
>> 
>> So best of luck with the branch approach — But please remember, Merging
>> back will be hard, Just my 2 cents.
>> 
>> — Anu
>> 
>> 
>> 
>> 
>> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zh...@gmail.com>
>> wrote:
>> 
>>> Hi,
>>> 
>>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate
>>> branch with it's own ARM CI seems a really good idea.
>>> By doing this we won't break any of the undergoing development in trunk
>> and
>>> a CI can be a very good way to show what are the
>>> current problems and what have been fixed, it will also provide a very
>> good
>>> view for contributors that are intrested to working on
>>> this. We can finally merge back the branch to trunk until the community
>>> thinks it is good enough and stable enough. We can donate
>>> ARM machines to the existing CI system for the job.
>>> 
>>> I wonder if this approch possible?
>>> 
>>> BR,
>>> 
>>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com>
>>> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> Thanks Vinay for bring this up, I am a member of "Openlab" community
>>>> mentioned by Vinay. I am working on building and
>>>> testing Hadoop components on aarch64 server these days, besides the
>>> missing
>>>> dependices of ARM platform issues #1 #2 #3
>>>> mentioned by Vinay, other similar issue has also be found, such as the
>>>> "PhantomJS" dependent package also missing for aarch64.
>>>> 
>>>> To promote the ARM support for Hadoop, we have discussed and hoped to
>> add
>>>> an ARM specific CI to Hadoop repo. we are not
>>>> sure about if there is any potential effect or confilict on the trunk
>>>> branch, so maybe creating a ARM specific branch for doing these stuff
>>>> is a better choice, what do you think?
>>>> 
>>>> Hope to hear thoughts from you :)
>>>> 
>>>> BR,
>>>> Liu sheng
>>>> 
>>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
>>>> 
>>>>> Hi Folks,
>>>>> 
>>>>> ARM is becoming famous lately in its processing capability and has
>> got
>>>> the
>>>>> potential to run Bigdata workloads.
>>>>> Many users have been moving to ARM machines due to its low cost.
>>>>> 
>>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry
>> PI)
>>>> for
>>>>> experimental purposes. Today ARM architecture is taking some of the
>>>>> serverside processing as well. So there will be/is a real need of
>>> Hadoop
>>>> to
>>>>> support ARM architecture as well.
>>>>> 
>>>>> There are bunch of users who are trying out building Hadoop on ARM,
>>>> trying
>>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>>>>> 
>>>>> As of today, Hadoop does not compile on ARM due to below issues,
>> found
>>>> from
>>>>> testing done in openlab in [2].
>>>>> 
>>>>> 1. Protobuf :
>>>>> -------------------
>>>>>     Hadoop project (also some downstream projects) stuck to protobuf
>>>> 2.5.0
>>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is not
>>>> being
>>>>> maintained in the community. While protobuf 3.x is being actively
>>> adopted
>>>>> widely, still protobuf 3.x provides wire compatibility for proto2
>>>> messages.
>>>>> Due to some compilation issues in the generated java code, which can
>>>> induce
>>>>> problems in downstream. Due to this reason protobuf upgrade from
>> 2.5.0
>>>> was
>>>>> not taken up.
>>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
>>> classpath
>>>>> problem in downstream projects.
>>>>>    There are patches available to fix compilation in Hadoop. But
>> need
>>> to
>>>>> find a way to upgrade protobuf to latest version and still maintain
>> the
>>>>> downstream's classpath using shading feature of Hadoop build.
>>>>> 
>>>>>     There is a Jira for protobuf upgrade[3] created even before
>> shade
>>>>> support was added to Hadoop. Now need to revisit the Jira and
>> continue
>>>>> explore possibilities.
>>>>> 
>>>>> 2. leveldbjni:
>>>>> ---------------
>>>>>    Current leveldbjni used in YARN doesnot support ARM architecture,
>>>> need
>>>>> to check whether any of the future versions support ARM and can
>> hadoop
>>>>> upgrade to that version.
>>>>> 
>>>>> 
>>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>>>>> -------------------------
>>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
>>> default
>>>> in
>>>>> the maven repository. Workaround is to build it locally and keep in
>>> local
>>>>> maven repository.
>>>>> Need to check whether any future versions of 'protoc-gen-grpc-java'
>> is
>>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
>>>>> 
>>>>> 
>>>>> Once the compilation issues are solved, then there might be many
>> native
>>>>> code related issues due to different architectures.
>>>>> So to explore everything, need to join hands together and proceed.
>>>>> 
>>>>> 
>>>>> Let us discuss and check, whether any body else out there who also
>> need
>>>> the
>>>>> support of Hadoop on ARM architectures and ready to lend their hands
>>> and
>>>>> time in this work.
>>>>> 
>>>>> 
>>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>>>>> [2]
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>>>>> 
>>>>> -Vinay
>> 

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Ayush Saxena <ay...@gmail.com>]

Thanx Vinay for the initiative, Makes sense to add support for different architectures.

+1, for the branch idea.
Good Luck!!!

-Ayush

> On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> 
> For HBase, we purged all the protobuf related things from the public API,
> and then upgraded to a shaded and relocated version of protobuf. We have
> created a repo for this:
> 
> https://github.com/apache/hbase-thirdparty
> 
> But since the hadoop dependencies still pull in the protobuf 2.5 jars, our
> coprocessors are still on protobuf 2.5. Recently we have opened a discuss
> on how to deal with the upgrading of coprocessor. Glad to see that the
> hadoop community is also willing to solve the problem.
> 
> Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
> 
>> +1, for the branch idea. Just FYI, Your biggest problem is proving that
>> Hadoop and the downstream projects work correctly after you upgrade core
>> components like Protobuf.
>> So while branching and working on a branch is easy, merging back after you
>> upgrade some of these core components is insanely hard. You might want to
>> make sure that community buys into upgrading these components in the trunk.
>> That way we will get testing and downstream components will notice when
>> things break.
>> 
>> That said, I have lobbied for the upgrade of Protobuf for a really long
>> time; I have argued that 2.5 is out of support and we cannot stay on that
>> branch forever; or we need to take ownership of the Protobuf 2.5 code base.
>> It has been rightly pointed to me that while all the arguments I make is
>> correct; it is a very complicated task to upgrade Protobuf, and the worst
>> part is we will not even know what breaks until downstream projects pick up
>> these changes and work against us.
>> 
>> If we work off the Hadoop version 3 — and assume that we have "shading" in
>> place for all deployments; it might be possible to get there; still a
>> daunting task.
>> 
>> So best of luck with the branch approach — But please remember, Merging
>> back will be hard, Just my 2 cents.
>> 
>> — Anu
>> 
>> 
>> 
>> 
>> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zh...@gmail.com>
>> wrote:
>> 
>>> Hi,
>>> 
>>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate
>>> branch with it's own ARM CI seems a really good idea.
>>> By doing this we won't break any of the undergoing development in trunk
>> and
>>> a CI can be a very good way to show what are the
>>> current problems and what have been fixed, it will also provide a very
>> good
>>> view for contributors that are intrested to working on
>>> this. We can finally merge back the branch to trunk until the community
>>> thinks it is good enough and stable enough. We can donate
>>> ARM machines to the existing CI system for the job.
>>> 
>>> I wonder if this approch possible?
>>> 
>>> BR,
>>> 
>>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com>
>>> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> Thanks Vinay for bring this up, I am a member of "Openlab" community
>>>> mentioned by Vinay. I am working on building and
>>>> testing Hadoop components on aarch64 server these days, besides the
>>> missing
>>>> dependices of ARM platform issues #1 #2 #3
>>>> mentioned by Vinay, other similar issue has also be found, such as the
>>>> "PhantomJS" dependent package also missing for aarch64.
>>>> 
>>>> To promote the ARM support for Hadoop, we have discussed and hoped to
>> add
>>>> an ARM specific CI to Hadoop repo. we are not
>>>> sure about if there is any potential effect or confilict on the trunk
>>>> branch, so maybe creating a ARM specific branch for doing these stuff
>>>> is a better choice, what do you think?
>>>> 
>>>> Hope to hear thoughts from you :)
>>>> 
>>>> BR,
>>>> Liu sheng
>>>> 
>>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
>>>> 
>>>>> Hi Folks,
>>>>> 
>>>>> ARM is becoming famous lately in its processing capability and has
>> got
>>>> the
>>>>> potential to run Bigdata workloads.
>>>>> Many users have been moving to ARM machines due to its low cost.
>>>>> 
>>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry
>> PI)
>>>> for
>>>>> experimental purposes. Today ARM architecture is taking some of the
>>>>> serverside processing as well. So there will be/is a real need of
>>> Hadoop
>>>> to
>>>>> support ARM architecture as well.
>>>>> 
>>>>> There are bunch of users who are trying out building Hadoop on ARM,
>>>> trying
>>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>>>>> 
>>>>> As of today, Hadoop does not compile on ARM due to below issues,
>> found
>>>> from
>>>>> testing done in openlab in [2].
>>>>> 
>>>>> 1. Protobuf :
>>>>> -------------------
>>>>>     Hadoop project (also some downstream projects) stuck to protobuf
>>>> 2.5.0
>>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is not
>>>> being
>>>>> maintained in the community. While protobuf 3.x is being actively
>>> adopted
>>>>> widely, still protobuf 3.x provides wire compatibility for proto2
>>>> messages.
>>>>> Due to some compilation issues in the generated java code, which can
>>>> induce
>>>>> problems in downstream. Due to this reason protobuf upgrade from
>> 2.5.0
>>>> was
>>>>> not taken up.
>>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
>>> classpath
>>>>> problem in downstream projects.
>>>>>    There are patches available to fix compilation in Hadoop. But
>> need
>>> to
>>>>> find a way to upgrade protobuf to latest version and still maintain
>> the
>>>>> downstream's classpath using shading feature of Hadoop build.
>>>>> 
>>>>>     There is a Jira for protobuf upgrade[3] created even before
>> shade
>>>>> support was added to Hadoop. Now need to revisit the Jira and
>> continue
>>>>> explore possibilities.
>>>>> 
>>>>> 2. leveldbjni:
>>>>> ---------------
>>>>>    Current leveldbjni used in YARN doesnot support ARM architecture,
>>>> need
>>>>> to check whether any of the future versions support ARM and can
>> hadoop
>>>>> upgrade to that version.
>>>>> 
>>>>> 
>>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>>>>> -------------------------
>>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
>>> default
>>>> in
>>>>> the maven repository. Workaround is to build it locally and keep in
>>> local
>>>>> maven repository.
>>>>> Need to check whether any future versions of 'protoc-gen-grpc-java'
>> is
>>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
>>>>> 
>>>>> 
>>>>> Once the compilation issues are solved, then there might be many
>> native
>>>>> code related issues due to different architectures.
>>>>> So to explore everything, need to join hands together and proceed.
>>>>> 
>>>>> 
>>>>> Let us discuss and check, whether any body else out there who also
>> need
>>>> the
>>>>> support of Hadoop on ARM architectures and ready to lend their hands
>>> and
>>>>> time in this work.
>>>>> 
>>>>> 
>>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>>>>> [2]
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>>>>> 
>>>>> -Vinay
>> 

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Ayush Saxena <ay...@gmail.com>]

Thanx Vinay for the initiative, Makes sense to add support for different architectures.

+1, for the branch idea.
Good Luck!!!

-Ayush

> On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> 
> For HBase, we purged all the protobuf related things from the public API,
> and then upgraded to a shaded and relocated version of protobuf. We have
> created a repo for this:
> 
> https://github.com/apache/hbase-thirdparty
> 
> But since the hadoop dependencies still pull in the protobuf 2.5 jars, our
> coprocessors are still on protobuf 2.5. Recently we have opened a discuss
> on how to deal with the upgrading of coprocessor. Glad to see that the
> hadoop community is also willing to solve the problem.
> 
> Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：
> 
>> +1, for the branch idea. Just FYI, Your biggest problem is proving that
>> Hadoop and the downstream projects work correctly after you upgrade core
>> components like Protobuf.
>> So while branching and working on a branch is easy, merging back after you
>> upgrade some of these core components is insanely hard. You might want to
>> make sure that community buys into upgrading these components in the trunk.
>> That way we will get testing and downstream components will notice when
>> things break.
>> 
>> That said, I have lobbied for the upgrade of Protobuf for a really long
>> time; I have argued that 2.5 is out of support and we cannot stay on that
>> branch forever; or we need to take ownership of the Protobuf 2.5 code base.
>> It has been rightly pointed to me that while all the arguments I make is
>> correct; it is a very complicated task to upgrade Protobuf, and the worst
>> part is we will not even know what breaks until downstream projects pick up
>> these changes and work against us.
>> 
>> If we work off the Hadoop version 3 — and assume that we have "shading" in
>> place for all deployments; it might be possible to get there; still a
>> daunting task.
>> 
>> So best of luck with the branch approach — But please remember, Merging
>> back will be hard, Just my 2 cents.
>> 
>> — Anu
>> 
>> 
>> 
>> 
>> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zh...@gmail.com>
>> wrote:
>> 
>>> Hi,
>>> 
>>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate
>>> branch with it's own ARM CI seems a really good idea.
>>> By doing this we won't break any of the undergoing development in trunk
>> and
>>> a CI can be a very good way to show what are the
>>> current problems and what have been fixed, it will also provide a very
>> good
>>> view for contributors that are intrested to working on
>>> this. We can finally merge back the branch to trunk until the community
>>> thinks it is good enough and stable enough. We can donate
>>> ARM machines to the existing CI system for the job.
>>> 
>>> I wonder if this approch possible?
>>> 
>>> BR,
>>> 
>>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com>
>>> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> Thanks Vinay for bring this up, I am a member of "Openlab" community
>>>> mentioned by Vinay. I am working on building and
>>>> testing Hadoop components on aarch64 server these days, besides the
>>> missing
>>>> dependices of ARM platform issues #1 #2 #3
>>>> mentioned by Vinay, other similar issue has also be found, such as the
>>>> "PhantomJS" dependent package also missing for aarch64.
>>>> 
>>>> To promote the ARM support for Hadoop, we have discussed and hoped to
>> add
>>>> an ARM specific CI to Hadoop repo. we are not
>>>> sure about if there is any potential effect or confilict on the trunk
>>>> branch, so maybe creating a ARM specific branch for doing these stuff
>>>> is a better choice, what do you think?
>>>> 
>>>> Hope to hear thoughts from you :)
>>>> 
>>>> BR,
>>>> Liu sheng
>>>> 
>>>> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
>>>> 
>>>>> Hi Folks,
>>>>> 
>>>>> ARM is becoming famous lately in its processing capability and has
>> got
>>>> the
>>>>> potential to run Bigdata workloads.
>>>>> Many users have been moving to ARM machines due to its low cost.
>>>>> 
>>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry
>> PI)
>>>> for
>>>>> experimental purposes. Today ARM architecture is taking some of the
>>>>> serverside processing as well. So there will be/is a real need of
>>> Hadoop
>>>> to
>>>>> support ARM architecture as well.
>>>>> 
>>>>> There are bunch of users who are trying out building Hadoop on ARM,
>>>> trying
>>>>> to add ARM CI to hadoop and facing issues[1]. Also some
>>>>> 
>>>>> As of today, Hadoop does not compile on ARM due to below issues,
>> found
>>>> from
>>>>> testing done in openlab in [2].
>>>>> 
>>>>> 1. Protobuf :
>>>>> -------------------
>>>>>     Hadoop project (also some downstream projects) stuck to protobuf
>>>> 2.5.0
>>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is not
>>>> being
>>>>> maintained in the community. While protobuf 3.x is being actively
>>> adopted
>>>>> widely, still protobuf 3.x provides wire compatibility for proto2
>>>> messages.
>>>>> Due to some compilation issues in the generated java code, which can
>>>> induce
>>>>> problems in downstream. Due to this reason protobuf upgrade from
>> 2.5.0
>>>> was
>>>>> not taken up.
>>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid
>>> classpath
>>>>> problem in downstream projects.
>>>>>    There are patches available to fix compilation in Hadoop. But
>> need
>>> to
>>>>> find a way to upgrade protobuf to latest version and still maintain
>> the
>>>>> downstream's classpath using shading feature of Hadoop build.
>>>>> 
>>>>>     There is a Jira for protobuf upgrade[3] created even before
>> shade
>>>>> support was added to Hadoop. Now need to revisit the Jira and
>> continue
>>>>> explore possibilities.
>>>>> 
>>>>> 2. leveldbjni:
>>>>> ---------------
>>>>>    Current leveldbjni used in YARN doesnot support ARM architecture,
>>>> need
>>>>> to check whether any of the future versions support ARM and can
>> hadoop
>>>>> upgrade to that version.
>>>>> 
>>>>> 
>>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>>>>> -------------------------
>>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
>>> default
>>>> in
>>>>> the maven repository. Workaround is to build it locally and keep in
>>> local
>>>>> maven repository.
>>>>> Need to check whether any future versions of 'protoc-gen-grpc-java'
>> is
>>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it?
>>>>> 
>>>>> 
>>>>> Once the compilation issues are solved, then there might be many
>> native
>>>>> code related issues due to different architectures.
>>>>> So to explore everything, need to join hands together and proceed.
>>>>> 
>>>>> 
>>>>> Let us discuss and check, whether any body else out there who also
>> need
>>>> the
>>>>> support of Hadoop on ARM architectures and ready to lend their hands
>>> and
>>>>> time in this work.
>>>>> 
>>>>> 
>>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>>>>> [2]
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>>>>> 
>>>>> -Vinay
>> 

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: [DISCUSS] ARM/aarch64 support for Hadoop

["张铎 (Duo Zhang)" <pa...@gmail.com>]

For HBase, we purged all the protobuf related things from the public API,
and then upgraded to a shaded and relocated version of protobuf. We have
created a repo for this:

https://github.com/apache/hbase-thirdparty

But since the hadoop dependencies still pull in the protobuf 2.5 jars, our
coprocessors are still on protobuf 2.5. Recently we have opened a discuss
on how to deal with the upgrading of coprocessor. Glad to see that the
hadoop community is also willing to solve the problem.

Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：

> +1, for the branch idea. Just FYI, Your biggest problem is proving that
> Hadoop and the downstream projects work correctly after you upgrade core
> components like Protobuf.
> So while branching and working on a branch is easy, merging back after you
> upgrade some of these core components is insanely hard. You might want to
> make sure that community buys into upgrading these components in the trunk.
> That way we will get testing and downstream components will notice when
> things break.
>
> That said, I have lobbied for the upgrade of Protobuf for a really long
> time; I have argued that 2.5 is out of support and we cannot stay on that
> branch forever; or we need to take ownership of the Protobuf 2.5 code base.
> It has been rightly pointed to me that while all the arguments I make is
> correct; it is a very complicated task to upgrade Protobuf, and the worst
> part is we will not even know what breaks until downstream projects pick up
> these changes and work against us.
>
> If we work off the Hadoop version 3 — and assume that we have "shading" in
> place for all deployments; it might be possible to get there; still a
> daunting task.
>
> So best of luck with the branch approach — But please remember, Merging
> back will be hard, Just my 2 cents.
>
> — Anu
>
>
>
>
> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zh...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate
> > branch with it's own ARM CI seems a really good idea.
> > By doing this we won't break any of the undergoing development in trunk
> and
> > a CI can be a very good way to show what are the
> > current problems and what have been fixed, it will also provide a very
> good
> > view for contributors that are intrested to working on
> > this. We can finally merge back the branch to trunk until the community
> > thinks it is good enough and stable enough. We can donate
> > ARM machines to the existing CI system for the job.
> >
> > I wonder if this approch possible?
> >
> > BR,
> >
> > On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > Thanks Vinay for bring this up, I am a member of "Openlab" community
> > > mentioned by Vinay. I am working on building and
> > > testing Hadoop components on aarch64 server these days, besides the
> > missing
> > > dependices of ARM platform issues #1 #2 #3
> > > mentioned by Vinay, other similar issue has also be found, such as the
> > > "PhantomJS" dependent package also missing for aarch64.
> > >
> > > To promote the ARM support for Hadoop, we have discussed and hoped to
> add
> > > an ARM specific CI to Hadoop repo. we are not
> > > sure about if there is any potential effect or confilict on the trunk
> > > branch, so maybe creating a ARM specific branch for doing these stuff
> > > is a better choice, what do you think?
> > >
> > > Hope to hear thoughts from you :)
> > >
> > > BR,
> > > Liu sheng
> > >
> > > Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> > >
> > > > Hi Folks,
> > > >
> > > > ARM is becoming famous lately in its processing capability and has
> got
> > > the
> > > > potential to run Bigdata workloads.
> > > > Many users have been moving to ARM machines due to its low cost.
> > > >
> > > > In the past there were attempts to compile Hadoop on ARM (Rasberry
> PI)
> > > for
> > > > experimental purposes. Today ARM architecture is taking some of the
> > > > serverside processing as well. So there will be/is a real need of
> > Hadoop
> > > to
> > > > support ARM architecture as well.
> > > >
> > > > There are bunch of users who are trying out building Hadoop on ARM,
> > > trying
> > > > to add ARM CI to hadoop and facing issues[1]. Also some
> > > >
> > > > As of today, Hadoop does not compile on ARM due to below issues,
> found
> > > from
> > > > testing done in openlab in [2].
> > > >
> > > > 1. Protobuf :
> > > > -------------------
> > > >      Hadoop project (also some downstream projects) stuck to protobuf
> > > 2.5.0
> > > > version, due to backward compatibility reasons. Protobuf-2.5.0 is not
> > > being
> > > > maintained in the community. While protobuf 3.x is being actively
> > adopted
> > > > widely, still protobuf 3.x provides wire compatibility for proto2
> > > messages.
> > > > Due to some compilation issues in the generated java code, which can
> > > induce
> > > > problems in downstream. Due to this reason protobuf upgrade from
> 2.5.0
> > > was
> > > > not taken up.
> > > > In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> > classpath
> > > > problem in downstream projects.
> > > >     There are patches available to fix compilation in Hadoop. But
> need
> > to
> > > > find a way to upgrade protobuf to latest version and still maintain
> the
> > > > downstream's classpath using shading feature of Hadoop build.
> > > >
> > > >      There is a Jira for protobuf upgrade[3] created even before
> shade
> > > > support was added to Hadoop. Now need to revisit the Jira and
> continue
> > > > explore possibilities.
> > > >
> > > > 2. leveldbjni:
> > > > ---------------
> > > >     Current leveldbjni used in YARN doesnot support ARM architecture,
> > > need
> > > > to check whether any of the future versions support ARM and can
> hadoop
> > > > upgrade to that version.
> > > >
> > > >
> > > > 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> > > > -------------------------
> > > > 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
> > default
> > > in
> > > > the maven repository. Workaround is to build it locally and keep in
> > local
> > > > maven repository.
> > > > Need to check whether any future versions of 'protoc-gen-grpc-java'
> is
> > > > having ARM executable and whether hadoop-yarn-csi can upgrade it?
> > > >
> > > >
> > > > Once the compilation issues are solved, then there might be many
> native
> > > > code related issues due to different architectures.
> > > > So to explore everything, need to join hands together and proceed.
> > > >
> > > >
> > > > Let us discuss and check, whether any body else out there who also
> need
> > > the
> > > > support of Hadoop on ARM architectures and ready to lend their hands
> > and
> > > > time in this work.
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/HADOOP-16358
> > > > [2]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> > > > [3] https://issues.apache.org/jira/browse/HADOOP-13363
> > > >
> > > > -Vinay
> > > >
> > >
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

["张铎 (Duo Zhang)" <pa...@gmail.com>]

For HBase, we purged all the protobuf related things from the public API,
and then upgraded to a shaded and relocated version of protobuf. We have
created a repo for this:

https://github.com/apache/hbase-thirdparty

But since the hadoop dependencies still pull in the protobuf 2.5 jars, our
coprocessors are still on protobuf 2.5. Recently we have opened a discuss
on how to deal with the upgrading of coprocessor. Glad to see that the
hadoop community is also willing to solve the problem.

Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：

> +1, for the branch idea. Just FYI, Your biggest problem is proving that
> Hadoop and the downstream projects work correctly after you upgrade core
> components like Protobuf.
> So while branching and working on a branch is easy, merging back after you
> upgrade some of these core components is insanely hard. You might want to
> make sure that community buys into upgrading these components in the trunk.
> That way we will get testing and downstream components will notice when
> things break.
>
> That said, I have lobbied for the upgrade of Protobuf for a really long
> time; I have argued that 2.5 is out of support and we cannot stay on that
> branch forever; or we need to take ownership of the Protobuf 2.5 code base.
> It has been rightly pointed to me that while all the arguments I make is
> correct; it is a very complicated task to upgrade Protobuf, and the worst
> part is we will not even know what breaks until downstream projects pick up
> these changes and work against us.
>
> If we work off the Hadoop version 3 — and assume that we have "shading" in
> place for all deployments; it might be possible to get there; still a
> daunting task.
>
> So best of luck with the branch approach — But please remember, Merging
> back will be hard, Just my 2 cents.
>
> — Anu
>
>
>
>
> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zh...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate
> > branch with it's own ARM CI seems a really good idea.
> > By doing this we won't break any of the undergoing development in trunk
> and
> > a CI can be a very good way to show what are the
> > current problems and what have been fixed, it will also provide a very
> good
> > view for contributors that are intrested to working on
> > this. We can finally merge back the branch to trunk until the community
> > thinks it is good enough and stable enough. We can donate
> > ARM machines to the existing CI system for the job.
> >
> > I wonder if this approch possible?
> >
> > BR,
> >
> > On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > Thanks Vinay for bring this up, I am a member of "Openlab" community
> > > mentioned by Vinay. I am working on building and
> > > testing Hadoop components on aarch64 server these days, besides the
> > missing
> > > dependices of ARM platform issues #1 #2 #3
> > > mentioned by Vinay, other similar issue has also be found, such as the
> > > "PhantomJS" dependent package also missing for aarch64.
> > >
> > > To promote the ARM support for Hadoop, we have discussed and hoped to
> add
> > > an ARM specific CI to Hadoop repo. we are not
> > > sure about if there is any potential effect or confilict on the trunk
> > > branch, so maybe creating a ARM specific branch for doing these stuff
> > > is a better choice, what do you think?
> > >
> > > Hope to hear thoughts from you :)
> > >
> > > BR,
> > > Liu sheng
> > >
> > > Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> > >
> > > > Hi Folks,
> > > >
> > > > ARM is becoming famous lately in its processing capability and has
> got
> > > the
> > > > potential to run Bigdata workloads.
> > > > Many users have been moving to ARM machines due to its low cost.
> > > >
> > > > In the past there were attempts to compile Hadoop on ARM (Rasberry
> PI)
> > > for
> > > > experimental purposes. Today ARM architecture is taking some of the
> > > > serverside processing as well. So there will be/is a real need of
> > Hadoop
> > > to
> > > > support ARM architecture as well.
> > > >
> > > > There are bunch of users who are trying out building Hadoop on ARM,
> > > trying
> > > > to add ARM CI to hadoop and facing issues[1]. Also some
> > > >
> > > > As of today, Hadoop does not compile on ARM due to below issues,
> found
> > > from
> > > > testing done in openlab in [2].
> > > >
> > > > 1. Protobuf :
> > > > -------------------
> > > >      Hadoop project (also some downstream projects) stuck to protobuf
> > > 2.5.0
> > > > version, due to backward compatibility reasons. Protobuf-2.5.0 is not
> > > being
> > > > maintained in the community. While protobuf 3.x is being actively
> > adopted
> > > > widely, still protobuf 3.x provides wire compatibility for proto2
> > > messages.
> > > > Due to some compilation issues in the generated java code, which can
> > > induce
> > > > problems in downstream. Due to this reason protobuf upgrade from
> 2.5.0
> > > was
> > > > not taken up.
> > > > In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> > classpath
> > > > problem in downstream projects.
> > > >     There are patches available to fix compilation in Hadoop. But
> need
> > to
> > > > find a way to upgrade protobuf to latest version and still maintain
> the
> > > > downstream's classpath using shading feature of Hadoop build.
> > > >
> > > >      There is a Jira for protobuf upgrade[3] created even before
> shade
> > > > support was added to Hadoop. Now need to revisit the Jira and
> continue
> > > > explore possibilities.
> > > >
> > > > 2. leveldbjni:
> > > > ---------------
> > > >     Current leveldbjni used in YARN doesnot support ARM architecture,
> > > need
> > > > to check whether any of the future versions support ARM and can
> hadoop
> > > > upgrade to that version.
> > > >
> > > >
> > > > 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> > > > -------------------------
> > > > 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
> > default
> > > in
> > > > the maven repository. Workaround is to build it locally and keep in
> > local
> > > > maven repository.
> > > > Need to check whether any future versions of 'protoc-gen-grpc-java'
> is
> > > > having ARM executable and whether hadoop-yarn-csi can upgrade it?
> > > >
> > > >
> > > > Once the compilation issues are solved, then there might be many
> native
> > > > code related issues due to different architectures.
> > > > So to explore everything, need to join hands together and proceed.
> > > >
> > > >
> > > > Let us discuss and check, whether any body else out there who also
> need
> > > the
> > > > support of Hadoop on ARM architectures and ready to lend their hands
> > and
> > > > time in this work.
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/HADOOP-16358
> > > > [2]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> > > > [3] https://issues.apache.org/jira/browse/HADOOP-13363
> > > >
> > > > -Vinay
> > > >
> > >
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

["张铎 (Duo Zhang)" <pa...@gmail.com>]

For HBase, we purged all the protobuf related things from the public API,
and then upgraded to a shaded and relocated version of protobuf. We have
created a repo for this:

https://github.com/apache/hbase-thirdparty

But since the hadoop dependencies still pull in the protobuf 2.5 jars, our
coprocessors are still on protobuf 2.5. Recently we have opened a discuss
on how to deal with the upgrading of coprocessor. Glad to see that the
hadoop community is also willing to solve the problem.

Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：

> +1, for the branch idea. Just FYI, Your biggest problem is proving that
> Hadoop and the downstream projects work correctly after you upgrade core
> components like Protobuf.
> So while branching and working on a branch is easy, merging back after you
> upgrade some of these core components is insanely hard. You might want to
> make sure that community buys into upgrading these components in the trunk.
> That way we will get testing and downstream components will notice when
> things break.
>
> That said, I have lobbied for the upgrade of Protobuf for a really long
> time; I have argued that 2.5 is out of support and we cannot stay on that
> branch forever; or we need to take ownership of the Protobuf 2.5 code base.
> It has been rightly pointed to me that while all the arguments I make is
> correct; it is a very complicated task to upgrade Protobuf, and the worst
> part is we will not even know what breaks until downstream projects pick up
> these changes and work against us.
>
> If we work off the Hadoop version 3 — and assume that we have "shading" in
> place for all deployments; it might be possible to get there; still a
> daunting task.
>
> So best of luck with the branch approach — But please remember, Merging
> back will be hard, Just my 2 cents.
>
> — Anu
>
>
>
>
> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zh...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate
> > branch with it's own ARM CI seems a really good idea.
> > By doing this we won't break any of the undergoing development in trunk
> and
> > a CI can be a very good way to show what are the
> > current problems and what have been fixed, it will also provide a very
> good
> > view for contributors that are intrested to working on
> > this. We can finally merge back the branch to trunk until the community
> > thinks it is good enough and stable enough. We can donate
> > ARM machines to the existing CI system for the job.
> >
> > I wonder if this approch possible?
> >
> > BR,
> >
> > On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > Thanks Vinay for bring this up, I am a member of "Openlab" community
> > > mentioned by Vinay. I am working on building and
> > > testing Hadoop components on aarch64 server these days, besides the
> > missing
> > > dependices of ARM platform issues #1 #2 #3
> > > mentioned by Vinay, other similar issue has also be found, such as the
> > > "PhantomJS" dependent package also missing for aarch64.
> > >
> > > To promote the ARM support for Hadoop, we have discussed and hoped to
> add
> > > an ARM specific CI to Hadoop repo. we are not
> > > sure about if there is any potential effect or confilict on the trunk
> > > branch, so maybe creating a ARM specific branch for doing these stuff
> > > is a better choice, what do you think?
> > >
> > > Hope to hear thoughts from you :)
> > >
> > > BR,
> > > Liu sheng
> > >
> > > Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> > >
> > > > Hi Folks,
> > > >
> > > > ARM is becoming famous lately in its processing capability and has
> got
> > > the
> > > > potential to run Bigdata workloads.
> > > > Many users have been moving to ARM machines due to its low cost.
> > > >
> > > > In the past there were attempts to compile Hadoop on ARM (Rasberry
> PI)
> > > for
> > > > experimental purposes. Today ARM architecture is taking some of the
> > > > serverside processing as well. So there will be/is a real need of
> > Hadoop
> > > to
> > > > support ARM architecture as well.
> > > >
> > > > There are bunch of users who are trying out building Hadoop on ARM,
> > > trying
> > > > to add ARM CI to hadoop and facing issues[1]. Also some
> > > >
> > > > As of today, Hadoop does not compile on ARM due to below issues,
> found
> > > from
> > > > testing done in openlab in [2].
> > > >
> > > > 1. Protobuf :
> > > > -------------------
> > > >      Hadoop project (also some downstream projects) stuck to protobuf
> > > 2.5.0
> > > > version, due to backward compatibility reasons. Protobuf-2.5.0 is not
> > > being
> > > > maintained in the community. While protobuf 3.x is being actively
> > adopted
> > > > widely, still protobuf 3.x provides wire compatibility for proto2
> > > messages.
> > > > Due to some compilation issues in the generated java code, which can
> > > induce
> > > > problems in downstream. Due to this reason protobuf upgrade from
> 2.5.0
> > > was
> > > > not taken up.
> > > > In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> > classpath
> > > > problem in downstream projects.
> > > >     There are patches available to fix compilation in Hadoop. But
> need
> > to
> > > > find a way to upgrade protobuf to latest version and still maintain
> the
> > > > downstream's classpath using shading feature of Hadoop build.
> > > >
> > > >      There is a Jira for protobuf upgrade[3] created even before
> shade
> > > > support was added to Hadoop. Now need to revisit the Jira and
> continue
> > > > explore possibilities.
> > > >
> > > > 2. leveldbjni:
> > > > ---------------
> > > >     Current leveldbjni used in YARN doesnot support ARM architecture,
> > > need
> > > > to check whether any of the future versions support ARM and can
> hadoop
> > > > upgrade to that version.
> > > >
> > > >
> > > > 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> > > > -------------------------
> > > > 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
> > default
> > > in
> > > > the maven repository. Workaround is to build it locally and keep in
> > local
> > > > maven repository.
> > > > Need to check whether any future versions of 'protoc-gen-grpc-java'
> is
> > > > having ARM executable and whether hadoop-yarn-csi can upgrade it?
> > > >
> > > >
> > > > Once the compilation issues are solved, then there might be many
> native
> > > > code related issues due to different architectures.
> > > > So to explore everything, need to join hands together and proceed.
> > > >
> > > >
> > > > Let us discuss and check, whether any body else out there who also
> need
> > > the
> > > > support of Hadoop on ARM architectures and ready to lend their hands
> > and
> > > > time in this work.
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/HADOOP-16358
> > > > [2]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> > > > [3] https://issues.apache.org/jira/browse/HADOOP-13363
> > > >
> > > > -Vinay
> > > >
> > >
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

["张铎 (Duo Zhang)" <pa...@gmail.com>]

For HBase, we purged all the protobuf related things from the public API,
and then upgraded to a shaded and relocated version of protobuf. We have
created a repo for this:

https://github.com/apache/hbase-thirdparty

But since the hadoop dependencies still pull in the protobuf 2.5 jars, our
coprocessors are still on protobuf 2.5. Recently we have opened a discuss
on how to deal with the upgrading of coprocessor. Glad to see that the
hadoop community is also willing to solve the problem.

Anu Engineer <ae...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道：

> +1, for the branch idea. Just FYI, Your biggest problem is proving that
> Hadoop and the downstream projects work correctly after you upgrade core
> components like Protobuf.
> So while branching and working on a branch is easy, merging back after you
> upgrade some of these core components is insanely hard. You might want to
> make sure that community buys into upgrading these components in the trunk.
> That way we will get testing and downstream components will notice when
> things break.
>
> That said, I have lobbied for the upgrade of Protobuf for a really long
> time; I have argued that 2.5 is out of support and we cannot stay on that
> branch forever; or we need to take ownership of the Protobuf 2.5 code base.
> It has been rightly pointed to me that while all the arguments I make is
> correct; it is a very complicated task to upgrade Protobuf, and the worst
> part is we will not even know what breaks until downstream projects pick up
> these changes and work against us.
>
> If we work off the Hadoop version 3 — and assume that we have "shading" in
> place for all deployments; it might be possible to get there; still a
> daunting task.
>
> So best of luck with the branch approach — But please remember, Merging
> back will be hard, Just my 2 cents.
>
> — Anu
>
>
>
>
> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zh...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate
> > branch with it's own ARM CI seems a really good idea.
> > By doing this we won't break any of the undergoing development in trunk
> and
> > a CI can be a very good way to show what are the
> > current problems and what have been fixed, it will also provide a very
> good
> > view for contributors that are intrested to working on
> > this. We can finally merge back the branch to trunk until the community
> > thinks it is good enough and stable enough. We can donate
> > ARM machines to the existing CI system for the job.
> >
> > I wonder if this approch possible?
> >
> > BR,
> >
> > On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > Thanks Vinay for bring this up, I am a member of "Openlab" community
> > > mentioned by Vinay. I am working on building and
> > > testing Hadoop components on aarch64 server these days, besides the
> > missing
> > > dependices of ARM platform issues #1 #2 #3
> > > mentioned by Vinay, other similar issue has also be found, such as the
> > > "PhantomJS" dependent package also missing for aarch64.
> > >
> > > To promote the ARM support for Hadoop, we have discussed and hoped to
> add
> > > an ARM specific CI to Hadoop repo. we are not
> > > sure about if there is any potential effect or confilict on the trunk
> > > branch, so maybe creating a ARM specific branch for doing these stuff
> > > is a better choice, what do you think?
> > >
> > > Hope to hear thoughts from you :)
> > >
> > > BR,
> > > Liu sheng
> > >
> > > Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> > >
> > > > Hi Folks,
> > > >
> > > > ARM is becoming famous lately in its processing capability and has
> got
> > > the
> > > > potential to run Bigdata workloads.
> > > > Many users have been moving to ARM machines due to its low cost.
> > > >
> > > > In the past there were attempts to compile Hadoop on ARM (Rasberry
> PI)
> > > for
> > > > experimental purposes. Today ARM architecture is taking some of the
> > > > serverside processing as well. So there will be/is a real need of
> > Hadoop
> > > to
> > > > support ARM architecture as well.
> > > >
> > > > There are bunch of users who are trying out building Hadoop on ARM,
> > > trying
> > > > to add ARM CI to hadoop and facing issues[1]. Also some
> > > >
> > > > As of today, Hadoop does not compile on ARM due to below issues,
> found
> > > from
> > > > testing done in openlab in [2].
> > > >
> > > > 1. Protobuf :
> > > > -------------------
> > > >      Hadoop project (also some downstream projects) stuck to protobuf
> > > 2.5.0
> > > > version, due to backward compatibility reasons. Protobuf-2.5.0 is not
> > > being
> > > > maintained in the community. While protobuf 3.x is being actively
> > adopted
> > > > widely, still protobuf 3.x provides wire compatibility for proto2
> > > messages.
> > > > Due to some compilation issues in the generated java code, which can
> > > induce
> > > > problems in downstream. Due to this reason protobuf upgrade from
> 2.5.0
> > > was
> > > > not taken up.
> > > > In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> > classpath
> > > > problem in downstream projects.
> > > >     There are patches available to fix compilation in Hadoop. But
> need
> > to
> > > > find a way to upgrade protobuf to latest version and still maintain
> the
> > > > downstream's classpath using shading feature of Hadoop build.
> > > >
> > > >      There is a Jira for protobuf upgrade[3] created even before
> shade
> > > > support was added to Hadoop. Now need to revisit the Jira and
> continue
> > > > explore possibilities.
> > > >
> > > > 2. leveldbjni:
> > > > ---------------
> > > >     Current leveldbjni used in YARN doesnot support ARM architecture,
> > > need
> > > > to check whether any of the future versions support ARM and can
> hadoop
> > > > upgrade to that version.
> > > >
> > > >
> > > > 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> > > > -------------------------
> > > > 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
> > default
> > > in
> > > > the maven repository. Workaround is to build it locally and keep in
> > local
> > > > maven repository.
> > > > Need to check whether any future versions of 'protoc-gen-grpc-java'
> is
> > > > having ARM executable and whether hadoop-yarn-csi can upgrade it?
> > > >
> > > >
> > > > Once the compilation issues are solved, then there might be many
> native
> > > > code related issues due to different architectures.
> > > > So to explore everything, need to join hands together and proceed.
> > > >
> > > >
> > > > Let us discuss and check, whether any body else out there who also
> need
> > > the
> > > > support of Hadoop on ARM architectures and ready to lend their hands
> > and
> > > > time in this work.
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/HADOOP-16358
> > > > [2]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> > > > [3] https://issues.apache.org/jira/browse/HADOOP-13363
> > > >
> > > > -Vinay
> > > >
> > >
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Anu Engineer <ae...@cloudera.com.INVALID>]

+1, for the branch idea. Just FYI, Your biggest problem is proving that
Hadoop and the downstream projects work correctly after you upgrade core
components like Protobuf.
So while branching and working on a branch is easy, merging back after you
upgrade some of these core components is insanely hard. You might want to
make sure that community buys into upgrading these components in the trunk.
That way we will get testing and downstream components will notice when
things break.

That said, I have lobbied for the upgrade of Protobuf for a really long
time; I have argued that 2.5 is out of support and we cannot stay on that
branch forever; or we need to take ownership of the Protobuf 2.5 code base.
It has been rightly pointed to me that while all the arguments I make is
correct; it is a very complicated task to upgrade Protobuf, and the worst
part is we will not even know what breaks until downstream projects pick up
these changes and work against us.

If we work off the Hadoop version 3 — and assume that we have "shading" in
place for all deployments; it might be possible to get there; still a
daunting task.

So best of luck with the branch approach — But please remember, Merging
back will be hard, Just my 2 cents.

— Anu




On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zh...@gmail.com>
wrote:

> Hi,
>
> Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate
> branch with it's own ARM CI seems a really good idea.
> By doing this we won't break any of the undergoing development in trunk and
> a CI can be a very good way to show what are the
> current problems and what have been fixed, it will also provide a very good
> view for contributors that are intrested to working on
> this. We can finally merge back the branch to trunk until the community
> thinks it is good enough and stable enough. We can donate
> ARM machines to the existing CI system for the job.
>
> I wonder if this approch possible?
>
> BR,
>
> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com> wrote:
>
> > Hi,
> >
> > Thanks Vinay for bring this up, I am a member of "Openlab" community
> > mentioned by Vinay. I am working on building and
> > testing Hadoop components on aarch64 server these days, besides the
> missing
> > dependices of ARM platform issues #1 #2 #3
> > mentioned by Vinay, other similar issue has also be found, such as the
> > "PhantomJS" dependent package also missing for aarch64.
> >
> > To promote the ARM support for Hadoop, we have discussed and hoped to add
> > an ARM specific CI to Hadoop repo. we are not
> > sure about if there is any potential effect or confilict on the trunk
> > branch, so maybe creating a ARM specific branch for doing these stuff
> > is a better choice, what do you think?
> >
> > Hope to hear thoughts from you :)
> >
> > BR,
> > Liu sheng
> >
> > Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> >
> > > Hi Folks,
> > >
> > > ARM is becoming famous lately in its processing capability and has got
> > the
> > > potential to run Bigdata workloads.
> > > Many users have been moving to ARM machines due to its low cost.
> > >
> > > In the past there were attempts to compile Hadoop on ARM (Rasberry PI)
> > for
> > > experimental purposes. Today ARM architecture is taking some of the
> > > serverside processing as well. So there will be/is a real need of
> Hadoop
> > to
> > > support ARM architecture as well.
> > >
> > > There are bunch of users who are trying out building Hadoop on ARM,
> > trying
> > > to add ARM CI to hadoop and facing issues[1]. Also some
> > >
> > > As of today, Hadoop does not compile on ARM due to below issues, found
> > from
> > > testing done in openlab in [2].
> > >
> > > 1. Protobuf :
> > > -------------------
> > >      Hadoop project (also some downstream projects) stuck to protobuf
> > 2.5.0
> > > version, due to backward compatibility reasons. Protobuf-2.5.0 is not
> > being
> > > maintained in the community. While protobuf 3.x is being actively
> adopted
> > > widely, still protobuf 3.x provides wire compatibility for proto2
> > messages.
> > > Due to some compilation issues in the generated java code, which can
> > induce
> > > problems in downstream. Due to this reason protobuf upgrade from 2.5.0
> > was
> > > not taken up.
> > > In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> classpath
> > > problem in downstream projects.
> > >     There are patches available to fix compilation in Hadoop. But need
> to
> > > find a way to upgrade protobuf to latest version and still maintain the
> > > downstream's classpath using shading feature of Hadoop build.
> > >
> > >      There is a Jira for protobuf upgrade[3] created even before shade
> > > support was added to Hadoop. Now need to revisit the Jira and continue
> > > explore possibilities.
> > >
> > > 2. leveldbjni:
> > > ---------------
> > >     Current leveldbjni used in YARN doesnot support ARM architecture,
> > need
> > > to check whether any of the future versions support ARM and can hadoop
> > > upgrade to that version.
> > >
> > >
> > > 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> > > -------------------------
> > > 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
> default
> > in
> > > the maven repository. Workaround is to build it locally and keep in
> local
> > > maven repository.
> > > Need to check whether any future versions of 'protoc-gen-grpc-java' is
> > > having ARM executable and whether hadoop-yarn-csi can upgrade it?
> > >
> > >
> > > Once the compilation issues are solved, then there might be many native
> > > code related issues due to different architectures.
> > > So to explore everything, need to join hands together and proceed.
> > >
> > >
> > > Let us discuss and check, whether any body else out there who also need
> > the
> > > support of Hadoop on ARM architectures and ready to lend their hands
> and
> > > time in this work.
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/HADOOP-16358
> > > [2]
> > >
> > >
> >
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> > > [3] https://issues.apache.org/jira/browse/HADOOP-13363
> > >
> > > -Vinay
> > >
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Anu Engineer <ae...@cloudera.com.INVALID>]

+1, for the branch idea. Just FYI, Your biggest problem is proving that
Hadoop and the downstream projects work correctly after you upgrade core
components like Protobuf.
So while branching and working on a branch is easy, merging back after you
upgrade some of these core components is insanely hard. You might want to
make sure that community buys into upgrading these components in the trunk.
That way we will get testing and downstream components will notice when
things break.

That said, I have lobbied for the upgrade of Protobuf for a really long
time; I have argued that 2.5 is out of support and we cannot stay on that
branch forever; or we need to take ownership of the Protobuf 2.5 code base.
It has been rightly pointed to me that while all the arguments I make is
correct; it is a very complicated task to upgrade Protobuf, and the worst
part is we will not even know what breaks until downstream projects pick up
these changes and work against us.

If we work off the Hadoop version 3 — and assume that we have "shading" in
place for all deployments; it might be possible to get there; still a
daunting task.

So best of luck with the branch approach — But please remember, Merging
back will be hard, Just my 2 cents.

— Anu




On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zh...@gmail.com>
wrote:

> Hi,
>
> Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate
> branch with it's own ARM CI seems a really good idea.
> By doing this we won't break any of the undergoing development in trunk and
> a CI can be a very good way to show what are the
> current problems and what have been fixed, it will also provide a very good
> view for contributors that are intrested to working on
> this. We can finally merge back the branch to trunk until the community
> thinks it is good enough and stable enough. We can donate
> ARM machines to the existing CI system for the job.
>
> I wonder if this approch possible?
>
> BR,
>
> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com> wrote:
>
> > Hi,
> >
> > Thanks Vinay for bring this up, I am a member of "Openlab" community
> > mentioned by Vinay. I am working on building and
> > testing Hadoop components on aarch64 server these days, besides the
> missing
> > dependices of ARM platform issues #1 #2 #3
> > mentioned by Vinay, other similar issue has also be found, such as the
> > "PhantomJS" dependent package also missing for aarch64.
> >
> > To promote the ARM support for Hadoop, we have discussed and hoped to add
> > an ARM specific CI to Hadoop repo. we are not
> > sure about if there is any potential effect or confilict on the trunk
> > branch, so maybe creating a ARM specific branch for doing these stuff
> > is a better choice, what do you think?
> >
> > Hope to hear thoughts from you :)
> >
> > BR,
> > Liu sheng
> >
> > Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> >
> > > Hi Folks,
> > >
> > > ARM is becoming famous lately in its processing capability and has got
> > the
> > > potential to run Bigdata workloads.
> > > Many users have been moving to ARM machines due to its low cost.
> > >
> > > In the past there were attempts to compile Hadoop on ARM (Rasberry PI)
> > for
> > > experimental purposes. Today ARM architecture is taking some of the
> > > serverside processing as well. So there will be/is a real need of
> Hadoop
> > to
> > > support ARM architecture as well.
> > >
> > > There are bunch of users who are trying out building Hadoop on ARM,
> > trying
> > > to add ARM CI to hadoop and facing issues[1]. Also some
> > >
> > > As of today, Hadoop does not compile on ARM due to below issues, found
> > from
> > > testing done in openlab in [2].
> > >
> > > 1. Protobuf :
> > > -------------------
> > >      Hadoop project (also some downstream projects) stuck to protobuf
> > 2.5.0
> > > version, due to backward compatibility reasons. Protobuf-2.5.0 is not
> > being
> > > maintained in the community. While protobuf 3.x is being actively
> adopted
> > > widely, still protobuf 3.x provides wire compatibility for proto2
> > messages.
> > > Due to some compilation issues in the generated java code, which can
> > induce
> > > problems in downstream. Due to this reason protobuf upgrade from 2.5.0
> > was
> > > not taken up.
> > > In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> classpath
> > > problem in downstream projects.
> > >     There are patches available to fix compilation in Hadoop. But need
> to
> > > find a way to upgrade protobuf to latest version and still maintain the
> > > downstream's classpath using shading feature of Hadoop build.
> > >
> > >      There is a Jira for protobuf upgrade[3] created even before shade
> > > support was added to Hadoop. Now need to revisit the Jira and continue
> > > explore possibilities.
> > >
> > > 2. leveldbjni:
> > > ---------------
> > >     Current leveldbjni used in YARN doesnot support ARM architecture,
> > need
> > > to check whether any of the future versions support ARM and can hadoop
> > > upgrade to that version.
> > >
> > >
> > > 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> > > -------------------------
> > > 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
> default
> > in
> > > the maven repository. Workaround is to build it locally and keep in
> local
> > > maven repository.
> > > Need to check whether any future versions of 'protoc-gen-grpc-java' is
> > > having ARM executable and whether hadoop-yarn-csi can upgrade it?
> > >
> > >
> > > Once the compilation issues are solved, then there might be many native
> > > code related issues due to different architectures.
> > > So to explore everything, need to join hands together and proceed.
> > >
> > >
> > > Let us discuss and check, whether any body else out there who also need
> > the
> > > support of Hadoop on ARM architectures and ready to lend their hands
> and
> > > time in this work.
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/HADOOP-16358
> > > [2]
> > >
> > >
> >
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> > > [3] https://issues.apache.org/jira/browse/HADOOP-13363
> > >
> > > -Vinay
> > >
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Anu Engineer <ae...@cloudera.com.INVALID>]

+1, for the branch idea. Just FYI, Your biggest problem is proving that
Hadoop and the downstream projects work correctly after you upgrade core
components like Protobuf.
So while branching and working on a branch is easy, merging back after you
upgrade some of these core components is insanely hard. You might want to
make sure that community buys into upgrading these components in the trunk.
That way we will get testing and downstream components will notice when
things break.

That said, I have lobbied for the upgrade of Protobuf for a really long
time; I have argued that 2.5 is out of support and we cannot stay on that
branch forever; or we need to take ownership of the Protobuf 2.5 code base.
It has been rightly pointed to me that while all the arguments I make is
correct; it is a very complicated task to upgrade Protobuf, and the worst
part is we will not even know what breaks until downstream projects pick up
these changes and work against us.

If we work off the Hadoop version 3 — and assume that we have "shading" in
place for all deployments; it might be possible to get there; still a
daunting task.

So best of luck with the branch approach — But please remember, Merging
back will be hard, Just my 2 cents.

— Anu




On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zh...@gmail.com>
wrote:

> Hi,
>
> Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate
> branch with it's own ARM CI seems a really good idea.
> By doing this we won't break any of the undergoing development in trunk and
> a CI can be a very good way to show what are the
> current problems and what have been fixed, it will also provide a very good
> view for contributors that are intrested to working on
> this. We can finally merge back the branch to trunk until the community
> thinks it is good enough and stable enough. We can donate
> ARM machines to the existing CI system for the job.
>
> I wonder if this approch possible?
>
> BR,
>
> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com> wrote:
>
> > Hi,
> >
> > Thanks Vinay for bring this up, I am a member of "Openlab" community
> > mentioned by Vinay. I am working on building and
> > testing Hadoop components on aarch64 server these days, besides the
> missing
> > dependices of ARM platform issues #1 #2 #3
> > mentioned by Vinay, other similar issue has also be found, such as the
> > "PhantomJS" dependent package also missing for aarch64.
> >
> > To promote the ARM support for Hadoop, we have discussed and hoped to add
> > an ARM specific CI to Hadoop repo. we are not
> > sure about if there is any potential effect or confilict on the trunk
> > branch, so maybe creating a ARM specific branch for doing these stuff
> > is a better choice, what do you think?
> >
> > Hope to hear thoughts from you :)
> >
> > BR,
> > Liu sheng
> >
> > Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> >
> > > Hi Folks,
> > >
> > > ARM is becoming famous lately in its processing capability and has got
> > the
> > > potential to run Bigdata workloads.
> > > Many users have been moving to ARM machines due to its low cost.
> > >
> > > In the past there were attempts to compile Hadoop on ARM (Rasberry PI)
> > for
> > > experimental purposes. Today ARM architecture is taking some of the
> > > serverside processing as well. So there will be/is a real need of
> Hadoop
> > to
> > > support ARM architecture as well.
> > >
> > > There are bunch of users who are trying out building Hadoop on ARM,
> > trying
> > > to add ARM CI to hadoop and facing issues[1]. Also some
> > >
> > > As of today, Hadoop does not compile on ARM due to below issues, found
> > from
> > > testing done in openlab in [2].
> > >
> > > 1. Protobuf :
> > > -------------------
> > >      Hadoop project (also some downstream projects) stuck to protobuf
> > 2.5.0
> > > version, due to backward compatibility reasons. Protobuf-2.5.0 is not
> > being
> > > maintained in the community. While protobuf 3.x is being actively
> adopted
> > > widely, still protobuf 3.x provides wire compatibility for proto2
> > messages.
> > > Due to some compilation issues in the generated java code, which can
> > induce
> > > problems in downstream. Due to this reason protobuf upgrade from 2.5.0
> > was
> > > not taken up.
> > > In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> classpath
> > > problem in downstream projects.
> > >     There are patches available to fix compilation in Hadoop. But need
> to
> > > find a way to upgrade protobuf to latest version and still maintain the
> > > downstream's classpath using shading feature of Hadoop build.
> > >
> > >      There is a Jira for protobuf upgrade[3] created even before shade
> > > support was added to Hadoop. Now need to revisit the Jira and continue
> > > explore possibilities.
> > >
> > > 2. leveldbjni:
> > > ---------------
> > >     Current leveldbjni used in YARN doesnot support ARM architecture,
> > need
> > > to check whether any of the future versions support ARM and can hadoop
> > > upgrade to that version.
> > >
> > >
> > > 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> > > -------------------------
> > > 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
> default
> > in
> > > the maven repository. Workaround is to build it locally and keep in
> local
> > > maven repository.
> > > Need to check whether any future versions of 'protoc-gen-grpc-java' is
> > > having ARM executable and whether hadoop-yarn-csi can upgrade it?
> > >
> > >
> > > Once the compilation issues are solved, then there might be many native
> > > code related issues due to different architectures.
> > > So to explore everything, need to join hands together and proceed.
> > >
> > >
> > > Let us discuss and check, whether any body else out there who also need
> > the
> > > support of Hadoop on ARM architectures and ready to lend their hands
> and
> > > time in this work.
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/HADOOP-16358
> > > [2]
> > >
> > >
> >
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> > > [3] https://issues.apache.org/jira/browse/HADOOP-13363
> > >
> > > -Vinay
> > >
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Anu Engineer <ae...@cloudera.com.INVALID>]

+1, for the branch idea. Just FYI, Your biggest problem is proving that
Hadoop and the downstream projects work correctly after you upgrade core
components like Protobuf.
So while branching and working on a branch is easy, merging back after you
upgrade some of these core components is insanely hard. You might want to
make sure that community buys into upgrading these components in the trunk.
That way we will get testing and downstream components will notice when
things break.

That said, I have lobbied for the upgrade of Protobuf for a really long
time; I have argued that 2.5 is out of support and we cannot stay on that
branch forever; or we need to take ownership of the Protobuf 2.5 code base.
It has been rightly pointed to me that while all the arguments I make is
correct; it is a very complicated task to upgrade Protobuf, and the worst
part is we will not even know what breaks until downstream projects pick up
these changes and work against us.

If we work off the Hadoop version 3 — and assume that we have "shading" in
place for all deployments; it might be possible to get there; still a
daunting task.

So best of luck with the branch approach — But please remember, Merging
back will be hard, Just my 2 cents.

— Anu




On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <zh...@gmail.com>
wrote:

> Hi,
>
> Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate
> branch with it's own ARM CI seems a really good idea.
> By doing this we won't break any of the undergoing development in trunk and
> a CI can be a very good way to show what are the
> current problems and what have been fixed, it will also provide a very good
> view for contributors that are intrested to working on
> this. We can finally merge back the branch to trunk until the community
> thinks it is good enough and stable enough. We can donate
> ARM machines to the existing CI system for the job.
>
> I wonder if this approch possible?
>
> BR,
>
> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com> wrote:
>
> > Hi,
> >
> > Thanks Vinay for bring this up, I am a member of "Openlab" community
> > mentioned by Vinay. I am working on building and
> > testing Hadoop components on aarch64 server these days, besides the
> missing
> > dependices of ARM platform issues #1 #2 #3
> > mentioned by Vinay, other similar issue has also be found, such as the
> > "PhantomJS" dependent package also missing for aarch64.
> >
> > To promote the ARM support for Hadoop, we have discussed and hoped to add
> > an ARM specific CI to Hadoop repo. we are not
> > sure about if there is any potential effect or confilict on the trunk
> > branch, so maybe creating a ARM specific branch for doing these stuff
> > is a better choice, what do you think?
> >
> > Hope to hear thoughts from you :)
> >
> > BR,
> > Liu sheng
> >
> > Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
> >
> > > Hi Folks,
> > >
> > > ARM is becoming famous lately in its processing capability and has got
> > the
> > > potential to run Bigdata workloads.
> > > Many users have been moving to ARM machines due to its low cost.
> > >
> > > In the past there were attempts to compile Hadoop on ARM (Rasberry PI)
> > for
> > > experimental purposes. Today ARM architecture is taking some of the
> > > serverside processing as well. So there will be/is a real need of
> Hadoop
> > to
> > > support ARM architecture as well.
> > >
> > > There are bunch of users who are trying out building Hadoop on ARM,
> > trying
> > > to add ARM CI to hadoop and facing issues[1]. Also some
> > >
> > > As of today, Hadoop does not compile on ARM due to below issues, found
> > from
> > > testing done in openlab in [2].
> > >
> > > 1. Protobuf :
> > > -------------------
> > >      Hadoop project (also some downstream projects) stuck to protobuf
> > 2.5.0
> > > version, due to backward compatibility reasons. Protobuf-2.5.0 is not
> > being
> > > maintained in the community. While protobuf 3.x is being actively
> adopted
> > > widely, still protobuf 3.x provides wire compatibility for proto2
> > messages.
> > > Due to some compilation issues in the generated java code, which can
> > induce
> > > problems in downstream. Due to this reason protobuf upgrade from 2.5.0
> > was
> > > not taken up.
> > > In 3.0.0 onwards, hadoop supports shading of libraries to avoid
> classpath
> > > problem in downstream projects.
> > >     There are patches available to fix compilation in Hadoop. But need
> to
> > > find a way to upgrade protobuf to latest version and still maintain the
> > > downstream's classpath using shading feature of Hadoop build.
> > >
> > >      There is a Jira for protobuf upgrade[3] created even before shade
> > > support was added to Hadoop. Now need to revisit the Jira and continue
> > > explore possibilities.
> > >
> > > 2. leveldbjni:
> > > ---------------
> > >     Current leveldbjni used in YARN doesnot support ARM architecture,
> > need
> > > to check whether any of the future versions support ARM and can hadoop
> > > upgrade to that version.
> > >
> > >
> > > 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> > > -------------------------
> > > 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by
> default
> > in
> > > the maven repository. Workaround is to build it locally and keep in
> local
> > > maven repository.
> > > Need to check whether any future versions of 'protoc-gen-grpc-java' is
> > > having ARM executable and whether hadoop-yarn-csi can upgrade it?
> > >
> > >
> > > Once the compilation issues are solved, then there might be many native
> > > code related issues due to different architectures.
> > > So to explore everything, need to join hands together and proceed.
> > >
> > >
> > > Let us discuss and check, whether any body else out there who also need
> > the
> > > support of Hadoop on ARM architectures and ready to lend their hands
> and
> > > time in this work.
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/HADOOP-16358
> > > [2]
> > >
> > >
> >
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> > > [3] https://issues.apache.org/jira/browse/HADOOP-13363
> > >
> > > -Vinay
> > >
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Zhenyu Zheng <zh...@gmail.com>]

Hi,

Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate
branch with it's own ARM CI seems a really good idea.
By doing this we won't break any of the undergoing development in trunk and
a CI can be a very good way to show what are the
current problems and what have been fixed, it will also provide a very good
view for contributors that are intrested to working on
this. We can finally merge back the branch to trunk until the community
thinks it is good enough and stable enough. We can donate
ARM machines to the existing CI system for the job.

I wonder if this approch possible?

BR,

On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com> wrote:

> Hi,
>
> Thanks Vinay for bring this up, I am a member of "Openlab" community
> mentioned by Vinay. I am working on building and
> testing Hadoop components on aarch64 server these days, besides the missing
> dependices of ARM platform issues #1 #2 #3
> mentioned by Vinay, other similar issue has also be found, such as the
> "PhantomJS" dependent package also missing for aarch64.
>
> To promote the ARM support for Hadoop, we have discussed and hoped to add
> an ARM specific CI to Hadoop repo. we are not
> sure about if there is any potential effect or confilict on the trunk
> branch, so maybe creating a ARM specific branch for doing these stuff
> is a better choice, what do you think?
>
> Hope to hear thoughts from you :)
>
> BR,
> Liu sheng
>
> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
>
> > Hi Folks,
> >
> > ARM is becoming famous lately in its processing capability and has got
> the
> > potential to run Bigdata workloads.
> > Many users have been moving to ARM machines due to its low cost.
> >
> > In the past there were attempts to compile Hadoop on ARM (Rasberry PI)
> for
> > experimental purposes. Today ARM architecture is taking some of the
> > serverside processing as well. So there will be/is a real need of Hadoop
> to
> > support ARM architecture as well.
> >
> > There are bunch of users who are trying out building Hadoop on ARM,
> trying
> > to add ARM CI to hadoop and facing issues[1]. Also some
> >
> > As of today, Hadoop does not compile on ARM due to below issues, found
> from
> > testing done in openlab in [2].
> >
> > 1. Protobuf :
> > -------------------
> >      Hadoop project (also some downstream projects) stuck to protobuf
> 2.5.0
> > version, due to backward compatibility reasons. Protobuf-2.5.0 is not
> being
> > maintained in the community. While protobuf 3.x is being actively adopted
> > widely, still protobuf 3.x provides wire compatibility for proto2
> messages.
> > Due to some compilation issues in the generated java code, which can
> induce
> > problems in downstream. Due to this reason protobuf upgrade from 2.5.0
> was
> > not taken up.
> > In 3.0.0 onwards, hadoop supports shading of libraries to avoid classpath
> > problem in downstream projects.
> >     There are patches available to fix compilation in Hadoop. But need to
> > find a way to upgrade protobuf to latest version and still maintain the
> > downstream's classpath using shading feature of Hadoop build.
> >
> >      There is a Jira for protobuf upgrade[3] created even before shade
> > support was added to Hadoop. Now need to revisit the Jira and continue
> > explore possibilities.
> >
> > 2. leveldbjni:
> > ---------------
> >     Current leveldbjni used in YARN doesnot support ARM architecture,
> need
> > to check whether any of the future versions support ARM and can hadoop
> > upgrade to that version.
> >
> >
> > 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> > -------------------------
> > 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by default
> in
> > the maven repository. Workaround is to build it locally and keep in local
> > maven repository.
> > Need to check whether any future versions of 'protoc-gen-grpc-java' is
> > having ARM executable and whether hadoop-yarn-csi can upgrade it?
> >
> >
> > Once the compilation issues are solved, then there might be many native
> > code related issues due to different architectures.
> > So to explore everything, need to join hands together and proceed.
> >
> >
> > Let us discuss and check, whether any body else out there who also need
> the
> > support of Hadoop on ARM architectures and ready to lend their hands and
> > time in this work.
> >
> >
> > [1] https://issues.apache.org/jira/browse/HADOOP-16358
> > [2]
> >
> >
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> > [3] https://issues.apache.org/jira/browse/HADOOP-13363
> >
> > -Vinay
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Zhenyu Zheng <zh...@gmail.com>]

Hi,

Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate
branch with it's own ARM CI seems a really good idea.
By doing this we won't break any of the undergoing development in trunk and
a CI can be a very good way to show what are the
current problems and what have been fixed, it will also provide a very good
view for contributors that are intrested to working on
this. We can finally merge back the branch to trunk until the community
thinks it is good enough and stable enough. We can donate
ARM machines to the existing CI system for the job.

I wonder if this approch possible?

BR,

On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <li...@gmail.com> wrote:

> Hi,
>
> Thanks Vinay for bring this up, I am a member of "Openlab" community
> mentioned by Vinay. I am working on building and
> testing Hadoop components on aarch64 server these days, besides the missing
> dependices of ARM platform issues #1 #2 #3
> mentioned by Vinay, other similar issue has also be found, such as the
> "PhantomJS" dependent package also missing for aarch64.
>
> To promote the ARM support for Hadoop, we have discussed and hoped to add
> an ARM specific CI to Hadoop repo. we are not
> sure about if there is any potential effect or confilict on the trunk
> branch, so maybe creating a ARM specific branch for doing these stuff
> is a better choice, what do you think?
>
> Hope to hear thoughts from you :)
>
> BR,
> Liu sheng
>
> Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：
>
> > Hi Folks,
> >
> > ARM is becoming famous lately in its processing capability and has got
> the
> > potential to run Bigdata workloads.
> > Many users have been moving to ARM machines due to its low cost.
> >
> > In the past there were attempts to compile Hadoop on ARM (Rasberry PI)
> for
> > experimental purposes. Today ARM architecture is taking some of the
> > serverside processing as well. So there will be/is a real need of Hadoop
> to
> > support ARM architecture as well.
> >
> > There are bunch of users who are trying out building Hadoop on ARM,
> trying
> > to add ARM CI to hadoop and facing issues[1]. Also some
> >
> > As of today, Hadoop does not compile on ARM due to below issues, found
> from
> > testing done in openlab in [2].
> >
> > 1. Protobuf :
> > -------------------
> >      Hadoop project (also some downstream projects) stuck to protobuf
> 2.5.0
> > version, due to backward compatibility reasons. Protobuf-2.5.0 is not
> being
> > maintained in the community. While protobuf 3.x is being actively adopted
> > widely, still protobuf 3.x provides wire compatibility for proto2
> messages.
> > Due to some compilation issues in the generated java code, which can
> induce
> > problems in downstream. Due to this reason protobuf upgrade from 2.5.0
> was
> > not taken up.
> > In 3.0.0 onwards, hadoop supports shading of libraries to avoid classpath
> > problem in downstream projects.
> >     There are patches available to fix compilation in Hadoop. But need to
> > find a way to upgrade protobuf to latest version and still maintain the
> > downstream's classpath using shading feature of Hadoop build.
> >
> >      There is a Jira for protobuf upgrade[3] created even before shade
> > support was added to Hadoop. Now need to revisit the Jira and continue
> > explore possibilities.
> >
> > 2. leveldbjni:
> > ---------------
> >     Current leveldbjni used in YARN doesnot support ARM architecture,
> need
> > to check whether any of the future versions support ARM and can hadoop
> > upgrade to that version.
> >
> >
> > 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> > -------------------------
> > 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by default
> in
> > the maven repository. Workaround is to build it locally and keep in local
> > maven repository.
> > Need to check whether any future versions of 'protoc-gen-grpc-java' is
> > having ARM executable and whether hadoop-yarn-csi can upgrade it?
> >
> >
> > Once the compilation issues are solved, then there might be many native
> > code related issues due to different architectures.
> > So to explore everything, need to join hands together and proceed.
> >
> >
> > Let us discuss and check, whether any body else out there who also need
> the
> > support of Hadoop on ARM architectures and ready to lend their hands and
> > time in this work.
> >
> >
> > [1] https://issues.apache.org/jira/browse/HADOOP-16358
> > [2]
> >
> >
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> > [3] https://issues.apache.org/jira/browse/HADOOP-13363
> >
> > -Vinay
> >
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Sheng Liu <li...@gmail.com>]

Hi,

Thanks Vinay for bring this up, I am a member of "Openlab" community
mentioned by Vinay. I am working on building and
testing Hadoop components on aarch64 server these days, besides the missing
dependices of ARM platform issues #1 #2 #3
mentioned by Vinay, other similar issue has also be found, such as the
"PhantomJS" dependent package also missing for aarch64.

To promote the ARM support for Hadoop, we have discussed and hoped to add
an ARM specific CI to Hadoop repo. we are not
sure about if there is any potential effect or confilict on the trunk
branch, so maybe creating a ARM specific branch for doing these stuff
is a better choice, what do you think?

Hope to hear thoughts from you :)

BR,
Liu sheng

Vinayakumar B <vi...@apache.org> 于2019年8月27日周二 上午5:34写道：

> Hi Folks,
>
> ARM is becoming famous lately in its processing capability and has got the
> potential to run Bigdata workloads.
> Many users have been moving to ARM machines due to its low cost.
>
> In the past there were attempts to compile Hadoop on ARM (Rasberry PI) for
> experimental purposes. Today ARM architecture is taking some of the
> serverside processing as well. So there will be/is a real need of Hadoop to
> support ARM architecture as well.
>
> There are bunch of users who are trying out building Hadoop on ARM, trying
> to add ARM CI to hadoop and facing issues[1]. Also some
>
> As of today, Hadoop does not compile on ARM due to below issues, found from
> testing done in openlab in [2].
>
> 1. Protobuf :
> -------------------
>      Hadoop project (also some downstream projects) stuck to protobuf 2.5.0
> version, due to backward compatibility reasons. Protobuf-2.5.0 is not being
> maintained in the community. While protobuf 3.x is being actively adopted
> widely, still protobuf 3.x provides wire compatibility for proto2 messages.
> Due to some compilation issues in the generated java code, which can induce
> problems in downstream. Due to this reason protobuf upgrade from 2.5.0 was
> not taken up.
> In 3.0.0 onwards, hadoop supports shading of libraries to avoid classpath
> problem in downstream projects.
>     There are patches available to fix compilation in Hadoop. But need to
> find a way to upgrade protobuf to latest version and still maintain the
> downstream's classpath using shading feature of Hadoop build.
>
>      There is a Jira for protobuf upgrade[3] created even before shade
> support was added to Hadoop. Now need to revisit the Jira and continue
> explore possibilities.
>
> 2. leveldbjni:
> ---------------
>     Current leveldbjni used in YARN doesnot support ARM architecture, need
> to check whether any of the future versions support ARM and can hadoop
> upgrade to that version.
>
>
> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> -------------------------
> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by default in
> the maven repository. Workaround is to build it locally and keep in local
> maven repository.
> Need to check whether any future versions of 'protoc-gen-grpc-java' is
> having ARM executable and whether hadoop-yarn-csi can upgrade it?
>
>
> Once the compilation issues are solved, then there might be many native
> code related issues due to different architectures.
> So to explore everything, need to join hands together and proceed.
>
>
> Let us discuss and check, whether any body else out there who also need the
> support of Hadoop on ARM architectures and ready to lend their hands and
> time in this work.
>
>
> [1] https://issues.apache.org/jira/browse/HADOOP-16358
> [2]
>
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>
> -Vinay
>