You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Chinna Rao Lalam <la...@gmail.com> on 2020/03/10 12:10:46 UTC
HIVE building on ARM
Hi All,
We are working on HIVE building and running on ARM architecture. As part of
this i want to add one periodic HIVE JENKINS JOB on ARM machines. Which
helps to identity gaps for ARM, If any issues identified we will keep on
working to fix those issues.
Currently to run HIVE on aarch64, we need to upgrade to protobuf-3.7.1.
HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939> upgraded
protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are working
on this.
Thanks,
Chinna Rao Lalam
Re: HIVE building on ARM
Posted by Chinna Rao Lalam <la...@gmail.com>.
Hi Ashutosh,
I want to migrate
https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/ this
job to the new system https://ci-hadoop.apache.org. Please grant me
rights to create jobs on the new Jenkins https://ci-builds.apache.org. My
LDAP userid : chinnaraol.
I have requested the same in apache infra. Thank you.
Thank you Stamatis,Zoltan for the pointers.
Thanks,
Chinna
On Thu, Jun 18, 2020 at 5:36 PM Stamatis Zampetakis <za...@gmail.com>
wrote:
> Hello Chinna,
>
> The hudson-jobadmin privilege can be granted by PMC chairs.
> I don't know if there is any particular policy in Hive on who should have
> this privilege so I guess you should request it from Ashutosh.
>
> Best,
> Stamatis
>
> On Thu, Jun 18, 2020 at 12:05 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
>
>> Hey Chinna!
>>
>> On 6/18/20 11:43 AM, Chinna Rao Lalam wrote:
>> > As you said, migrating this job to the new ci-hadoop instance looks
>> good as
>> > Hadoop also shares the same armN slaves.
>>
>> Sounds great!
>>
>> > I am able to login the new ci-hadoop instance with Apache LDAP
>> credentials,
>> > but i am not able to see the job creation option. Should I request
>> access
>> > or the process for creation of a job is different than jenkin?.
>> > Please guide me to create the new job in the ci-hadoop instance. I will
>> > migrate this job after connecting the armN slaves to the new system.
>>
>>
>> I've also logged in - and apparently I've create job rights; I'm happy to
>> help, but the best would be to self-service yourselft :)
>> I think you may miss the "hudson-jobadmin" privilege.
>> Probably Gavin (or someone on the infra team) could help you with that..
>> to talk to them quickly - you can reach them on the #asfinfra channel (on
>> the asf-slack).
>>
>> The migration effort is coordinated thru the hadoop-migrations mailing
>> list (I've cc-ed that list)
>> you may want to subscribe to it by sending a mail to:
>> hadoop-migrations-subscribe@infra.apache.org
>>
>> cheers,
>> Zoltan
>>
>>
>>
>> >
>> > Thanks
>> > Chinna
>> >
>> > On Wed, Jun 17, 2020 at 11:57 AM Zhenyu Zheng <
>> zhengzhenyulixi@gmail.com>
>> > wrote:
>> >
>> >> Hi Zoltan,
>> >>
>> >> Thanks alot for the information, so looks like one possible solution
>> is as
>> >> you suggest, move the current ARM2 and ARM3 (those two were donate to
>> >> builds.apache.org by us) to the new ci-hadoop cluster and set up the
>> jobs
>> >> just as what has been done in current jenkins.
>> >>
>> >> I will also ask our team member works on other projects to find out
>> what
>> >> the status of other projects is.
>> >>
>> >> BR,
>> >>
>> >> On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
>> >>
>> >>> Hey,
>> >>>
>> >>> There is an effort by the Apache Infra to change the way Jenkins
>> stuff is
>> >>> organized; a couple months ago Gavin wrote an email about it:
>> >>>
>> >>>
>> http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3CCAN0Gg1dOdEPzatJz9bOFE-2veR7QG7H0HMvYjmsLdgjr8_RPew@mail.gmail.com%3E
>> >>> The resources for running these jobs are coming from the H0~H21 slaves
>> >>> which will be migrated to the new jenkins master eventually.
>> >>>
>> >>> >> So please
>> >>> >> suggest a way which direction we can move and can you share some
>> >>> details
>> >>> >> about the new ci-hadoop instance.
>> >>>
>> >>> Since Hadoop testing is also happening on ARM - I think the best
>> would be
>> >>> to also migrate the armN slaves and the Hive arm nightly over to the
>> new
>> >>> ci-hadoop instance.
>> >>>
>> >>> On 6/16/20 8:40 AM, Zhenyu Zheng wrote:
>> >>>> Thanks for the info, I wonder if where does the resource of ci-hadoop
>> >>> and
>> >>>> hive-test-kube come from? Do they include ARM resources?
>> >>>
>> >>> Interesting question; the resources for Hive testing are donated by
>> >>> Cloudera.
>> >>> About the ARM workers I think Chinna could provide more details.
>> >>> ...I've no idea don't know who sponsors the Hxx slaves
>> >>>
>> >>>> Can you provide some more information about how the new
>> hive-test-kube
>> >>> is
>> >>>> running?
>> >>> It's basically a Jenkins instance which is using kubernetes pods to
>> run
>> >>> things.
>> >>> The whole thing is running on a GKE cluster.
>> >>> While I was working on it I collected stuff needed for it in this
>> repo:
>> >>> https://github.com/kgyrtkirk/hive-test-kube/
>> >>> it should be possible to start a new deployment using that stuff
>> >>>
>> >>> cheers,
>> >>> Zoltan
>> >>>
>> >>>>
>> >>>> BR,
>> >>>> Kevin Zheng
>> >>>>
>> >>>> On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam <
>> >>>> lalamchinnarao13@gmail.com> wrote:
>> >>>>
>> >>>>> Hi Zoltan,
>> >>>>>
>> >>>>> Thanks for the update.
>> >>>>>
>> >>>>> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is
>> >>>>> targeting to run hive tests daily on "arm" slaves, it is using 2 arm
>> >>>>> slaves.
>> >>>>> To find any potential issues with "arm" and fix the issues. So
>> please
>> >>>>> suggest a way which direction we can move and can you share some
>> >>> details
>> >>>>> about the new ci-hadoop instance.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Chinna
>> >>>>>
>> >>>>> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <ki...@rxd.hu>
>> wrote:
>> >>>>>
>> >>>>>> Hey all,
>> >>>>>>
>> >>>>>> In an ticket (INFRA-20416) Gavin asked me if we are completely off
>> >>>>>> builds.apache.org - when I went over the jobs I've saw that
>> >>>>>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running
>> there
>> >>>>>> once a day.
>> >>>>>>
>> >>>>>> Since builds.apache.org will be shut down in sometime in the
>> future
>> >>> - we
>> >>>>>> should move this job to the new ci-hadoop instance or to
>> >>> hive-test-kube.
>> >>>>>> The key feature of the job is that it runs the test on the "armX"
>> >>> slaves;
>> >>>>>> which are statically configured on b.a.o.
>> >>>>>> Not sure which way to go - but we will have to move in some
>> direction.
>> >>>>>>
>> >>>>>> cheers,
>> >>>>>> Zoltan
>> >>>>>>
>> >>>>>>
>> >>>>>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote:
>> >>>>>>> Hi Chinna,
>> >>>>>>>
>> >>>>>>> Thanks alot for the reply, I uploaded a patch and also a github PR
>> >>> for
>> >>>>>>> https://issues.apache.org/jira/browse/HIVE-21939 .
>> >>>>>>> In the patch, I bumped the protobuf used in standalone-metadata to
>> >>> 2.6.1
>> >>>>>>> and added a new profile, this profile will identify
>> >>>>>>> the hardware architecture and if it is Aarch64, it will override
>> the
>> >>>>>>> protobuf group.id and package to com.github.os72 which
>> >>>>>>> includes ARM support. For X86 platform, Hive will still download
>> the
>> >>>>>>> protobuf packages from org.google repo. I think with
>> >>>>>>> this method, we can keep the influence to existing x86 users to
>> the
>> >>>>>>> minimum. I hope this could be a acceptable short-term
>> >>>>>>> solution.
>> >>>>>>>
>> >>>>>>> I've manually tested on my machine and the github PR travis CI
>> test
>> >>> has
>> >>>>>>> already passed, so the build process is OK, so let's
>> >>>>>>> wait for the full test result from builds.apache.org.
>> >>>>>>>
>> >>>>>>> BR,
>> >>>>>>>
>> >>>>>>> Zhenyu
>> >>>>>>>
>> >>>>>>> On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <
>> >>>>>> lalamchinnarao13@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> Hi Zhenyu,
>> >>>>>>>>
>> >>>>>>>> Until HBase dependency resolved, without effecting the existing
>> code
>> >>>>>> on X86
>> >>>>>>>> i suggest create a separate profile with "os72" repo.
>> >>>>>>>>
>> >>>>>>>> Down the line we should have common version for both X86 and ARM.
>> >>>>>>>>
>> >>>>>>>> Hope It Helps,
>> >>>>>>>> Chinna
>> >>>>>>>>
>> >>>>>>>> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <
>> >>>>>> zhengzhenyulixi@gmail.com>
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> Hi Chinna, David and others might interested,
>> >>>>>>>>>
>> >>>>>>>>> Thanks for bring this up, we are currently working on improving
>> >>>>>> enabling
>> >>>>>>>>> big-data software on the ARM platform,
>> >>>>>>>>> we have already done fixes and providing CIs to some of the
>> >>> well-know
>> >>>>>>>>> projects like:
>> >>>>>>>>> 1. Hadoop:
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
>> >>>>>>>>> 2. Spark:
>> https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
>> >>>>>>>>> 3. HBase:
>> >>>>>>>>>
>> >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
>> >>>>>>>>>
>> >>>>>>>>> And we are now working on projects including Hive, Kudu, etc.
>> >>>>>>>>>
>> >>>>>>>>> Regarding to the protobuf upgrades in Hive, except upgrading to
>> 3.x
>> >>>>>> and
>> >>>>>>>>> break dependency for HBase, there can
>> >>>>>>>>> be some possible short-term plan(or walk-arounds), doing thes
>> can
>> >>> make
>> >>>>>>>> Hive
>> >>>>>>>>> work on ARM without break any
>> >>>>>>>>> dependencies, and then we can interact with Hbase project to see
>> >>> how
>> >>>>>> can
>> >>>>>>>> we
>> >>>>>>>>> both upgrade to 3.x(since this
>> >>>>>>>>> make take some time).
>> >>>>>>>>>
>> >>>>>>>>> Those possible solutions can be:
>> >>>>>>>>> 1. Using pre-patched protobuf 2.5.0 with ARM support
>> >>>>>>>>> from org.openlabtesting repo, some projects(HBase did
>> >>>>>>>>> this: https://github.com/apache/hbase/pull/959, and we will
>> add a
>> >>>>>>>> profile
>> >>>>>>>>> for this, it will detact the host arch and
>> >>>>>>>>> will only use this pre-patched protobuf when the host arch is
>> ARM
>> >>> so
>> >>>>>> that
>> >>>>>>>>> nothing will be affected for existing x86
>> >>>>>>>>> users; Spark and Hadoop used some other package from this repo
>> as
>> >>> they
>> >>>>>>>> have
>> >>>>>>>>> already upgraded protobuf).
>> >>>>>>>>>
>> >>>>>>>>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
>> >>>>>>>>>
>> >>>>>>
>> >>>
>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
>> >>>>>>>>> Hive already used ``protoc-jar-maven-plugin`` from this repo:
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>
>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>
>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
>> >>>>>>>>> so it is a minor update.
>> >>>>>>>>>
>> >>>>>>>>> I've tested both ways, worked OK on the ARM machine.
>> >>>>>>>>>
>> >>>>>>>>> So my ideas will be that we make this a two-step-work, first
>> step
>> >>> we
>> >>>>>> use
>> >>>>>>>>> one of the short-term plan to enable
>> >>>>>>>>> Hive on ARM, and 2nd Step, we look for the long run to discuss
>> with
>> >>>>>> HBase
>> >>>>>>>>> to find out a fesiable solution.
>> >>>>>>>>>
>> >>>>>>>>> BR,
>> >>>>>>>>>
>> >>>>>>>>> Zhenyu Zheng
>> >>>>>>>>>
>> >>>>>>>>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
>> >>>>>>>>> lalamchinnarao13@gmail.com>
>> >>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Thanks David for the quick reply.
>> >>>>>>>>>> Yes currently HBase protobuf version mismatch is the issue.
>> >>>>>>>>>>
>> >>>>>>>>>> We are investigating any common version of protobuf is there to
>> >>>>>> support
>> >>>>>>>>> for
>> >>>>>>>>>> HBase an HIVE.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati
>> >>>>>>>>> <dlavati@cloudera.com.invalid
>> >>>>>>>>>>>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Hi,
>> >>>>>>>>>>>
>> >>>>>>>>>>> I recently investigated this in
>> >>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it
>> didn't
>> >>> look
>> >>>>>>>>>>> feasible, as protobuf has to be in sync across the whole
>> stack,
>> >>> but
>> >>>>>>>> the
>> >>>>>>>>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x,
>> >>> see
>> >>>>>> my
>> >>>>>>>>>>> comment on the jira and
>> >>>>>>>>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414
>> .
>> >>>>>>>>>>>
>> >>>>>>>>>>> Kind Regards,
>> >>>>>>>>>>> David
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
>> >>>>>>>>>>> lalamchinnarao13@gmail.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Hi All,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> We are working on HIVE building and running on ARM
>> >>> architecture. As
>> >>>>>>>>>> part
>> >>>>>>>>>>> of
>> >>>>>>>>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM
>> >>> machines.
>> >>>>>>>>> Which
>> >>>>>>>>>>>> helps to identity gaps for ARM, If any issues identified we
>> will
>> >>>>>>>> keep
>> >>>>>>>>>> on
>> >>>>>>>>>>>> working to fix those issues.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Currently to run HIVE on aarch64, we need to upgrade to
>> >>>>>>>>> protobuf-3.7.1.
>> >>>>>>>>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939
>> >
>> >>>>>>>>> upgraded
>> >>>>>>>>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures.
>> We
>> >>> are
>> >>>>>>>>>>> working
>> >>>>>>>>>>>> on this.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>> Chinna Rao Lalam
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> --
>> >>>>>>>>>> Hope It Helps,
>> >>>>>>>>>> Chinna
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> Hope It Helps,
>> >>>>>>>> Chinna
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Hope It Helps,
>> >>>>> Chinna
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>
--
Hope It Helps,
Chinna
Re: HIVE building on ARM
Posted by Stamatis Zampetakis <za...@gmail.com>.
Hello Chinna,
The hudson-jobadmin privilege can be granted by PMC chairs.
I don't know if there is any particular policy in Hive on who should have
this privilege so I guess you should request it from Ashutosh.
Best,
Stamatis
On Thu, Jun 18, 2020 at 12:05 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
> Hey Chinna!
>
> On 6/18/20 11:43 AM, Chinna Rao Lalam wrote:
> > As you said, migrating this job to the new ci-hadoop instance looks good
> as
> > Hadoop also shares the same armN slaves.
>
> Sounds great!
>
> > I am able to login the new ci-hadoop instance with Apache LDAP
> credentials,
> > but i am not able to see the job creation option. Should I request access
> > or the process for creation of a job is different than jenkin?.
> > Please guide me to create the new job in the ci-hadoop instance. I will
> > migrate this job after connecting the armN slaves to the new system.
>
>
> I've also logged in - and apparently I've create job rights; I'm happy to
> help, but the best would be to self-service yourselft :)
> I think you may miss the "hudson-jobadmin" privilege.
> Probably Gavin (or someone on the infra team) could help you with that..
> to talk to them quickly - you can reach them on the #asfinfra channel (on
> the asf-slack).
>
> The migration effort is coordinated thru the hadoop-migrations mailing
> list (I've cc-ed that list)
> you may want to subscribe to it by sending a mail to:
> hadoop-migrations-subscribe@infra.apache.org
>
> cheers,
> Zoltan
>
>
>
> >
> > Thanks
> > Chinna
> >
> > On Wed, Jun 17, 2020 at 11:57 AM Zhenyu Zheng <zhengzhenyulixi@gmail.com
> >
> > wrote:
> >
> >> Hi Zoltan,
> >>
> >> Thanks alot for the information, so looks like one possible solution is
> as
> >> you suggest, move the current ARM2 and ARM3 (those two were donate to
> >> builds.apache.org by us) to the new ci-hadoop cluster and set up the
> jobs
> >> just as what has been done in current jenkins.
> >>
> >> I will also ask our team member works on other projects to find out what
> >> the status of other projects is.
> >>
> >> BR,
> >>
> >> On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
> >>
> >>> Hey,
> >>>
> >>> There is an effort by the Apache Infra to change the way Jenkins stuff
> is
> >>> organized; a couple months ago Gavin wrote an email about it:
> >>>
> >>>
> http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3CCAN0Gg1dOdEPzatJz9bOFE-2veR7QG7H0HMvYjmsLdgjr8_RPew@mail.gmail.com%3E
> >>> The resources for running these jobs are coming from the H0~H21 slaves
> >>> which will be migrated to the new jenkins master eventually.
> >>>
> >>> >> So please
> >>> >> suggest a way which direction we can move and can you share some
> >>> details
> >>> >> about the new ci-hadoop instance.
> >>>
> >>> Since Hadoop testing is also happening on ARM - I think the best would
> be
> >>> to also migrate the armN slaves and the Hive arm nightly over to the
> new
> >>> ci-hadoop instance.
> >>>
> >>> On 6/16/20 8:40 AM, Zhenyu Zheng wrote:
> >>>> Thanks for the info, I wonder if where does the resource of ci-hadoop
> >>> and
> >>>> hive-test-kube come from? Do they include ARM resources?
> >>>
> >>> Interesting question; the resources for Hive testing are donated by
> >>> Cloudera.
> >>> About the ARM workers I think Chinna could provide more details.
> >>> ...I've no idea don't know who sponsors the Hxx slaves
> >>>
> >>>> Can you provide some more information about how the new hive-test-kube
> >>> is
> >>>> running?
> >>> It's basically a Jenkins instance which is using kubernetes pods to run
> >>> things.
> >>> The whole thing is running on a GKE cluster.
> >>> While I was working on it I collected stuff needed for it in this repo:
> >>> https://github.com/kgyrtkirk/hive-test-kube/
> >>> it should be possible to start a new deployment using that stuff
> >>>
> >>> cheers,
> >>> Zoltan
> >>>
> >>>>
> >>>> BR,
> >>>> Kevin Zheng
> >>>>
> >>>> On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam <
> >>>> lalamchinnarao13@gmail.com> wrote:
> >>>>
> >>>>> Hi Zoltan,
> >>>>>
> >>>>> Thanks for the update.
> >>>>>
> >>>>> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is
> >>>>> targeting to run hive tests daily on "arm" slaves, it is using 2 arm
> >>>>> slaves.
> >>>>> To find any potential issues with "arm" and fix the issues. So please
> >>>>> suggest a way which direction we can move and can you share some
> >>> details
> >>>>> about the new ci-hadoop instance.
> >>>>>
> >>>>> Thanks,
> >>>>> Chinna
> >>>>>
> >>>>> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <ki...@rxd.hu>
> wrote:
> >>>>>
> >>>>>> Hey all,
> >>>>>>
> >>>>>> In an ticket (INFRA-20416) Gavin asked me if we are completely off
> >>>>>> builds.apache.org - when I went over the jobs I've saw that
> >>>>>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running
> there
> >>>>>> once a day.
> >>>>>>
> >>>>>> Since builds.apache.org will be shut down in sometime in the future
> >>> - we
> >>>>>> should move this job to the new ci-hadoop instance or to
> >>> hive-test-kube.
> >>>>>> The key feature of the job is that it runs the test on the "armX"
> >>> slaves;
> >>>>>> which are statically configured on b.a.o.
> >>>>>> Not sure which way to go - but we will have to move in some
> direction.
> >>>>>>
> >>>>>> cheers,
> >>>>>> Zoltan
> >>>>>>
> >>>>>>
> >>>>>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote:
> >>>>>>> Hi Chinna,
> >>>>>>>
> >>>>>>> Thanks alot for the reply, I uploaded a patch and also a github PR
> >>> for
> >>>>>>> https://issues.apache.org/jira/browse/HIVE-21939 .
> >>>>>>> In the patch, I bumped the protobuf used in standalone-metadata to
> >>> 2.6.1
> >>>>>>> and added a new profile, this profile will identify
> >>>>>>> the hardware architecture and if it is Aarch64, it will override
> the
> >>>>>>> protobuf group.id and package to com.github.os72 which
> >>>>>>> includes ARM support. For X86 platform, Hive will still download
> the
> >>>>>>> protobuf packages from org.google repo. I think with
> >>>>>>> this method, we can keep the influence to existing x86 users to the
> >>>>>>> minimum. I hope this could be a acceptable short-term
> >>>>>>> solution.
> >>>>>>>
> >>>>>>> I've manually tested on my machine and the github PR travis CI test
> >>> has
> >>>>>>> already passed, so the build process is OK, so let's
> >>>>>>> wait for the full test result from builds.apache.org.
> >>>>>>>
> >>>>>>> BR,
> >>>>>>>
> >>>>>>> Zhenyu
> >>>>>>>
> >>>>>>> On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <
> >>>>>> lalamchinnarao13@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi Zhenyu,
> >>>>>>>>
> >>>>>>>> Until HBase dependency resolved, without effecting the existing
> code
> >>>>>> on X86
> >>>>>>>> i suggest create a separate profile with "os72" repo.
> >>>>>>>>
> >>>>>>>> Down the line we should have common version for both X86 and ARM.
> >>>>>>>>
> >>>>>>>> Hope It Helps,
> >>>>>>>> Chinna
> >>>>>>>>
> >>>>>>>> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <
> >>>>>> zhengzhenyulixi@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Chinna, David and others might interested,
> >>>>>>>>>
> >>>>>>>>> Thanks for bring this up, we are currently working on improving
> >>>>>> enabling
> >>>>>>>>> big-data software on the ARM platform,
> >>>>>>>>> we have already done fixes and providing CIs to some of the
> >>> well-know
> >>>>>>>>> projects like:
> >>>>>>>>> 1. Hadoop:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
> >>>>>>>>> 2. Spark:
> https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
> >>>>>>>>> 3. HBase:
> >>>>>>>>>
> >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
> >>>>>>>>>
> >>>>>>>>> And we are now working on projects including Hive, Kudu, etc.
> >>>>>>>>>
> >>>>>>>>> Regarding to the protobuf upgrades in Hive, except upgrading to
> 3.x
> >>>>>> and
> >>>>>>>>> break dependency for HBase, there can
> >>>>>>>>> be some possible short-term plan(or walk-arounds), doing thes can
> >>> make
> >>>>>>>> Hive
> >>>>>>>>> work on ARM without break any
> >>>>>>>>> dependencies, and then we can interact with Hbase project to see
> >>> how
> >>>>>> can
> >>>>>>>> we
> >>>>>>>>> both upgrade to 3.x(since this
> >>>>>>>>> make take some time).
> >>>>>>>>>
> >>>>>>>>> Those possible solutions can be:
> >>>>>>>>> 1. Using pre-patched protobuf 2.5.0 with ARM support
> >>>>>>>>> from org.openlabtesting repo, some projects(HBase did
> >>>>>>>>> this: https://github.com/apache/hbase/pull/959, and we will add
> a
> >>>>>>>> profile
> >>>>>>>>> for this, it will detact the host arch and
> >>>>>>>>> will only use this pre-patched protobuf when the host arch is ARM
> >>> so
> >>>>>> that
> >>>>>>>>> nothing will be affected for existing x86
> >>>>>>>>> users; Spark and Hadoop used some other package from this repo as
> >>> they
> >>>>>>>> have
> >>>>>>>>> already upgraded protobuf).
> >>>>>>>>>
> >>>>>>>>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
> >>>>>>>>>
> >>>>>>
> >>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
> >>>>>>>>> Hive already used ``protoc-jar-maven-plugin`` from this repo:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
> >>>>>>>>> so it is a minor update.
> >>>>>>>>>
> >>>>>>>>> I've tested both ways, worked OK on the ARM machine.
> >>>>>>>>>
> >>>>>>>>> So my ideas will be that we make this a two-step-work, first step
> >>> we
> >>>>>> use
> >>>>>>>>> one of the short-term plan to enable
> >>>>>>>>> Hive on ARM, and 2nd Step, we look for the long run to discuss
> with
> >>>>>> HBase
> >>>>>>>>> to find out a fesiable solution.
> >>>>>>>>>
> >>>>>>>>> BR,
> >>>>>>>>>
> >>>>>>>>> Zhenyu Zheng
> >>>>>>>>>
> >>>>>>>>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
> >>>>>>>>> lalamchinnarao13@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Thanks David for the quick reply.
> >>>>>>>>>> Yes currently HBase protobuf version mismatch is the issue.
> >>>>>>>>>>
> >>>>>>>>>> We are investigating any common version of protobuf is there to
> >>>>>> support
> >>>>>>>>> for
> >>>>>>>>>> HBase an HIVE.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati
> >>>>>>>>> <dlavati@cloudera.com.invalid
> >>>>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> I recently investigated this in
> >>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't
> >>> look
> >>>>>>>>>>> feasible, as protobuf has to be in sync across the whole stack,
> >>> but
> >>>>>>>> the
> >>>>>>>>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x,
> >>> see
> >>>>>> my
> >>>>>>>>>>> comment on the jira and
> >>>>>>>>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
> >>>>>>>>>>>
> >>>>>>>>>>> Kind Regards,
> >>>>>>>>>>> David
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
> >>>>>>>>>>> lalamchinnarao13@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>
> >>>>>>>>>>>> We are working on HIVE building and running on ARM
> >>> architecture. As
> >>>>>>>>>> part
> >>>>>>>>>>> of
> >>>>>>>>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM
> >>> machines.
> >>>>>>>>> Which
> >>>>>>>>>>>> helps to identity gaps for ARM, If any issues identified we
> will
> >>>>>>>> keep
> >>>>>>>>>> on
> >>>>>>>>>>>> working to fix those issues.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Currently to run HIVE on aarch64, we need to upgrade to
> >>>>>>>>> protobuf-3.7.1.
> >>>>>>>>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939>
> >>>>>>>>> upgraded
> >>>>>>>>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures.
> We
> >>> are
> >>>>>>>>>>> working
> >>>>>>>>>>>> on this.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Chinna Rao Lalam
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Hope It Helps,
> >>>>>>>>>> Chinna
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Hope It Helps,
> >>>>>>>> Chinna
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Hope It Helps,
> >>>>> Chinna
> >>>>>
> >>>>
> >>>
> >>
> >
>
Re: HIVE building on ARM
Posted by Zoltan Haindrich <ki...@rxd.hu>.
Hey Chinna!
On 6/18/20 11:43 AM, Chinna Rao Lalam wrote:
> As you said, migrating this job to the new ci-hadoop instance looks good as
> Hadoop also shares the same armN slaves.
Sounds great!
> I am able to login the new ci-hadoop instance with Apache LDAP credentials,
> but i am not able to see the job creation option. Should I request access
> or the process for creation of a job is different than jenkin?.
> Please guide me to create the new job in the ci-hadoop instance. I will
> migrate this job after connecting the armN slaves to the new system.
I've also logged in - and apparently I've create job rights; I'm happy to help, but the best would be to self-service yourselft :)
I think you may miss the "hudson-jobadmin" privilege.
Probably Gavin (or someone on the infra team) could help you with that..
to talk to them quickly - you can reach them on the #asfinfra channel (on the asf-slack).
The migration effort is coordinated thru the hadoop-migrations mailing list (I've cc-ed that list)
you may want to subscribe to it by sending a mail to: hadoop-migrations-subscribe@infra.apache.org
cheers,
Zoltan
>
> Thanks
> Chinna
>
> On Wed, Jun 17, 2020 at 11:57 AM Zhenyu Zheng <zh...@gmail.com>
> wrote:
>
>> Hi Zoltan,
>>
>> Thanks alot for the information, so looks like one possible solution is as
>> you suggest, move the current ARM2 and ARM3 (those two were donate to
>> builds.apache.org by us) to the new ci-hadoop cluster and set up the jobs
>> just as what has been done in current jenkins.
>>
>> I will also ask our team member works on other projects to find out what
>> the status of other projects is.
>>
>> BR,
>>
>> On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
>>
>>> Hey,
>>>
>>> There is an effort by the Apache Infra to change the way Jenkins stuff is
>>> organized; a couple months ago Gavin wrote an email about it:
>>>
>>> http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3CCAN0Gg1dOdEPzatJz9bOFE-2veR7QG7H0HMvYjmsLdgjr8_RPew@mail.gmail.com%3E
>>> The resources for running these jobs are coming from the H0~H21 slaves
>>> which will be migrated to the new jenkins master eventually.
>>>
>>> >> So please
>>> >> suggest a way which direction we can move and can you share some
>>> details
>>> >> about the new ci-hadoop instance.
>>>
>>> Since Hadoop testing is also happening on ARM - I think the best would be
>>> to also migrate the armN slaves and the Hive arm nightly over to the new
>>> ci-hadoop instance.
>>>
>>> On 6/16/20 8:40 AM, Zhenyu Zheng wrote:
>>>> Thanks for the info, I wonder if where does the resource of ci-hadoop
>>> and
>>>> hive-test-kube come from? Do they include ARM resources?
>>>
>>> Interesting question; the resources for Hive testing are donated by
>>> Cloudera.
>>> About the ARM workers I think Chinna could provide more details.
>>> ...I've no idea don't know who sponsors the Hxx slaves
>>>
>>>> Can you provide some more information about how the new hive-test-kube
>>> is
>>>> running?
>>> It's basically a Jenkins instance which is using kubernetes pods to run
>>> things.
>>> The whole thing is running on a GKE cluster.
>>> While I was working on it I collected stuff needed for it in this repo:
>>> https://github.com/kgyrtkirk/hive-test-kube/
>>> it should be possible to start a new deployment using that stuff
>>>
>>> cheers,
>>> Zoltan
>>>
>>>>
>>>> BR,
>>>> Kevin Zheng
>>>>
>>>> On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam <
>>>> lalamchinnarao13@gmail.com> wrote:
>>>>
>>>>> Hi Zoltan,
>>>>>
>>>>> Thanks for the update.
>>>>>
>>>>> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is
>>>>> targeting to run hive tests daily on "arm" slaves, it is using 2 arm
>>>>> slaves.
>>>>> To find any potential issues with "arm" and fix the issues. So please
>>>>> suggest a way which direction we can move and can you share some
>>> details
>>>>> about the new ci-hadoop instance.
>>>>>
>>>>> Thanks,
>>>>> Chinna
>>>>>
>>>>> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
>>>>>
>>>>>> Hey all,
>>>>>>
>>>>>> In an ticket (INFRA-20416) Gavin asked me if we are completely off
>>>>>> builds.apache.org - when I went over the jobs I've saw that
>>>>>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there
>>>>>> once a day.
>>>>>>
>>>>>> Since builds.apache.org will be shut down in sometime in the future
>>> - we
>>>>>> should move this job to the new ci-hadoop instance or to
>>> hive-test-kube.
>>>>>> The key feature of the job is that it runs the test on the "armX"
>>> slaves;
>>>>>> which are statically configured on b.a.o.
>>>>>> Not sure which way to go - but we will have to move in some direction.
>>>>>>
>>>>>> cheers,
>>>>>> Zoltan
>>>>>>
>>>>>>
>>>>>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote:
>>>>>>> Hi Chinna,
>>>>>>>
>>>>>>> Thanks alot for the reply, I uploaded a patch and also a github PR
>>> for
>>>>>>> https://issues.apache.org/jira/browse/HIVE-21939 .
>>>>>>> In the patch, I bumped the protobuf used in standalone-metadata to
>>> 2.6.1
>>>>>>> and added a new profile, this profile will identify
>>>>>>> the hardware architecture and if it is Aarch64, it will override the
>>>>>>> protobuf group.id and package to com.github.os72 which
>>>>>>> includes ARM support. For X86 platform, Hive will still download the
>>>>>>> protobuf packages from org.google repo. I think with
>>>>>>> this method, we can keep the influence to existing x86 users to the
>>>>>>> minimum. I hope this could be a acceptable short-term
>>>>>>> solution.
>>>>>>>
>>>>>>> I've manually tested on my machine and the github PR travis CI test
>>> has
>>>>>>> already passed, so the build process is OK, so let's
>>>>>>> wait for the full test result from builds.apache.org.
>>>>>>>
>>>>>>> BR,
>>>>>>>
>>>>>>> Zhenyu
>>>>>>>
>>>>>>> On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <
>>>>>> lalamchinnarao13@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Zhenyu,
>>>>>>>>
>>>>>>>> Until HBase dependency resolved, without effecting the existing code
>>>>>> on X86
>>>>>>>> i suggest create a separate profile with "os72" repo.
>>>>>>>>
>>>>>>>> Down the line we should have common version for both X86 and ARM.
>>>>>>>>
>>>>>>>> Hope It Helps,
>>>>>>>> Chinna
>>>>>>>>
>>>>>>>> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <
>>>>>> zhengzhenyulixi@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Chinna, David and others might interested,
>>>>>>>>>
>>>>>>>>> Thanks for bring this up, we are currently working on improving
>>>>>> enabling
>>>>>>>>> big-data software on the ARM platform,
>>>>>>>>> we have already done fixes and providing CIs to some of the
>>> well-know
>>>>>>>>> projects like:
>>>>>>>>> 1. Hadoop:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
>>>>>>>>> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
>>>>>>>>> 3. HBase:
>>>>>>>>>
>>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
>>>>>>>>>
>>>>>>>>> And we are now working on projects including Hive, Kudu, etc.
>>>>>>>>>
>>>>>>>>> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x
>>>>>> and
>>>>>>>>> break dependency for HBase, there can
>>>>>>>>> be some possible short-term plan(or walk-arounds), doing thes can
>>> make
>>>>>>>> Hive
>>>>>>>>> work on ARM without break any
>>>>>>>>> dependencies, and then we can interact with Hbase project to see
>>> how
>>>>>> can
>>>>>>>> we
>>>>>>>>> both upgrade to 3.x(since this
>>>>>>>>> make take some time).
>>>>>>>>>
>>>>>>>>> Those possible solutions can be:
>>>>>>>>> 1. Using pre-patched protobuf 2.5.0 with ARM support
>>>>>>>>> from org.openlabtesting repo, some projects(HBase did
>>>>>>>>> this: https://github.com/apache/hbase/pull/959, and we will add a
>>>>>>>> profile
>>>>>>>>> for this, it will detact the host arch and
>>>>>>>>> will only use this pre-patched protobuf when the host arch is ARM
>>> so
>>>>>> that
>>>>>>>>> nothing will be affected for existing x86
>>>>>>>>> users; Spark and Hadoop used some other package from this repo as
>>> they
>>>>>>>> have
>>>>>>>>> already upgraded protobuf).
>>>>>>>>>
>>>>>>>>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
>>>>>>>>>
>>>>>>
>>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
>>>>>>>>> Hive already used ``protoc-jar-maven-plugin`` from this repo:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
>>>>>>>>> so it is a minor update.
>>>>>>>>>
>>>>>>>>> I've tested both ways, worked OK on the ARM machine.
>>>>>>>>>
>>>>>>>>> So my ideas will be that we make this a two-step-work, first step
>>> we
>>>>>> use
>>>>>>>>> one of the short-term plan to enable
>>>>>>>>> Hive on ARM, and 2nd Step, we look for the long run to discuss with
>>>>>> HBase
>>>>>>>>> to find out a fesiable solution.
>>>>>>>>>
>>>>>>>>> BR,
>>>>>>>>>
>>>>>>>>> Zhenyu Zheng
>>>>>>>>>
>>>>>>>>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
>>>>>>>>> lalamchinnarao13@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks David for the quick reply.
>>>>>>>>>> Yes currently HBase protobuf version mismatch is the issue.
>>>>>>>>>>
>>>>>>>>>> We are investigating any common version of protobuf is there to
>>>>>> support
>>>>>>>>> for
>>>>>>>>>> HBase an HIVE.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati
>>>>>>>>> <dlavati@cloudera.com.invalid
>>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I recently investigated this in
>>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't
>>> look
>>>>>>>>>>> feasible, as protobuf has to be in sync across the whole stack,
>>> but
>>>>>>>> the
>>>>>>>>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x,
>>> see
>>>>>> my
>>>>>>>>>>> comment on the jira and
>>>>>>>>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
>>>>>>>>>>>
>>>>>>>>>>> Kind Regards,
>>>>>>>>>>> David
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
>>>>>>>>>>> lalamchinnarao13@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> We are working on HIVE building and running on ARM
>>> architecture. As
>>>>>>>>>> part
>>>>>>>>>>> of
>>>>>>>>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM
>>> machines.
>>>>>>>>> Which
>>>>>>>>>>>> helps to identity gaps for ARM, If any issues identified we will
>>>>>>>> keep
>>>>>>>>>> on
>>>>>>>>>>>> working to fix those issues.
>>>>>>>>>>>>
>>>>>>>>>>>> Currently to run HIVE on aarch64, we need to upgrade to
>>>>>>>>> protobuf-3.7.1.
>>>>>>>>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939>
>>>>>>>>> upgraded
>>>>>>>>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We
>>> are
>>>>>>>>>>> working
>>>>>>>>>>>> on this.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Chinna Rao Lalam
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Hope It Helps,
>>>>>>>>>> Chinna
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Hope It Helps,
>>>>>>>> Chinna
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Hope It Helps,
>>>>> Chinna
>>>>>
>>>>
>>>
>>
>
Re: HIVE building on ARM
Posted by Zoltan Haindrich <ki...@rxd.hu>.
Hey Chinna!
On 6/18/20 11:43 AM, Chinna Rao Lalam wrote:
> As you said, migrating this job to the new ci-hadoop instance looks good as
> Hadoop also shares the same armN slaves.
Sounds great!
> I am able to login the new ci-hadoop instance with Apache LDAP credentials,
> but i am not able to see the job creation option. Should I request access
> or the process for creation of a job is different than jenkin?.
> Please guide me to create the new job in the ci-hadoop instance. I will
> migrate this job after connecting the armN slaves to the new system.
I've also logged in - and apparently I've create job rights; I'm happy to help, but the best would be to self-service yourselft :)
I think you may miss the "hudson-jobadmin" privilege.
Probably Gavin (or someone on the infra team) could help you with that..
to talk to them quickly - you can reach them on the #asfinfra channel (on the asf-slack).
The migration effort is coordinated thru the hadoop-migrations mailing list (I've cc-ed that list)
you may want to subscribe to it by sending a mail to: hadoop-migrations-subscribe@infra.apache.org
cheers,
Zoltan
>
> Thanks
> Chinna
>
> On Wed, Jun 17, 2020 at 11:57 AM Zhenyu Zheng <zh...@gmail.com>
> wrote:
>
>> Hi Zoltan,
>>
>> Thanks alot for the information, so looks like one possible solution is as
>> you suggest, move the current ARM2 and ARM3 (those two were donate to
>> builds.apache.org by us) to the new ci-hadoop cluster and set up the jobs
>> just as what has been done in current jenkins.
>>
>> I will also ask our team member works on other projects to find out what
>> the status of other projects is.
>>
>> BR,
>>
>> On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
>>
>>> Hey,
>>>
>>> There is an effort by the Apache Infra to change the way Jenkins stuff is
>>> organized; a couple months ago Gavin wrote an email about it:
>>>
>>> http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3CCAN0Gg1dOdEPzatJz9bOFE-2veR7QG7H0HMvYjmsLdgjr8_RPew@mail.gmail.com%3E
>>> The resources for running these jobs are coming from the H0~H21 slaves
>>> which will be migrated to the new jenkins master eventually.
>>>
>>> >> So please
>>> >> suggest a way which direction we can move and can you share some
>>> details
>>> >> about the new ci-hadoop instance.
>>>
>>> Since Hadoop testing is also happening on ARM - I think the best would be
>>> to also migrate the armN slaves and the Hive arm nightly over to the new
>>> ci-hadoop instance.
>>>
>>> On 6/16/20 8:40 AM, Zhenyu Zheng wrote:
>>>> Thanks for the info, I wonder if where does the resource of ci-hadoop
>>> and
>>>> hive-test-kube come from? Do they include ARM resources?
>>>
>>> Interesting question; the resources for Hive testing are donated by
>>> Cloudera.
>>> About the ARM workers I think Chinna could provide more details.
>>> ...I've no idea don't know who sponsors the Hxx slaves
>>>
>>>> Can you provide some more information about how the new hive-test-kube
>>> is
>>>> running?
>>> It's basically a Jenkins instance which is using kubernetes pods to run
>>> things.
>>> The whole thing is running on a GKE cluster.
>>> While I was working on it I collected stuff needed for it in this repo:
>>> https://github.com/kgyrtkirk/hive-test-kube/
>>> it should be possible to start a new deployment using that stuff
>>>
>>> cheers,
>>> Zoltan
>>>
>>>>
>>>> BR,
>>>> Kevin Zheng
>>>>
>>>> On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam <
>>>> lalamchinnarao13@gmail.com> wrote:
>>>>
>>>>> Hi Zoltan,
>>>>>
>>>>> Thanks for the update.
>>>>>
>>>>> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is
>>>>> targeting to run hive tests daily on "arm" slaves, it is using 2 arm
>>>>> slaves.
>>>>> To find any potential issues with "arm" and fix the issues. So please
>>>>> suggest a way which direction we can move and can you share some
>>> details
>>>>> about the new ci-hadoop instance.
>>>>>
>>>>> Thanks,
>>>>> Chinna
>>>>>
>>>>> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
>>>>>
>>>>>> Hey all,
>>>>>>
>>>>>> In an ticket (INFRA-20416) Gavin asked me if we are completely off
>>>>>> builds.apache.org - when I went over the jobs I've saw that
>>>>>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there
>>>>>> once a day.
>>>>>>
>>>>>> Since builds.apache.org will be shut down in sometime in the future
>>> - we
>>>>>> should move this job to the new ci-hadoop instance or to
>>> hive-test-kube.
>>>>>> The key feature of the job is that it runs the test on the "armX"
>>> slaves;
>>>>>> which are statically configured on b.a.o.
>>>>>> Not sure which way to go - but we will have to move in some direction.
>>>>>>
>>>>>> cheers,
>>>>>> Zoltan
>>>>>>
>>>>>>
>>>>>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote:
>>>>>>> Hi Chinna,
>>>>>>>
>>>>>>> Thanks alot for the reply, I uploaded a patch and also a github PR
>>> for
>>>>>>> https://issues.apache.org/jira/browse/HIVE-21939 .
>>>>>>> In the patch, I bumped the protobuf used in standalone-metadata to
>>> 2.6.1
>>>>>>> and added a new profile, this profile will identify
>>>>>>> the hardware architecture and if it is Aarch64, it will override the
>>>>>>> protobuf group.id and package to com.github.os72 which
>>>>>>> includes ARM support. For X86 platform, Hive will still download the
>>>>>>> protobuf packages from org.google repo. I think with
>>>>>>> this method, we can keep the influence to existing x86 users to the
>>>>>>> minimum. I hope this could be a acceptable short-term
>>>>>>> solution.
>>>>>>>
>>>>>>> I've manually tested on my machine and the github PR travis CI test
>>> has
>>>>>>> already passed, so the build process is OK, so let's
>>>>>>> wait for the full test result from builds.apache.org.
>>>>>>>
>>>>>>> BR,
>>>>>>>
>>>>>>> Zhenyu
>>>>>>>
>>>>>>> On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <
>>>>>> lalamchinnarao13@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Zhenyu,
>>>>>>>>
>>>>>>>> Until HBase dependency resolved, without effecting the existing code
>>>>>> on X86
>>>>>>>> i suggest create a separate profile with "os72" repo.
>>>>>>>>
>>>>>>>> Down the line we should have common version for both X86 and ARM.
>>>>>>>>
>>>>>>>> Hope It Helps,
>>>>>>>> Chinna
>>>>>>>>
>>>>>>>> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <
>>>>>> zhengzhenyulixi@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Chinna, David and others might interested,
>>>>>>>>>
>>>>>>>>> Thanks for bring this up, we are currently working on improving
>>>>>> enabling
>>>>>>>>> big-data software on the ARM platform,
>>>>>>>>> we have already done fixes and providing CIs to some of the
>>> well-know
>>>>>>>>> projects like:
>>>>>>>>> 1. Hadoop:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
>>>>>>>>> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
>>>>>>>>> 3. HBase:
>>>>>>>>>
>>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
>>>>>>>>>
>>>>>>>>> And we are now working on projects including Hive, Kudu, etc.
>>>>>>>>>
>>>>>>>>> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x
>>>>>> and
>>>>>>>>> break dependency for HBase, there can
>>>>>>>>> be some possible short-term plan(or walk-arounds), doing thes can
>>> make
>>>>>>>> Hive
>>>>>>>>> work on ARM without break any
>>>>>>>>> dependencies, and then we can interact with Hbase project to see
>>> how
>>>>>> can
>>>>>>>> we
>>>>>>>>> both upgrade to 3.x(since this
>>>>>>>>> make take some time).
>>>>>>>>>
>>>>>>>>> Those possible solutions can be:
>>>>>>>>> 1. Using pre-patched protobuf 2.5.0 with ARM support
>>>>>>>>> from org.openlabtesting repo, some projects(HBase did
>>>>>>>>> this: https://github.com/apache/hbase/pull/959, and we will add a
>>>>>>>> profile
>>>>>>>>> for this, it will detact the host arch and
>>>>>>>>> will only use this pre-patched protobuf when the host arch is ARM
>>> so
>>>>>> that
>>>>>>>>> nothing will be affected for existing x86
>>>>>>>>> users; Spark and Hadoop used some other package from this repo as
>>> they
>>>>>>>> have
>>>>>>>>> already upgraded protobuf).
>>>>>>>>>
>>>>>>>>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
>>>>>>>>>
>>>>>>
>>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
>>>>>>>>> Hive already used ``protoc-jar-maven-plugin`` from this repo:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
>>>>>>>>> so it is a minor update.
>>>>>>>>>
>>>>>>>>> I've tested both ways, worked OK on the ARM machine.
>>>>>>>>>
>>>>>>>>> So my ideas will be that we make this a two-step-work, first step
>>> we
>>>>>> use
>>>>>>>>> one of the short-term plan to enable
>>>>>>>>> Hive on ARM, and 2nd Step, we look for the long run to discuss with
>>>>>> HBase
>>>>>>>>> to find out a fesiable solution.
>>>>>>>>>
>>>>>>>>> BR,
>>>>>>>>>
>>>>>>>>> Zhenyu Zheng
>>>>>>>>>
>>>>>>>>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
>>>>>>>>> lalamchinnarao13@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks David for the quick reply.
>>>>>>>>>> Yes currently HBase protobuf version mismatch is the issue.
>>>>>>>>>>
>>>>>>>>>> We are investigating any common version of protobuf is there to
>>>>>> support
>>>>>>>>> for
>>>>>>>>>> HBase an HIVE.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati
>>>>>>>>> <dlavati@cloudera.com.invalid
>>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I recently investigated this in
>>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't
>>> look
>>>>>>>>>>> feasible, as protobuf has to be in sync across the whole stack,
>>> but
>>>>>>>> the
>>>>>>>>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x,
>>> see
>>>>>> my
>>>>>>>>>>> comment on the jira and
>>>>>>>>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
>>>>>>>>>>>
>>>>>>>>>>> Kind Regards,
>>>>>>>>>>> David
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
>>>>>>>>>>> lalamchinnarao13@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> We are working on HIVE building and running on ARM
>>> architecture. As
>>>>>>>>>> part
>>>>>>>>>>> of
>>>>>>>>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM
>>> machines.
>>>>>>>>> Which
>>>>>>>>>>>> helps to identity gaps for ARM, If any issues identified we will
>>>>>>>> keep
>>>>>>>>>> on
>>>>>>>>>>>> working to fix those issues.
>>>>>>>>>>>>
>>>>>>>>>>>> Currently to run HIVE on aarch64, we need to upgrade to
>>>>>>>>> protobuf-3.7.1.
>>>>>>>>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939>
>>>>>>>>> upgraded
>>>>>>>>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We
>>> are
>>>>>>>>>>> working
>>>>>>>>>>>> on this.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Chinna Rao Lalam
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Hope It Helps,
>>>>>>>>>> Chinna
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Hope It Helps,
>>>>>>>> Chinna
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Hope It Helps,
>>>>> Chinna
>>>>>
>>>>
>>>
>>
>
Re: HIVE building on ARM
Posted by Chinna Rao Lalam <la...@gmail.com>.
Hi Zoltan,
As you said, migrating this job to the new ci-hadoop instance looks good as
Hadoop also shares the same armN slaves.
I am able to login the new ci-hadoop instance with Apache LDAP credentials,
but i am not able to see the job creation option. Should I request access
or the process for creation of a job is different than jenkin?.
Please guide me to create the new job in the ci-hadoop instance. I will
migrate this job after connecting the armN slaves to the new system.
Thanks
Chinna
On Wed, Jun 17, 2020 at 11:57 AM Zhenyu Zheng <zh...@gmail.com>
wrote:
> Hi Zoltan,
>
> Thanks alot for the information, so looks like one possible solution is as
> you suggest, move the current ARM2 and ARM3 (those two were donate to
> builds.apache.org by us) to the new ci-hadoop cluster and set up the jobs
> just as what has been done in current jenkins.
>
> I will also ask our team member works on other projects to find out what
> the status of other projects is.
>
> BR,
>
> On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
>
>> Hey,
>>
>> There is an effort by the Apache Infra to change the way Jenkins stuff is
>> organized; a couple months ago Gavin wrote an email about it:
>>
>> http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3CCAN0Gg1dOdEPzatJz9bOFE-2veR7QG7H0HMvYjmsLdgjr8_RPew@mail.gmail.com%3E
>> The resources for running these jobs are coming from the H0~H21 slaves
>> which will be migrated to the new jenkins master eventually.
>>
>> >> So please
>> >> suggest a way which direction we can move and can you share some
>> details
>> >> about the new ci-hadoop instance.
>>
>> Since Hadoop testing is also happening on ARM - I think the best would be
>> to also migrate the armN slaves and the Hive arm nightly over to the new
>> ci-hadoop instance.
>>
>> On 6/16/20 8:40 AM, Zhenyu Zheng wrote:
>> > Thanks for the info, I wonder if where does the resource of ci-hadoop
>> and
>> > hive-test-kube come from? Do they include ARM resources?
>>
>> Interesting question; the resources for Hive testing are donated by
>> Cloudera.
>> About the ARM workers I think Chinna could provide more details.
>> ...I've no idea don't know who sponsors the Hxx slaves
>>
>> > Can you provide some more information about how the new hive-test-kube
>> is
>> > running?
>> It's basically a Jenkins instance which is using kubernetes pods to run
>> things.
>> The whole thing is running on a GKE cluster.
>> While I was working on it I collected stuff needed for it in this repo:
>> https://github.com/kgyrtkirk/hive-test-kube/
>> it should be possible to start a new deployment using that stuff
>>
>> cheers,
>> Zoltan
>>
>> >
>> > BR,
>> > Kevin Zheng
>> >
>> > On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam <
>> > lalamchinnarao13@gmail.com> wrote:
>> >
>> >> Hi Zoltan,
>> >>
>> >> Thanks for the update.
>> >>
>> >> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is
>> >> targeting to run hive tests daily on "arm" slaves, it is using 2 arm
>> >> slaves.
>> >> To find any potential issues with "arm" and fix the issues. So please
>> >> suggest a way which direction we can move and can you share some
>> details
>> >> about the new ci-hadoop instance.
>> >>
>> >> Thanks,
>> >> Chinna
>> >>
>> >> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
>> >>
>> >>> Hey all,
>> >>>
>> >>> In an ticket (INFRA-20416) Gavin asked me if we are completely off
>> >>> builds.apache.org - when I went over the jobs I've saw that
>> >>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there
>> >>> once a day.
>> >>>
>> >>> Since builds.apache.org will be shut down in sometime in the future
>> - we
>> >>> should move this job to the new ci-hadoop instance or to
>> hive-test-kube.
>> >>> The key feature of the job is that it runs the test on the "armX"
>> slaves;
>> >>> which are statically configured on b.a.o.
>> >>> Not sure which way to go - but we will have to move in some direction.
>> >>>
>> >>> cheers,
>> >>> Zoltan
>> >>>
>> >>>
>> >>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote:
>> >>>> Hi Chinna,
>> >>>>
>> >>>> Thanks alot for the reply, I uploaded a patch and also a github PR
>> for
>> >>>> https://issues.apache.org/jira/browse/HIVE-21939 .
>> >>>> In the patch, I bumped the protobuf used in standalone-metadata to
>> 2.6.1
>> >>>> and added a new profile, this profile will identify
>> >>>> the hardware architecture and if it is Aarch64, it will override the
>> >>>> protobuf group.id and package to com.github.os72 which
>> >>>> includes ARM support. For X86 platform, Hive will still download the
>> >>>> protobuf packages from org.google repo. I think with
>> >>>> this method, we can keep the influence to existing x86 users to the
>> >>>> minimum. I hope this could be a acceptable short-term
>> >>>> solution.
>> >>>>
>> >>>> I've manually tested on my machine and the github PR travis CI test
>> has
>> >>>> already passed, so the build process is OK, so let's
>> >>>> wait for the full test result from builds.apache.org.
>> >>>>
>> >>>> BR,
>> >>>>
>> >>>> Zhenyu
>> >>>>
>> >>>> On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <
>> >>> lalamchinnarao13@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>>> Hi Zhenyu,
>> >>>>>
>> >>>>> Until HBase dependency resolved, without effecting the existing code
>> >>> on X86
>> >>>>> i suggest create a separate profile with "os72" repo.
>> >>>>>
>> >>>>> Down the line we should have common version for both X86 and ARM.
>> >>>>>
>> >>>>> Hope It Helps,
>> >>>>> Chinna
>> >>>>>
>> >>>>> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <
>> >>> zhengzhenyulixi@gmail.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Hi Chinna, David and others might interested,
>> >>>>>>
>> >>>>>> Thanks for bring this up, we are currently working on improving
>> >>> enabling
>> >>>>>> big-data software on the ARM platform,
>> >>>>>> we have already done fixes and providing CIs to some of the
>> well-know
>> >>>>>> projects like:
>> >>>>>> 1. Hadoop:
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
>> >>>>>> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
>> >>>>>> 3. HBase:
>> >>>>>>
>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
>> >>>>>>
>> >>>>>> And we are now working on projects including Hive, Kudu, etc.
>> >>>>>>
>> >>>>>> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x
>> >>> and
>> >>>>>> break dependency for HBase, there can
>> >>>>>> be some possible short-term plan(or walk-arounds), doing thes can
>> make
>> >>>>> Hive
>> >>>>>> work on ARM without break any
>> >>>>>> dependencies, and then we can interact with Hbase project to see
>> how
>> >>> can
>> >>>>> we
>> >>>>>> both upgrade to 3.x(since this
>> >>>>>> make take some time).
>> >>>>>>
>> >>>>>> Those possible solutions can be:
>> >>>>>> 1. Using pre-patched protobuf 2.5.0 with ARM support
>> >>>>>> from org.openlabtesting repo, some projects(HBase did
>> >>>>>> this: https://github.com/apache/hbase/pull/959, and we will add a
>> >>>>> profile
>> >>>>>> for this, it will detact the host arch and
>> >>>>>> will only use this pre-patched protobuf when the host arch is ARM
>> so
>> >>> that
>> >>>>>> nothing will be affected for existing x86
>> >>>>>> users; Spark and Hadoop used some other package from this repo as
>> they
>> >>>>> have
>> >>>>>> already upgraded protobuf).
>> >>>>>>
>> >>>>>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
>> >>>>>>
>> >>>
>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
>> >>>>>> Hive already used ``protoc-jar-maven-plugin`` from this repo:
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
>> >>>>>> so it is a minor update.
>> >>>>>>
>> >>>>>> I've tested both ways, worked OK on the ARM machine.
>> >>>>>>
>> >>>>>> So my ideas will be that we make this a two-step-work, first step
>> we
>> >>> use
>> >>>>>> one of the short-term plan to enable
>> >>>>>> Hive on ARM, and 2nd Step, we look for the long run to discuss with
>> >>> HBase
>> >>>>>> to find out a fesiable solution.
>> >>>>>>
>> >>>>>> BR,
>> >>>>>>
>> >>>>>> Zhenyu Zheng
>> >>>>>>
>> >>>>>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
>> >>>>>> lalamchinnarao13@gmail.com>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> Thanks David for the quick reply.
>> >>>>>>> Yes currently HBase protobuf version mismatch is the issue.
>> >>>>>>>
>> >>>>>>> We are investigating any common version of protobuf is there to
>> >>> support
>> >>>>>> for
>> >>>>>>> HBase an HIVE.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati
>> >>>>>> <dlavati@cloudera.com.invalid
>> >>>>>>>>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> Hi,
>> >>>>>>>>
>> >>>>>>>> I recently investigated this in
>> >>>>>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't
>> look
>> >>>>>>>> feasible, as protobuf has to be in sync across the whole stack,
>> but
>> >>>>> the
>> >>>>>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x,
>> see
>> >>> my
>> >>>>>>>> comment on the jira and
>> >>>>>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
>> >>>>>>>>
>> >>>>>>>> Kind Regards,
>> >>>>>>>> David
>> >>>>>>>>
>> >>>>>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
>> >>>>>>>> lalamchinnarao13@gmail.com>
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> Hi All,
>> >>>>>>>>>
>> >>>>>>>>> We are working on HIVE building and running on ARM
>> architecture. As
>> >>>>>>> part
>> >>>>>>>> of
>> >>>>>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM
>> machines.
>> >>>>>> Which
>> >>>>>>>>> helps to identity gaps for ARM, If any issues identified we will
>> >>>>> keep
>> >>>>>>> on
>> >>>>>>>>> working to fix those issues.
>> >>>>>>>>>
>> >>>>>>>>> Currently to run HIVE on aarch64, we need to upgrade to
>> >>>>>> protobuf-3.7.1.
>> >>>>>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939>
>> >>>>>> upgraded
>> >>>>>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We
>> are
>> >>>>>>>> working
>> >>>>>>>>> on this.
>> >>>>>>>>>
>> >>>>>>>>> Thanks,
>> >>>>>>>>> Chinna Rao Lalam
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>> Hope It Helps,
>> >>>>>>> Chinna
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Hope It Helps,
>> >>>>> Chinna
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >>
>> >> --
>> >> Hope It Helps,
>> >> Chinna
>> >>
>> >
>>
>
--
Hope It Helps,
Chinna
Re: HIVE building on ARM
Posted by Zhenyu Zheng <zh...@gmail.com>.
Hi Zoltan,
Thanks alot for the information, so looks like one possible solution is as
you suggest, move the current ARM2 and ARM3 (those two were donate to
builds.apache.org by us) to the new ci-hadoop cluster and set up the jobs
just as what has been done in current jenkins.
I will also ask our team member works on other projects to find out what
the status of other projects is.
BR,
On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
> Hey,
>
> There is an effort by the Apache Infra to change the way Jenkins stuff is
> organized; a couple months ago Gavin wrote an email about it:
>
> http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3CCAN0Gg1dOdEPzatJz9bOFE-2veR7QG7H0HMvYjmsLdgjr8_RPew@mail.gmail.com%3E
> The resources for running these jobs are coming from the H0~H21 slaves
> which will be migrated to the new jenkins master eventually.
>
> >> So please
> >> suggest a way which direction we can move and can you share some
> details
> >> about the new ci-hadoop instance.
>
> Since Hadoop testing is also happening on ARM - I think the best would be
> to also migrate the armN slaves and the Hive arm nightly over to the new
> ci-hadoop instance.
>
> On 6/16/20 8:40 AM, Zhenyu Zheng wrote:
> > Thanks for the info, I wonder if where does the resource of ci-hadoop and
> > hive-test-kube come from? Do they include ARM resources?
>
> Interesting question; the resources for Hive testing are donated by
> Cloudera.
> About the ARM workers I think Chinna could provide more details.
> ...I've no idea don't know who sponsors the Hxx slaves
>
> > Can you provide some more information about how the new hive-test-kube is
> > running?
> It's basically a Jenkins instance which is using kubernetes pods to run
> things.
> The whole thing is running on a GKE cluster.
> While I was working on it I collected stuff needed for it in this repo:
> https://github.com/kgyrtkirk/hive-test-kube/
> it should be possible to start a new deployment using that stuff
>
> cheers,
> Zoltan
>
> >
> > BR,
> > Kevin Zheng
> >
> > On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam <
> > lalamchinnarao13@gmail.com> wrote:
> >
> >> Hi Zoltan,
> >>
> >> Thanks for the update.
> >>
> >> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is
> >> targeting to run hive tests daily on "arm" slaves, it is using 2 arm
> >> slaves.
> >> To find any potential issues with "arm" and fix the issues. So please
> >> suggest a way which direction we can move and can you share some details
> >> about the new ci-hadoop instance.
> >>
> >> Thanks,
> >> Chinna
> >>
> >> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
> >>
> >>> Hey all,
> >>>
> >>> In an ticket (INFRA-20416) Gavin asked me if we are completely off
> >>> builds.apache.org - when I went over the jobs I've saw that
> >>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there
> >>> once a day.
> >>>
> >>> Since builds.apache.org will be shut down in sometime in the future -
> we
> >>> should move this job to the new ci-hadoop instance or to
> hive-test-kube.
> >>> The key feature of the job is that it runs the test on the "armX"
> slaves;
> >>> which are statically configured on b.a.o.
> >>> Not sure which way to go - but we will have to move in some direction.
> >>>
> >>> cheers,
> >>> Zoltan
> >>>
> >>>
> >>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote:
> >>>> Hi Chinna,
> >>>>
> >>>> Thanks alot for the reply, I uploaded a patch and also a github PR for
> >>>> https://issues.apache.org/jira/browse/HIVE-21939 .
> >>>> In the patch, I bumped the protobuf used in standalone-metadata to
> 2.6.1
> >>>> and added a new profile, this profile will identify
> >>>> the hardware architecture and if it is Aarch64, it will override the
> >>>> protobuf group.id and package to com.github.os72 which
> >>>> includes ARM support. For X86 platform, Hive will still download the
> >>>> protobuf packages from org.google repo. I think with
> >>>> this method, we can keep the influence to existing x86 users to the
> >>>> minimum. I hope this could be a acceptable short-term
> >>>> solution.
> >>>>
> >>>> I've manually tested on my machine and the github PR travis CI test
> has
> >>>> already passed, so the build process is OK, so let's
> >>>> wait for the full test result from builds.apache.org.
> >>>>
> >>>> BR,
> >>>>
> >>>> Zhenyu
> >>>>
> >>>> On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <
> >>> lalamchinnarao13@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Zhenyu,
> >>>>>
> >>>>> Until HBase dependency resolved, without effecting the existing code
> >>> on X86
> >>>>> i suggest create a separate profile with "os72" repo.
> >>>>>
> >>>>> Down the line we should have common version for both X86 and ARM.
> >>>>>
> >>>>> Hope It Helps,
> >>>>> Chinna
> >>>>>
> >>>>> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <
> >>> zhengzhenyulixi@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi Chinna, David and others might interested,
> >>>>>>
> >>>>>> Thanks for bring this up, we are currently working on improving
> >>> enabling
> >>>>>> big-data software on the ARM platform,
> >>>>>> we have already done fixes and providing CIs to some of the
> well-know
> >>>>>> projects like:
> >>>>>> 1. Hadoop:
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
> >>>>>> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
> >>>>>> 3. HBase:
> >>>>>>
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
> >>>>>>
> >>>>>> And we are now working on projects including Hive, Kudu, etc.
> >>>>>>
> >>>>>> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x
> >>> and
> >>>>>> break dependency for HBase, there can
> >>>>>> be some possible short-term plan(or walk-arounds), doing thes can
> make
> >>>>> Hive
> >>>>>> work on ARM without break any
> >>>>>> dependencies, and then we can interact with Hbase project to see how
> >>> can
> >>>>> we
> >>>>>> both upgrade to 3.x(since this
> >>>>>> make take some time).
> >>>>>>
> >>>>>> Those possible solutions can be:
> >>>>>> 1. Using pre-patched protobuf 2.5.0 with ARM support
> >>>>>> from org.openlabtesting repo, some projects(HBase did
> >>>>>> this: https://github.com/apache/hbase/pull/959, and we will add a
> >>>>> profile
> >>>>>> for this, it will detact the host arch and
> >>>>>> will only use this pre-patched protobuf when the host arch is ARM so
> >>> that
> >>>>>> nothing will be affected for existing x86
> >>>>>> users; Spark and Hadoop used some other package from this repo as
> they
> >>>>> have
> >>>>>> already upgraded protobuf).
> >>>>>>
> >>>>>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
> >>>>>>
> >>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
> >>>>>> Hive already used ``protoc-jar-maven-plugin`` from this repo:
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
> >>>>>> so it is a minor update.
> >>>>>>
> >>>>>> I've tested both ways, worked OK on the ARM machine.
> >>>>>>
> >>>>>> So my ideas will be that we make this a two-step-work, first step we
> >>> use
> >>>>>> one of the short-term plan to enable
> >>>>>> Hive on ARM, and 2nd Step, we look for the long run to discuss with
> >>> HBase
> >>>>>> to find out a fesiable solution.
> >>>>>>
> >>>>>> BR,
> >>>>>>
> >>>>>> Zhenyu Zheng
> >>>>>>
> >>>>>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
> >>>>>> lalamchinnarao13@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks David for the quick reply.
> >>>>>>> Yes currently HBase protobuf version mismatch is the issue.
> >>>>>>>
> >>>>>>> We are investigating any common version of protobuf is there to
> >>> support
> >>>>>> for
> >>>>>>> HBase an HIVE.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati
> >>>>>> <dlavati@cloudera.com.invalid
> >>>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> I recently investigated this in
> >>>>>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't
> look
> >>>>>>>> feasible, as protobuf has to be in sync across the whole stack,
> but
> >>>>> the
> >>>>>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x, see
> >>> my
> >>>>>>>> comment on the jira and
> >>>>>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
> >>>>>>>>
> >>>>>>>> Kind Regards,
> >>>>>>>> David
> >>>>>>>>
> >>>>>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
> >>>>>>>> lalamchinnarao13@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi All,
> >>>>>>>>>
> >>>>>>>>> We are working on HIVE building and running on ARM architecture.
> As
> >>>>>>> part
> >>>>>>>> of
> >>>>>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM machines.
> >>>>>> Which
> >>>>>>>>> helps to identity gaps for ARM, If any issues identified we will
> >>>>> keep
> >>>>>>> on
> >>>>>>>>> working to fix those issues.
> >>>>>>>>>
> >>>>>>>>> Currently to run HIVE on aarch64, we need to upgrade to
> >>>>>> protobuf-3.7.1.
> >>>>>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939>
> >>>>>> upgraded
> >>>>>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We
> are
> >>>>>>>> working
> >>>>>>>>> on this.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Chinna Rao Lalam
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Hope It Helps,
> >>>>>>> Chinna
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Hope It Helps,
> >>>>> Chinna
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> Hope It Helps,
> >> Chinna
> >>
> >
>
Re: HIVE building on ARM
Posted by Zoltan Haindrich <ki...@rxd.hu>.
Hey,
There is an effort by the Apache Infra to change the way Jenkins stuff is organized; a couple months ago Gavin wrote an email about it:
http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3CCAN0Gg1dOdEPzatJz9bOFE-2veR7QG7H0HMvYjmsLdgjr8_RPew@mail.gmail.com%3E
The resources for running these jobs are coming from the H0~H21 slaves which will be migrated to the new jenkins master eventually.
>> So please
>> suggest a way which direction we can move and can you share some details
>> about the new ci-hadoop instance.
Since Hadoop testing is also happening on ARM - I think the best would be to also migrate the armN slaves and the Hive arm nightly over to the new ci-hadoop instance.
On 6/16/20 8:40 AM, Zhenyu Zheng wrote:
> Thanks for the info, I wonder if where does the resource of ci-hadoop and
> hive-test-kube come from? Do they include ARM resources?
Interesting question; the resources for Hive testing are donated by Cloudera.
About the ARM workers I think Chinna could provide more details.
...I've no idea don't know who sponsors the Hxx slaves
> Can you provide some more information about how the new hive-test-kube is
> running?
It's basically a Jenkins instance which is using kubernetes pods to run things.
The whole thing is running on a GKE cluster.
While I was working on it I collected stuff needed for it in this repo:
https://github.com/kgyrtkirk/hive-test-kube/
it should be possible to start a new deployment using that stuff
cheers,
Zoltan
>
> BR,
> Kevin Zheng
>
> On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam <
> lalamchinnarao13@gmail.com> wrote:
>
>> Hi Zoltan,
>>
>> Thanks for the update.
>>
>> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is
>> targeting to run hive tests daily on "arm" slaves, it is using 2 arm
>> slaves.
>> To find any potential issues with "arm" and fix the issues. So please
>> suggest a way which direction we can move and can you share some details
>> about the new ci-hadoop instance.
>>
>> Thanks,
>> Chinna
>>
>> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
>>
>>> Hey all,
>>>
>>> In an ticket (INFRA-20416) Gavin asked me if we are completely off
>>> builds.apache.org - when I went over the jobs I've saw that
>>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there
>>> once a day.
>>>
>>> Since builds.apache.org will be shut down in sometime in the future - we
>>> should move this job to the new ci-hadoop instance or to hive-test-kube.
>>> The key feature of the job is that it runs the test on the "armX" slaves;
>>> which are statically configured on b.a.o.
>>> Not sure which way to go - but we will have to move in some direction.
>>>
>>> cheers,
>>> Zoltan
>>>
>>>
>>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote:
>>>> Hi Chinna,
>>>>
>>>> Thanks alot for the reply, I uploaded a patch and also a github PR for
>>>> https://issues.apache.org/jira/browse/HIVE-21939 .
>>>> In the patch, I bumped the protobuf used in standalone-metadata to 2.6.1
>>>> and added a new profile, this profile will identify
>>>> the hardware architecture and if it is Aarch64, it will override the
>>>> protobuf group.id and package to com.github.os72 which
>>>> includes ARM support. For X86 platform, Hive will still download the
>>>> protobuf packages from org.google repo. I think with
>>>> this method, we can keep the influence to existing x86 users to the
>>>> minimum. I hope this could be a acceptable short-term
>>>> solution.
>>>>
>>>> I've manually tested on my machine and the github PR travis CI test has
>>>> already passed, so the build process is OK, so let's
>>>> wait for the full test result from builds.apache.org.
>>>>
>>>> BR,
>>>>
>>>> Zhenyu
>>>>
>>>> On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <
>>> lalamchinnarao13@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Zhenyu,
>>>>>
>>>>> Until HBase dependency resolved, without effecting the existing code
>>> on X86
>>>>> i suggest create a separate profile with "os72" repo.
>>>>>
>>>>> Down the line we should have common version for both X86 and ARM.
>>>>>
>>>>> Hope It Helps,
>>>>> Chinna
>>>>>
>>>>> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <
>>> zhengzhenyulixi@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Chinna, David and others might interested,
>>>>>>
>>>>>> Thanks for bring this up, we are currently working on improving
>>> enabling
>>>>>> big-data software on the ARM platform,
>>>>>> we have already done fixes and providing CIs to some of the well-know
>>>>>> projects like:
>>>>>> 1. Hadoop:
>>>>>>
>>>>>>
>>>>>
>>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
>>>>>> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
>>>>>> 3. HBase:
>>>>>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
>>>>>>
>>>>>> And we are now working on projects including Hive, Kudu, etc.
>>>>>>
>>>>>> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x
>>> and
>>>>>> break dependency for HBase, there can
>>>>>> be some possible short-term plan(or walk-arounds), doing thes can make
>>>>> Hive
>>>>>> work on ARM without break any
>>>>>> dependencies, and then we can interact with Hbase project to see how
>>> can
>>>>> we
>>>>>> both upgrade to 3.x(since this
>>>>>> make take some time).
>>>>>>
>>>>>> Those possible solutions can be:
>>>>>> 1. Using pre-patched protobuf 2.5.0 with ARM support
>>>>>> from org.openlabtesting repo, some projects(HBase did
>>>>>> this: https://github.com/apache/hbase/pull/959, and we will add a
>>>>> profile
>>>>>> for this, it will detact the host arch and
>>>>>> will only use this pre-patched protobuf when the host arch is ARM so
>>> that
>>>>>> nothing will be affected for existing x86
>>>>>> users; Spark and Hadoop used some other package from this repo as they
>>>>> have
>>>>>> already upgraded protobuf).
>>>>>>
>>>>>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
>>>>>>
>>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
>>>>>> Hive already used ``protoc-jar-maven-plugin`` from this repo:
>>>>>>
>>>>>>
>>>>>
>>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
>>>>>> so it is a minor update.
>>>>>>
>>>>>> I've tested both ways, worked OK on the ARM machine.
>>>>>>
>>>>>> So my ideas will be that we make this a two-step-work, first step we
>>> use
>>>>>> one of the short-term plan to enable
>>>>>> Hive on ARM, and 2nd Step, we look for the long run to discuss with
>>> HBase
>>>>>> to find out a fesiable solution.
>>>>>>
>>>>>> BR,
>>>>>>
>>>>>> Zhenyu Zheng
>>>>>>
>>>>>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
>>>>>> lalamchinnarao13@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks David for the quick reply.
>>>>>>> Yes currently HBase protobuf version mismatch is the issue.
>>>>>>>
>>>>>>> We are investigating any common version of protobuf is there to
>>> support
>>>>>> for
>>>>>>> HBase an HIVE.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati
>>>>>> <dlavati@cloudera.com.invalid
>>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I recently investigated this in
>>>>>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look
>>>>>>>> feasible, as protobuf has to be in sync across the whole stack, but
>>>>> the
>>>>>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x, see
>>> my
>>>>>>>> comment on the jira and
>>>>>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
>>>>>>>>
>>>>>>>> Kind Regards,
>>>>>>>> David
>>>>>>>>
>>>>>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
>>>>>>>> lalamchinnarao13@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> We are working on HIVE building and running on ARM architecture. As
>>>>>>> part
>>>>>>>> of
>>>>>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM machines.
>>>>>> Which
>>>>>>>>> helps to identity gaps for ARM, If any issues identified we will
>>>>> keep
>>>>>>> on
>>>>>>>>> working to fix those issues.
>>>>>>>>>
>>>>>>>>> Currently to run HIVE on aarch64, we need to upgrade to
>>>>>> protobuf-3.7.1.
>>>>>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939>
>>>>>> upgraded
>>>>>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are
>>>>>>>> working
>>>>>>>>> on this.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Chinna Rao Lalam
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Hope It Helps,
>>>>>>> Chinna
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Hope It Helps,
>>>>> Chinna
>>>>>
>>>>
>>>
>>
>>
>> --
>> Hope It Helps,
>> Chinna
>>
>
Re: HIVE building on ARM
Posted by Zhenyu Zheng <zh...@gmail.com>.
Hi Zoltan, Chinna,
Thanks for the info, I wonder if where does the resource of ci-hadoop and
hive-test-kube come from? Do they include ARM resources?
Can you provide some more information about how the new hive-test-kube is
running?
BR,
Kevin Zheng
On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam <
lalamchinnarao13@gmail.com> wrote:
> Hi Zoltan,
>
> Thanks for the update.
>
> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is
> targeting to run hive tests daily on "arm" slaves, it is using 2 arm
> slaves.
> To find any potential issues with "arm" and fix the issues. So please
> suggest a way which direction we can move and can you share some details
> about the new ci-hadoop instance.
>
> Thanks,
> Chinna
>
> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
>
>> Hey all,
>>
>> In an ticket (INFRA-20416) Gavin asked me if we are completely off
>> builds.apache.org - when I went over the jobs I've saw that
>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there
>> once a day.
>>
>> Since builds.apache.org will be shut down in sometime in the future - we
>> should move this job to the new ci-hadoop instance or to hive-test-kube.
>> The key feature of the job is that it runs the test on the "armX" slaves;
>> which are statically configured on b.a.o.
>> Not sure which way to go - but we will have to move in some direction.
>>
>> cheers,
>> Zoltan
>>
>>
>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote:
>> > Hi Chinna,
>> >
>> > Thanks alot for the reply, I uploaded a patch and also a github PR for
>> > https://issues.apache.org/jira/browse/HIVE-21939 .
>> > In the patch, I bumped the protobuf used in standalone-metadata to 2.6.1
>> > and added a new profile, this profile will identify
>> > the hardware architecture and if it is Aarch64, it will override the
>> > protobuf group.id and package to com.github.os72 which
>> > includes ARM support. For X86 platform, Hive will still download the
>> > protobuf packages from org.google repo. I think with
>> > this method, we can keep the influence to existing x86 users to the
>> > minimum. I hope this could be a acceptable short-term
>> > solution.
>> >
>> > I've manually tested on my machine and the github PR travis CI test has
>> > already passed, so the build process is OK, so let's
>> > wait for the full test result from builds.apache.org.
>> >
>> > BR,
>> >
>> > Zhenyu
>> >
>> > On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <
>> lalamchinnarao13@gmail.com>
>> > wrote:
>> >
>> >> Hi Zhenyu,
>> >>
>> >> Until HBase dependency resolved, without effecting the existing code
>> on X86
>> >> i suggest create a separate profile with "os72" repo.
>> >>
>> >> Down the line we should have common version for both X86 and ARM.
>> >>
>> >> Hope It Helps,
>> >> Chinna
>> >>
>> >> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <
>> zhengzhenyulixi@gmail.com>
>> >> wrote:
>> >>
>> >>> Hi Chinna, David and others might interested,
>> >>>
>> >>> Thanks for bring this up, we are currently working on improving
>> enabling
>> >>> big-data software on the ARM platform,
>> >>> we have already done fixes and providing CIs to some of the well-know
>> >>> projects like:
>> >>> 1. Hadoop:
>> >>>
>> >>>
>> >>
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
>> >>> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
>> >>> 3. HBase:
>> >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
>> >>>
>> >>> And we are now working on projects including Hive, Kudu, etc.
>> >>>
>> >>> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x
>> and
>> >>> break dependency for HBase, there can
>> >>> be some possible short-term plan(or walk-arounds), doing thes can make
>> >> Hive
>> >>> work on ARM without break any
>> >>> dependencies, and then we can interact with Hbase project to see how
>> can
>> >> we
>> >>> both upgrade to 3.x(since this
>> >>> make take some time).
>> >>>
>> >>> Those possible solutions can be:
>> >>> 1. Using pre-patched protobuf 2.5.0 with ARM support
>> >>> from org.openlabtesting repo, some projects(HBase did
>> >>> this: https://github.com/apache/hbase/pull/959, and we will add a
>> >> profile
>> >>> for this, it will detact the host arch and
>> >>> will only use this pre-patched protobuf when the host arch is ARM so
>> that
>> >>> nothing will be affected for existing x86
>> >>> users; Spark and Hadoop used some other package from this repo as they
>> >> have
>> >>> already upgraded protobuf).
>> >>>
>> >>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
>> >>>
>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
>> >>> Hive already used ``protoc-jar-maven-plugin`` from this repo:
>> >>>
>> >>>
>> >>
>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
>> >>>
>> >>>
>> >>>
>> >>
>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
>> >>> so it is a minor update.
>> >>>
>> >>> I've tested both ways, worked OK on the ARM machine.
>> >>>
>> >>> So my ideas will be that we make this a two-step-work, first step we
>> use
>> >>> one of the short-term plan to enable
>> >>> Hive on ARM, and 2nd Step, we look for the long run to discuss with
>> HBase
>> >>> to find out a fesiable solution.
>> >>>
>> >>> BR,
>> >>>
>> >>> Zhenyu Zheng
>> >>>
>> >>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
>> >>> lalamchinnarao13@gmail.com>
>> >>> wrote:
>> >>>
>> >>>> Thanks David for the quick reply.
>> >>>> Yes currently HBase protobuf version mismatch is the issue.
>> >>>>
>> >>>> We are investigating any common version of protobuf is there to
>> support
>> >>> for
>> >>>> HBase an HIVE.
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati
>> >>> <dlavati@cloudera.com.invalid
>> >>>>>
>> >>>> wrote:
>> >>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> I recently investigated this in
>> >>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look
>> >>>>> feasible, as protobuf has to be in sync across the whole stack, but
>> >> the
>> >>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x, see
>> my
>> >>>>> comment on the jira and
>> >>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
>> >>>>>
>> >>>>> Kind Regards,
>> >>>>> David
>> >>>>>
>> >>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
>> >>>>> lalamchinnarao13@gmail.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Hi All,
>> >>>>>>
>> >>>>>> We are working on HIVE building and running on ARM architecture. As
>> >>>> part
>> >>>>> of
>> >>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM machines.
>> >>> Which
>> >>>>>> helps to identity gaps for ARM, If any issues identified we will
>> >> keep
>> >>>> on
>> >>>>>> working to fix those issues.
>> >>>>>>
>> >>>>>> Currently to run HIVE on aarch64, we need to upgrade to
>> >>> protobuf-3.7.1.
>> >>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939>
>> >>> upgraded
>> >>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are
>> >>>>> working
>> >>>>>> on this.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Chinna Rao Lalam
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Hope It Helps,
>> >>>> Chinna
>> >>>>
>> >>>
>> >>
>> >>
>> >> --
>> >> Hope It Helps,
>> >> Chinna
>> >>
>> >
>>
>
>
> --
> Hope It Helps,
> Chinna
>
Re: HIVE building on ARM
Posted by Chinna Rao Lalam <la...@gmail.com>.
Hi Zoltan,
Thanks for the update.
Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is
targeting to run hive tests daily on "arm" slaves, it is using 2 arm
slaves.
To find any potential issues with "arm" and fix the issues. So please
suggest a way which direction we can move and can you share some details
about the new ci-hadoop instance.
Thanks,
Chinna
On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
> Hey all,
>
> In an ticket (INFRA-20416) Gavin asked me if we are completely off
> builds.apache.org - when I went over the jobs I've saw that
> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there once
> a day.
>
> Since builds.apache.org will be shut down in sometime in the future - we
> should move this job to the new ci-hadoop instance or to hive-test-kube.
> The key feature of the job is that it runs the test on the "armX" slaves;
> which are statically configured on b.a.o.
> Not sure which way to go - but we will have to move in some direction.
>
> cheers,
> Zoltan
>
>
> On 3/13/20 7:22 AM, Zhenyu Zheng wrote:
> > Hi Chinna,
> >
> > Thanks alot for the reply, I uploaded a patch and also a github PR for
> > https://issues.apache.org/jira/browse/HIVE-21939 .
> > In the patch, I bumped the protobuf used in standalone-metadata to 2.6.1
> > and added a new profile, this profile will identify
> > the hardware architecture and if it is Aarch64, it will override the
> > protobuf group.id and package to com.github.os72 which
> > includes ARM support. For X86 platform, Hive will still download the
> > protobuf packages from org.google repo. I think with
> > this method, we can keep the influence to existing x86 users to the
> > minimum. I hope this could be a acceptable short-term
> > solution.
> >
> > I've manually tested on my machine and the github PR travis CI test has
> > already passed, so the build process is OK, so let's
> > wait for the full test result from builds.apache.org.
> >
> > BR,
> >
> > Zhenyu
> >
> > On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <
> lalamchinnarao13@gmail.com>
> > wrote:
> >
> >> Hi Zhenyu,
> >>
> >> Until HBase dependency resolved, without effecting the existing code on
> X86
> >> i suggest create a separate profile with "os72" repo.
> >>
> >> Down the line we should have common version for both X86 and ARM.
> >>
> >> Hope It Helps,
> >> Chinna
> >>
> >> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <zhengzhenyulixi@gmail.com
> >
> >> wrote:
> >>
> >>> Hi Chinna, David and others might interested,
> >>>
> >>> Thanks for bring this up, we are currently working on improving
> enabling
> >>> big-data software on the ARM platform,
> >>> we have already done fixes and providing CIs to some of the well-know
> >>> projects like:
> >>> 1. Hadoop:
> >>>
> >>>
> >>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
> >>> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
> >>> 3. HBase:
> >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
> >>>
> >>> And we are now working on projects including Hive, Kudu, etc.
> >>>
> >>> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x and
> >>> break dependency for HBase, there can
> >>> be some possible short-term plan(or walk-arounds), doing thes can make
> >> Hive
> >>> work on ARM without break any
> >>> dependencies, and then we can interact with Hbase project to see how
> can
> >> we
> >>> both upgrade to 3.x(since this
> >>> make take some time).
> >>>
> >>> Those possible solutions can be:
> >>> 1. Using pre-patched protobuf 2.5.0 with ARM support
> >>> from org.openlabtesting repo, some projects(HBase did
> >>> this: https://github.com/apache/hbase/pull/959, and we will add a
> >> profile
> >>> for this, it will detact the host arch and
> >>> will only use this pre-patched protobuf when the host arch is ARM so
> that
> >>> nothing will be affected for existing x86
> >>> users; Spark and Hadoop used some other package from this repo as they
> >> have
> >>> already upgraded protobuf).
> >>>
> >>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
> >>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
> >>> Hive already used ``protoc-jar-maven-plugin`` from this repo:
> >>>
> >>>
> >>
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
> >>>
> >>>
> >>>
> >>
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
> >>> so it is a minor update.
> >>>
> >>> I've tested both ways, worked OK on the ARM machine.
> >>>
> >>> So my ideas will be that we make this a two-step-work, first step we
> use
> >>> one of the short-term plan to enable
> >>> Hive on ARM, and 2nd Step, we look for the long run to discuss with
> HBase
> >>> to find out a fesiable solution.
> >>>
> >>> BR,
> >>>
> >>> Zhenyu Zheng
> >>>
> >>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
> >>> lalamchinnarao13@gmail.com>
> >>> wrote:
> >>>
> >>>> Thanks David for the quick reply.
> >>>> Yes currently HBase protobuf version mismatch is the issue.
> >>>>
> >>>> We are investigating any common version of protobuf is there to
> support
> >>> for
> >>>> HBase an HIVE.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati
> >>> <dlavati@cloudera.com.invalid
> >>>>>
> >>>> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> I recently investigated this in
> >>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look
> >>>>> feasible, as protobuf has to be in sync across the whole stack, but
> >> the
> >>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my
> >>>>> comment on the jira and
> >>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
> >>>>>
> >>>>> Kind Regards,
> >>>>> David
> >>>>>
> >>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
> >>>>> lalamchinnarao13@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi All,
> >>>>>>
> >>>>>> We are working on HIVE building and running on ARM architecture. As
> >>>> part
> >>>>> of
> >>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM machines.
> >>> Which
> >>>>>> helps to identity gaps for ARM, If any issues identified we will
> >> keep
> >>>> on
> >>>>>> working to fix those issues.
> >>>>>>
> >>>>>> Currently to run HIVE on aarch64, we need to upgrade to
> >>> protobuf-3.7.1.
> >>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939>
> >>> upgraded
> >>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are
> >>>>> working
> >>>>>> on this.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Chinna Rao Lalam
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Hope It Helps,
> >>>> Chinna
> >>>>
> >>>
> >>
> >>
> >> --
> >> Hope It Helps,
> >> Chinna
> >>
> >
>
--
Hope It Helps,
Chinna
Re: HIVE building on ARM
Posted by Zoltan Haindrich <ki...@rxd.hu>.
Hey all,
In an ticket (INFRA-20416) Gavin asked me if we are completely off builds.apache.org - when I went over the jobs I've saw that
https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there once a day.
Since builds.apache.org will be shut down in sometime in the future - we should move this job to the new ci-hadoop instance or to hive-test-kube.
The key feature of the job is that it runs the test on the "armX" slaves; which are statically configured on b.a.o.
Not sure which way to go - but we will have to move in some direction.
cheers,
Zoltan
On 3/13/20 7:22 AM, Zhenyu Zheng wrote:
> Hi Chinna,
>
> Thanks alot for the reply, I uploaded a patch and also a github PR for
> https://issues.apache.org/jira/browse/HIVE-21939 .
> In the patch, I bumped the protobuf used in standalone-metadata to 2.6.1
> and added a new profile, this profile will identify
> the hardware architecture and if it is Aarch64, it will override the
> protobuf group.id and package to com.github.os72 which
> includes ARM support. For X86 platform, Hive will still download the
> protobuf packages from org.google repo. I think with
> this method, we can keep the influence to existing x86 users to the
> minimum. I hope this could be a acceptable short-term
> solution.
>
> I've manually tested on my machine and the github PR travis CI test has
> already passed, so the build process is OK, so let's
> wait for the full test result from builds.apache.org.
>
> BR,
>
> Zhenyu
>
> On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <la...@gmail.com>
> wrote:
>
>> Hi Zhenyu,
>>
>> Until HBase dependency resolved, without effecting the existing code on X86
>> i suggest create a separate profile with "os72" repo.
>>
>> Down the line we should have common version for both X86 and ARM.
>>
>> Hope It Helps,
>> Chinna
>>
>> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <zh...@gmail.com>
>> wrote:
>>
>>> Hi Chinna, David and others might interested,
>>>
>>> Thanks for bring this up, we are currently working on improving enabling
>>> big-data software on the ARM platform,
>>> we have already done fixes and providing CIs to some of the well-know
>>> projects like:
>>> 1. Hadoop:
>>>
>>>
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
>>> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
>>> 3. HBase:
>>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
>>>
>>> And we are now working on projects including Hive, Kudu, etc.
>>>
>>> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x and
>>> break dependency for HBase, there can
>>> be some possible short-term plan(or walk-arounds), doing thes can make
>> Hive
>>> work on ARM without break any
>>> dependencies, and then we can interact with Hbase project to see how can
>> we
>>> both upgrade to 3.x(since this
>>> make take some time).
>>>
>>> Those possible solutions can be:
>>> 1. Using pre-patched protobuf 2.5.0 with ARM support
>>> from org.openlabtesting repo, some projects(HBase did
>>> this: https://github.com/apache/hbase/pull/959, and we will add a
>> profile
>>> for this, it will detact the host arch and
>>> will only use this pre-patched protobuf when the host arch is ARM so that
>>> nothing will be affected for existing x86
>>> users; Spark and Hadoop used some other package from this repo as they
>> have
>>> already upgraded protobuf).
>>>
>>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
>>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
>>> Hive already used ``protoc-jar-maven-plugin`` from this repo:
>>>
>>>
>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
>>>
>>>
>>>
>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
>>> so it is a minor update.
>>>
>>> I've tested both ways, worked OK on the ARM machine.
>>>
>>> So my ideas will be that we make this a two-step-work, first step we use
>>> one of the short-term plan to enable
>>> Hive on ARM, and 2nd Step, we look for the long run to discuss with HBase
>>> to find out a fesiable solution.
>>>
>>> BR,
>>>
>>> Zhenyu Zheng
>>>
>>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
>>> lalamchinnarao13@gmail.com>
>>> wrote:
>>>
>>>> Thanks David for the quick reply.
>>>> Yes currently HBase protobuf version mismatch is the issue.
>>>>
>>>> We are investigating any common version of protobuf is there to support
>>> for
>>>> HBase an HIVE.
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati
>>> <dlavati@cloudera.com.invalid
>>>>>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I recently investigated this in
>>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look
>>>>> feasible, as protobuf has to be in sync across the whole stack, but
>> the
>>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my
>>>>> comment on the jira and
>>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
>>>>>
>>>>> Kind Regards,
>>>>> David
>>>>>
>>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
>>>>> lalamchinnarao13@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> We are working on HIVE building and running on ARM architecture. As
>>>> part
>>>>> of
>>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM machines.
>>> Which
>>>>>> helps to identity gaps for ARM, If any issues identified we will
>> keep
>>>> on
>>>>>> working to fix those issues.
>>>>>>
>>>>>> Currently to run HIVE on aarch64, we need to upgrade to
>>> protobuf-3.7.1.
>>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939>
>>> upgraded
>>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are
>>>>> working
>>>>>> on this.
>>>>>>
>>>>>> Thanks,
>>>>>> Chinna Rao Lalam
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Hope It Helps,
>>>> Chinna
>>>>
>>>
>>
>>
>> --
>> Hope It Helps,
>> Chinna
>>
>
Re: HIVE building on ARM
Posted by Zhenyu Zheng <zh...@gmail.com>.
Hi Chinna,
Thanks alot for the reply, I uploaded a patch and also a github PR for
https://issues.apache.org/jira/browse/HIVE-21939 .
In the patch, I bumped the protobuf used in standalone-metadata to 2.6.1
and added a new profile, this profile will identify
the hardware architecture and if it is Aarch64, it will override the
protobuf group.id and package to com.github.os72 which
includes ARM support. For X86 platform, Hive will still download the
protobuf packages from org.google repo. I think with
this method, we can keep the influence to existing x86 users to the
minimum. I hope this could be a acceptable short-term
solution.
I've manually tested on my machine and the github PR travis CI test has
already passed, so the build process is OK, so let's
wait for the full test result from builds.apache.org.
BR,
Zhenyu
On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <la...@gmail.com>
wrote:
> Hi Zhenyu,
>
> Until HBase dependency resolved, without effecting the existing code on X86
> i suggest create a separate profile with "os72" repo.
>
> Down the line we should have common version for both X86 and ARM.
>
> Hope It Helps,
> Chinna
>
> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <zh...@gmail.com>
> wrote:
>
> > Hi Chinna, David and others might interested,
> >
> > Thanks for bring this up, we are currently working on improving enabling
> > big-data software on the ARM platform,
> > we have already done fixes and providing CIs to some of the well-know
> > projects like:
> > 1. Hadoop:
> >
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
> > 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
> > 3. HBase:
> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
> >
> > And we are now working on projects including Hive, Kudu, etc.
> >
> > Regarding to the protobuf upgrades in Hive, except upgrading to 3.x and
> > break dependency for HBase, there can
> > be some possible short-term plan(or walk-arounds), doing thes can make
> Hive
> > work on ARM without break any
> > dependencies, and then we can interact with Hbase project to see how can
> we
> > both upgrade to 3.x(since this
> > make take some time).
> >
> > Those possible solutions can be:
> > 1. Using pre-patched protobuf 2.5.0 with ARM support
> > from org.openlabtesting repo, some projects(HBase did
> > this: https://github.com/apache/hbase/pull/959, and we will add a
> profile
> > for this, it will detact the host arch and
> > will only use this pre-patched protobuf when the host arch is ARM so that
> > nothing will be affected for existing x86
> > users; Spark and Hadoop used some other package from this repo as they
> have
> > already upgraded protobuf).
> >
> > 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
> > https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
> > Hive already used ``protoc-jar-maven-plugin`` from this repo:
> >
> >
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
> >
> >
> >
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
> > so it is a minor update.
> >
> > I've tested both ways, worked OK on the ARM machine.
> >
> > So my ideas will be that we make this a two-step-work, first step we use
> > one of the short-term plan to enable
> > Hive on ARM, and 2nd Step, we look for the long run to discuss with HBase
> > to find out a fesiable solution.
> >
> > BR,
> >
> > Zhenyu Zheng
> >
> > On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
> > lalamchinnarao13@gmail.com>
> > wrote:
> >
> > > Thanks David for the quick reply.
> > > Yes currently HBase protobuf version mismatch is the issue.
> > >
> > > We are investigating any common version of protobuf is there to support
> > for
> > > HBase an HIVE.
> > >
> > >
> > >
> > >
> > > On Tue, Mar 10, 2020 at 6:08 PM David Lavati
> > <dlavati@cloudera.com.invalid
> > > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I recently investigated this in
> > > > https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look
> > > > feasible, as protobuf has to be in sync across the whole stack, but
> the
> > > > external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my
> > > > comment on the jira and
> > > > https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
> > > >
> > > > Kind Regards,
> > > > David
> > > >
> > > > On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
> > > > lalamchinnarao13@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > We are working on HIVE building and running on ARM architecture. As
> > > part
> > > > of
> > > > > this i want to add one periodic HIVE JENKINS JOB on ARM machines.
> > Which
> > > > > helps to identity gaps for ARM, If any issues identified we will
> keep
> > > on
> > > > > working to fix those issues.
> > > > >
> > > > > Currently to run HIVE on aarch64, we need to upgrade to
> > protobuf-3.7.1.
> > > > > HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939>
> > upgraded
> > > > > protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are
> > > > working
> > > > > on this.
> > > > >
> > > > > Thanks,
> > > > > Chinna Rao Lalam
> > > > >
> > > >
> > >
> > >
> > > --
> > > Hope It Helps,
> > > Chinna
> > >
> >
>
>
> --
> Hope It Helps,
> Chinna
>
Re: HIVE building on ARM
Posted by Chinna Rao Lalam <la...@gmail.com>.
Hi Zhenyu,
Until HBase dependency resolved, without effecting the existing code on X86
i suggest create a separate profile with "os72" repo.
Down the line we should have common version for both X86 and ARM.
Hope It Helps,
Chinna
On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <zh...@gmail.com>
wrote:
> Hi Chinna, David and others might interested,
>
> Thanks for bring this up, we are currently working on improving enabling
> big-data software on the ARM platform,
> we have already done fixes and providing CIs to some of the well-know
> projects like:
> 1. Hadoop:
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
> 3. HBase:
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
>
> And we are now working on projects including Hive, Kudu, etc.
>
> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x and
> break dependency for HBase, there can
> be some possible short-term plan(or walk-arounds), doing thes can make Hive
> work on ARM without break any
> dependencies, and then we can interact with Hbase project to see how can we
> both upgrade to 3.x(since this
> make take some time).
>
> Those possible solutions can be:
> 1. Using pre-patched protobuf 2.5.0 with ARM support
> from org.openlabtesting repo, some projects(HBase did
> this: https://github.com/apache/hbase/pull/959, and we will add a profile
> for this, it will detact the host arch and
> will only use this pre-patched protobuf when the host arch is ARM so that
> nothing will be affected for existing x86
> users; Spark and Hadoop used some other package from this repo as they have
> already upgraded protobuf).
>
> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
> Hive already used ``protoc-jar-maven-plugin`` from this repo:
>
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
>
>
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
> so it is a minor update.
>
> I've tested both ways, worked OK on the ARM machine.
>
> So my ideas will be that we make this a two-step-work, first step we use
> one of the short-term plan to enable
> Hive on ARM, and 2nd Step, we look for the long run to discuss with HBase
> to find out a fesiable solution.
>
> BR,
>
> Zhenyu Zheng
>
> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
> lalamchinnarao13@gmail.com>
> wrote:
>
> > Thanks David for the quick reply.
> > Yes currently HBase protobuf version mismatch is the issue.
> >
> > We are investigating any common version of protobuf is there to support
> for
> > HBase an HIVE.
> >
> >
> >
> >
> > On Tue, Mar 10, 2020 at 6:08 PM David Lavati
> <dlavati@cloudera.com.invalid
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I recently investigated this in
> > > https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look
> > > feasible, as protobuf has to be in sync across the whole stack, but the
> > > external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my
> > > comment on the jira and
> > > https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
> > >
> > > Kind Regards,
> > > David
> > >
> > > On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
> > > lalamchinnarao13@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > We are working on HIVE building and running on ARM architecture. As
> > part
> > > of
> > > > this i want to add one periodic HIVE JENKINS JOB on ARM machines.
> Which
> > > > helps to identity gaps for ARM, If any issues identified we will keep
> > on
> > > > working to fix those issues.
> > > >
> > > > Currently to run HIVE on aarch64, we need to upgrade to
> protobuf-3.7.1.
> > > > HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939>
> upgraded
> > > > protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are
> > > working
> > > > on this.
> > > >
> > > > Thanks,
> > > > Chinna Rao Lalam
> > > >
> > >
> >
> >
> > --
> > Hope It Helps,
> > Chinna
> >
>
--
Hope It Helps,
Chinna
Re: HIVE building on ARM
Posted by Chinna Rao Lalam <la...@gmail.com>.
Hi Zhenyu,
Until HBase dependency resolved, without effecting the existing code on X86
i suggest create a separate profile with "os72" repo.
Down the line we should have common version for both X86 and ARM.
Hope It Helps,
Chinna
On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <zh...@gmail.com>
wrote:
> Hi Chinna, David and others might interested,
>
> Thanks for bring this up, we are currently working on improving enabling
> big-data software on the ARM platform,
> we have already done fixes and providing CIs to some of the well-know
> projects like:
> 1. Hadoop:
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
> 3. HBase:
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
>
> And we are now working on projects including Hive, Kudu, etc.
>
> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x and
> break dependency for HBase, there can
> be some possible short-term plan(or walk-arounds), doing thes can make Hive
> work on ARM without break any
> dependencies, and then we can interact with Hbase project to see how can we
> both upgrade to 3.x(since this
> make take some time).
>
> Those possible solutions can be:
> 1. Using pre-patched protobuf 2.5.0 with ARM support
> from org.openlabtesting repo, some projects(HBase did
> this: https://github.com/apache/hbase/pull/959, and we will add a profile
> for this, it will detact the host arch and
> will only use this pre-patched protobuf when the host arch is ARM so that
> nothing will be affected for existing x86
> users; Spark and Hadoop used some other package from this repo as they have
> already upgraded protobuf).
>
> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
> Hive already used ``protoc-jar-maven-plugin`` from this repo:
>
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
>
>
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
> so it is a minor update.
>
> I've tested both ways, worked OK on the ARM machine.
>
> So my ideas will be that we make this a two-step-work, first step we use
> one of the short-term plan to enable
> Hive on ARM, and 2nd Step, we look for the long run to discuss with HBase
> to find out a fesiable solution.
>
> BR,
>
> Zhenyu Zheng
>
> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
> lalamchinnarao13@gmail.com>
> wrote:
>
> > Thanks David for the quick reply.
> > Yes currently HBase protobuf version mismatch is the issue.
> >
> > We are investigating any common version of protobuf is there to support
> for
> > HBase an HIVE.
> >
> >
> >
> >
> > On Tue, Mar 10, 2020 at 6:08 PM David Lavati
> <dlavati@cloudera.com.invalid
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I recently investigated this in
> > > https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look
> > > feasible, as protobuf has to be in sync across the whole stack, but the
> > > external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my
> > > comment on the jira and
> > > https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
> > >
> > > Kind Regards,
> > > David
> > >
> > > On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
> > > lalamchinnarao13@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > We are working on HIVE building and running on ARM architecture. As
> > part
> > > of
> > > > this i want to add one periodic HIVE JENKINS JOB on ARM machines.
> Which
> > > > helps to identity gaps for ARM, If any issues identified we will keep
> > on
> > > > working to fix those issues.
> > > >
> > > > Currently to run HIVE on aarch64, we need to upgrade to
> protobuf-3.7.1.
> > > > HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939>
> upgraded
> > > > protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are
> > > working
> > > > on this.
> > > >
> > > > Thanks,
> > > > Chinna Rao Lalam
> > > >
> > >
> >
> >
> > --
> > Hope It Helps,
> > Chinna
> >
>
--
Hope It Helps,
Chinna
Re: HIVE building on ARM
Posted by Zhenyu Zheng <zh...@gmail.com>.
Hi Chinna, David and others might interested,
Thanks for bring this up, we are currently working on improving enabling
big-data software on the ARM platform,
we have already done fixes and providing CIs to some of the well-know
projects like:
1. Hadoop:
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
3. HBase:
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
And we are now working on projects including Hive, Kudu, etc.
Regarding to the protobuf upgrades in Hive, except upgrading to 3.x and
break dependency for HBase, there can
be some possible short-term plan(or walk-arounds), doing thes can make Hive
work on ARM without break any
dependencies, and then we can interact with Hbase project to see how can we
both upgrade to 3.x(since this
make take some time).
Those possible solutions can be:
1. Using pre-patched protobuf 2.5.0 with ARM support
from org.openlabtesting repo, some projects(HBase did
this: https://github.com/apache/hbase/pull/959, and we will add a profile
for this, it will detact the host arch and
will only use this pre-patched protobuf when the host arch is ARM so that
nothing will be affected for existing x86
users; Spark and Hadoop used some other package from this repo as they have
already upgraded protobuf).
2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
Hive already used ``protoc-jar-maven-plugin`` from this repo:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
so it is a minor update.
I've tested both ways, worked OK on the ARM machine.
So my ideas will be that we make this a two-step-work, first step we use
one of the short-term plan to enable
Hive on ARM, and 2nd Step, we look for the long run to discuss with HBase
to find out a fesiable solution.
BR,
Zhenyu Zheng
On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <la...@gmail.com>
wrote:
> Thanks David for the quick reply.
> Yes currently HBase protobuf version mismatch is the issue.
>
> We are investigating any common version of protobuf is there to support for
> HBase an HIVE.
>
>
>
>
> On Tue, Mar 10, 2020 at 6:08 PM David Lavati <dlavati@cloudera.com.invalid
> >
> wrote:
>
> > Hi,
> >
> > I recently investigated this in
> > https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look
> > feasible, as protobuf has to be in sync across the whole stack, but the
> > external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my
> > comment on the jira and
> > https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
> >
> > Kind Regards,
> > David
> >
> > On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
> > lalamchinnarao13@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > We are working on HIVE building and running on ARM architecture. As
> part
> > of
> > > this i want to add one periodic HIVE JENKINS JOB on ARM machines. Which
> > > helps to identity gaps for ARM, If any issues identified we will keep
> on
> > > working to fix those issues.
> > >
> > > Currently to run HIVE on aarch64, we need to upgrade to protobuf-3.7.1.
> > > HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939> upgraded
> > > protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are
> > working
> > > on this.
> > >
> > > Thanks,
> > > Chinna Rao Lalam
> > >
> >
>
>
> --
> Hope It Helps,
> Chinna
>
Re: HIVE building on ARM
Posted by Chinna Rao Lalam <la...@gmail.com>.
Thanks David for the quick reply.
Yes currently HBase protobuf version mismatch is the issue.
We are investigating any common version of protobuf is there to support for
HBase an HIVE.
On Tue, Mar 10, 2020 at 6:08 PM David Lavati <dl...@cloudera.com.invalid>
wrote:
> Hi,
>
> I recently investigated this in
> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look
> feasible, as protobuf has to be in sync across the whole stack, but the
> external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my
> comment on the jira and
> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
>
> Kind Regards,
> David
>
> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
> lalamchinnarao13@gmail.com>
> wrote:
>
> > Hi All,
> >
> > We are working on HIVE building and running on ARM architecture. As part
> of
> > this i want to add one periodic HIVE JENKINS JOB on ARM machines. Which
> > helps to identity gaps for ARM, If any issues identified we will keep on
> > working to fix those issues.
> >
> > Currently to run HIVE on aarch64, we need to upgrade to protobuf-3.7.1.
> > HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939> upgraded
> > protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are
> working
> > on this.
> >
> > Thanks,
> > Chinna Rao Lalam
> >
>
--
Hope It Helps,
Chinna
Re: HIVE building on ARM
Posted by David Lavati <dl...@cloudera.com.INVALID>.
Hi,
I recently investigated this in
https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look
feasible, as protobuf has to be in sync across the whole stack, but the
external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my
comment on the jira and
https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
Kind Regards,
David
On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <la...@gmail.com>
wrote:
> Hi All,
>
> We are working on HIVE building and running on ARM architecture. As part of
> this i want to add one periodic HIVE JENKINS JOB on ARM machines. Which
> helps to identity gaps for ARM, If any issues identified we will keep on
> working to fix those issues.
>
> Currently to run HIVE on aarch64, we need to upgrade to protobuf-3.7.1.
> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939> upgraded
> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are working
> on this.
>
> Thanks,
> Chinna Rao Lalam
>