Re: HIVE building on ARM

[Chinna Rao Lalam <la...@gmail.com>]

Hi Ashutosh,

I want to migrate
https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/ this
job to the new  system https://ci-hadoop.apache.org.  Please grant me
rights to create jobs on the new Jenkins https://ci-builds.apache.org.  My
LDAP userid : chinnaraol.
I have requested the same in apache infra. Thank you.

Thank you Stamatis,Zoltan for the pointers.


Thanks,
Chinna

On Thu, Jun 18, 2020 at 5:36 PM Stamatis Zampetakis <za...@gmail.com>
wrote:

> Hello Chinna,
>
> The hudson-jobadmin privilege can be granted by PMC chairs.
> I don't know if there is any particular policy in Hive on who should have
> this privilege so I guess you should request it from Ashutosh.
>
> Best,
> Stamatis
>
> On Thu, Jun 18, 2020 at 12:05 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
>
>> Hey Chinna!
>>
>> On 6/18/20 11:43 AM, Chinna Rao Lalam wrote:
>> > As you said, migrating this job to the new ci-hadoop instance looks
>> good as
>> > Hadoop also shares the same armN slaves.
>>
>> Sounds great!
>>
>> > I am able to login the new ci-hadoop instance with Apache LDAP
>> credentials,
>> > but i am not able to see the job creation option. Should I request
>> access
>> > or the process for creation of a job is different than jenkin?.
>> > Please guide me to create the new job in the ci-hadoop instance. I will
>> > migrate this job after connecting the armN slaves to the new system.
>>
>>
>> I've also logged in - and apparently I've create job rights; I'm happy to
>> help, but the best would be to self-service yourselft :)
>> I think you may miss the "hudson-jobadmin" privilege.
>> Probably Gavin (or someone on the infra team) could help you with that..
>> to talk to them quickly - you can reach them on the #asfinfra channel (on
>> the asf-slack).
>>
>> The migration effort is coordinated thru the hadoop-migrations mailing
>> list (I've cc-ed that list)
>> you may want to subscribe to it by sending a mail to:
>> hadoop-migrations-subscribe@infra.apache.org
>>
>> cheers,
>> Zoltan
>>
>>
>>
>> >
>> > Thanks
>> > Chinna
>> >
>> > On Wed, Jun 17, 2020 at 11:57 AM Zhenyu Zheng <
>> zhengzhenyulixi@gmail.com>
>> > wrote:
>> >
>> >> Hi Zoltan,
>> >>
>> >> Thanks alot for the information, so looks like one possible solution
>> is as
>> >> you suggest, move the current ARM2 and ARM3 (those two were donate to
>> >> builds.apache.org by us) to the new ci-hadoop cluster and set up the
>> jobs
>> >> just as what has been done in current jenkins.
>> >>
>> >> I will also ask our team member works on other projects to find out
>> what
>> >> the status of other projects is.
>> >>
>> >> BR,
>> >>
>> >> On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
>> >>
>> >>> Hey,
>> >>>
>> >>> There is an effort by the Apache Infra to change the way Jenkins
>> stuff is
>> >>> organized; a couple months ago Gavin wrote an email about it:
>> >>>
>> >>>
>> http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3CCAN0Gg1dOdEPzatJz9bOFE-2veR7QG7H0HMvYjmsLdgjr8_RPew@mail.gmail.com%3E
>> >>> The resources for running these jobs are coming from the H0~H21 slaves
>> >>> which will be migrated to the new jenkins master eventually.
>> >>>
>> >>>   >> So please
>> >>>   >> suggest a way which direction we can move and can you share some
>> >>> details
>> >>>   >> about the new ci-hadoop instance.
>> >>>
>> >>> Since Hadoop testing is also happening on ARM - I think the best
>> would be
>> >>> to also migrate the armN slaves and the Hive arm nightly over to the
>> new
>> >>> ci-hadoop instance.
>> >>>
>> >>> On 6/16/20 8:40 AM, Zhenyu Zheng wrote:
>> >>>> Thanks for the info, I wonder if where does the resource of ci-hadoop
>> >>> and
>> >>>> hive-test-kube come from? Do they include ARM resources?
>> >>>
>> >>> Interesting question; the resources for Hive testing are donated by
>> >>> Cloudera.
>> >>> About the ARM workers I think Chinna could provide more details.
>> >>> ...I've no idea don't know who sponsors the Hxx slaves
>> >>>
>> >>>> Can you provide some more information about how the new
>> hive-test-kube
>> >>> is
>> >>>> running?
>> >>> It's basically a Jenkins instance which is using kubernetes pods to
>> run
>> >>> things.
>> >>> The whole thing is running on a GKE cluster.
>> >>> While I was working on it I collected stuff needed for it in this
>> repo:
>> >>> https://github.com/kgyrtkirk/hive-test-kube/
>> >>> it should be possible to start a new deployment using that stuff
>> >>>
>> >>> cheers,
>> >>> Zoltan
>> >>>
>> >>>>
>> >>>> BR,
>> >>>> Kevin Zheng
>> >>>>
>> >>>> On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam <
>> >>>> lalamchinnarao13@gmail.com> wrote:
>> >>>>
>> >>>>> Hi Zoltan,
>> >>>>>
>> >>>>> Thanks for the update.
>> >>>>>
>> >>>>> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is
>> >>>>> targeting to run hive tests daily on "arm" slaves, it is using 2 arm
>> >>>>> slaves.
>> >>>>> To find any potential issues with "arm" and fix the issues. So
>> please
>> >>>>> suggest a way which direction we can move and can you share some
>> >>> details
>> >>>>> about the new ci-hadoop instance.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Chinna
>> >>>>>
>> >>>>> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <ki...@rxd.hu>
>> wrote:
>> >>>>>
>> >>>>>> Hey all,
>> >>>>>>
>> >>>>>> In an ticket (INFRA-20416) Gavin asked me if we are completely off
>> >>>>>> builds.apache.org - when I went over the jobs I've saw that
>> >>>>>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running
>> there
>> >>>>>> once a day.
>> >>>>>>
>> >>>>>> Since builds.apache.org will be shut down in sometime in the
>> future
>> >>> - we
>> >>>>>> should move this job to the new ci-hadoop instance or to
>> >>> hive-test-kube.
>> >>>>>> The key feature of the job is that it runs the test on the "armX"
>> >>> slaves;
>> >>>>>> which are statically configured on b.a.o.
>> >>>>>> Not sure which way to go - but we will have to move in some
>> direction.
>> >>>>>>
>> >>>>>> cheers,
>> >>>>>> Zoltan
>> >>>>>>
>> >>>>>>
>> >>>>>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote:
>> >>>>>>> Hi Chinna,
>> >>>>>>>
>> >>>>>>> Thanks alot for the reply, I uploaded a patch and also a github PR
>> >>> for
>> >>>>>>> https://issues.apache.org/jira/browse/HIVE-21939 .
>> >>>>>>> In the patch, I bumped the protobuf used in standalone-metadata to
>> >>> 2.6.1
>> >>>>>>> and added a new profile, this profile will identify
>> >>>>>>> the hardware architecture and if it is Aarch64, it will override
>> the
>> >>>>>>> protobuf group.id and package to com.github.os72 which
>> >>>>>>> includes ARM support. For X86 platform, Hive will still download
>> the
>> >>>>>>> protobuf packages from org.google repo. I think with
>> >>>>>>> this method, we can keep the influence to existing x86 users to
>> the
>> >>>>>>> minimum. I hope this could be a acceptable short-term
>> >>>>>>> solution.
>> >>>>>>>
>> >>>>>>> I've manually tested on my machine and the github PR travis CI
>> test
>> >>> has
>> >>>>>>> already passed, so the build process is OK, so let's
>> >>>>>>> wait for the full test result from builds.apache.org.
>> >>>>>>>
>> >>>>>>> BR,
>> >>>>>>>
>> >>>>>>> Zhenyu
>> >>>>>>>
>> >>>>>>> On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <
>> >>>>>> lalamchinnarao13@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> Hi Zhenyu,
>> >>>>>>>>
>> >>>>>>>> Until HBase dependency resolved, without effecting the existing
>> code
>> >>>>>> on X86
>> >>>>>>>> i suggest create a separate profile with "os72" repo.
>> >>>>>>>>
>> >>>>>>>> Down the line we should have common version for both X86 and ARM.
>> >>>>>>>>
>> >>>>>>>> Hope It Helps,
>> >>>>>>>> Chinna
>> >>>>>>>>
>> >>>>>>>> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <
>> >>>>>> zhengzhenyulixi@gmail.com>
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> Hi Chinna, David and others might interested,
>> >>>>>>>>>
>> >>>>>>>>> Thanks for bring this up, we are currently working on improving
>> >>>>>> enabling
>> >>>>>>>>> big-data software on the ARM platform,
>> >>>>>>>>> we have already done fixes and providing CIs to some of the
>> >>> well-know
>> >>>>>>>>> projects like:
>> >>>>>>>>> 1. Hadoop:
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
>> >>>>>>>>> 2. Spark:
>> https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
>> >>>>>>>>> 3. HBase:
>> >>>>>>>>>
>> >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
>> >>>>>>>>>
>> >>>>>>>>> And we are now working on projects including Hive, Kudu, etc.
>> >>>>>>>>>
>> >>>>>>>>> Regarding to the protobuf upgrades in Hive, except upgrading to
>> 3.x
>> >>>>>> and
>> >>>>>>>>> break dependency for HBase, there can
>> >>>>>>>>> be some possible short-term plan(or walk-arounds), doing thes
>> can
>> >>> make
>> >>>>>>>> Hive
>> >>>>>>>>> work on ARM without break any
>> >>>>>>>>> dependencies, and then we can interact with Hbase project to see
>> >>> how
>> >>>>>> can
>> >>>>>>>> we
>> >>>>>>>>> both upgrade to 3.x(since this
>> >>>>>>>>> make take some time).
>> >>>>>>>>>
>> >>>>>>>>> Those possible solutions can be:
>> >>>>>>>>> 1. Using pre-patched protobuf 2.5.0 with ARM support
>> >>>>>>>>> from org.openlabtesting repo, some projects(HBase did
>> >>>>>>>>> this: https://github.com/apache/hbase/pull/959, and we will
>> add a
>> >>>>>>>> profile
>> >>>>>>>>> for this, it will detact the host arch and
>> >>>>>>>>> will only use this pre-patched protobuf when the host arch is
>> ARM
>> >>> so
>> >>>>>> that
>> >>>>>>>>> nothing will be affected for existing x86
>> >>>>>>>>> users; Spark and Hadoop used some other package from this repo
>> as
>> >>> they
>> >>>>>>>> have
>> >>>>>>>>> already upgraded protobuf).
>> >>>>>>>>>
>> >>>>>>>>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
>> >>>>>>>>>
>> >>>>>>
>> >>>
>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
>> >>>>>>>>> Hive already used ``protoc-jar-maven-plugin`` from this repo:
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>
>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>
>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
>> >>>>>>>>> so it is a minor update.
>> >>>>>>>>>
>> >>>>>>>>> I've tested both ways, worked OK on the ARM machine.
>> >>>>>>>>>
>> >>>>>>>>> So my ideas will be that we make this a two-step-work, first
>> step
>> >>> we
>> >>>>>> use
>> >>>>>>>>> one of the short-term plan to enable
>> >>>>>>>>> Hive on ARM, and 2nd Step, we look for the long run to discuss
>> with
>> >>>>>> HBase
>> >>>>>>>>> to find out a fesiable solution.
>> >>>>>>>>>
>> >>>>>>>>> BR,
>> >>>>>>>>>
>> >>>>>>>>> Zhenyu Zheng
>> >>>>>>>>>
>> >>>>>>>>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
>> >>>>>>>>> lalamchinnarao13@gmail.com>
>> >>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Thanks David for the quick reply.
>> >>>>>>>>>> Yes currently HBase protobuf version mismatch is the issue.
>> >>>>>>>>>>
>> >>>>>>>>>> We are investigating any common version of protobuf is there to
>> >>>>>> support
>> >>>>>>>>> for
>> >>>>>>>>>> HBase an HIVE.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati
>> >>>>>>>>> <dlavati@cloudera.com.invalid
>> >>>>>>>>>>>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Hi,
>> >>>>>>>>>>>
>> >>>>>>>>>>> I recently investigated this in
>> >>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it
>> didn't
>> >>> look
>> >>>>>>>>>>> feasible, as protobuf has to be in sync across the whole
>> stack,
>> >>> but
>> >>>>>>>> the
>> >>>>>>>>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x,
>> >>> see
>> >>>>>> my
>> >>>>>>>>>>> comment on the jira and
>> >>>>>>>>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414
>> .
>> >>>>>>>>>>>
>> >>>>>>>>>>> Kind Regards,
>> >>>>>>>>>>> David
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
>> >>>>>>>>>>> lalamchinnarao13@gmail.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Hi All,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> We are working on HIVE building and running on ARM
>> >>> architecture. As
>> >>>>>>>>>> part
>> >>>>>>>>>>> of
>> >>>>>>>>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM
>> >>> machines.
>> >>>>>>>>> Which
>> >>>>>>>>>>>> helps to identity gaps for ARM, If any issues identified we
>> will
>> >>>>>>>> keep
>> >>>>>>>>>> on
>> >>>>>>>>>>>> working to fix those issues.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Currently to run HIVE on aarch64, we need to upgrade to
>> >>>>>>>>> protobuf-3.7.1.
>> >>>>>>>>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939
>> >
>> >>>>>>>>> upgraded
>> >>>>>>>>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures.
>> We
>> >>> are
>> >>>>>>>>>>> working
>> >>>>>>>>>>>> on this.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>> Chinna Rao Lalam
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> --
>> >>>>>>>>>> Hope It Helps,
>> >>>>>>>>>> Chinna
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> Hope It Helps,
>> >>>>>>>> Chinna
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Hope It Helps,
>> >>>>> Chinna
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>

-- 
Hope It Helps,
Chinna