Re: [EXTERNAL] Re: Sync of Branch-3 & Branch-3.1 for 3.2.0 pipeline

[Aman Raj <ra...@microsoft.com.INVALID>]

Hi team,

Need help in a test case failing in branch-3. As you know we have been trying to ensure a Green branch-3. When we had started this task the total number of failures was 2900+. We have brought it down to less than 20 as of now. JIRA link to track the status : https://issues.apache.org/jira/browse/HIVE-26836

I am facing some issues in fixing the following tests :
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver

Issue :
PREHOOK: Input: default@src
PREHOOK: Output: default@src
Failed to monitor Job[-1] with exception 'java.lang.IllegalStateException(Connection to remote Spark driver was lost)' Last known state = SENT
Failed to execute spark task, with exception 'java.lang.IllegalStateException(RPC channel is closed.)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel is closed.

History :
Initially the tests had failed with errors which I fixed in the following task : https://issues.apache.org/jira/browse/HIVE-26940

Does anyone know what the issue is here ? There are 6-7 failures because of this test case. Link to the failed test cases for the stacktrace : http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3949/2/tests/
Thanks,
Aman.

________________________________
From: Aman Raj <ra...@microsoft.com.INVALID>
Sent: Friday, December 9, 2022 9:33 AM
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Sync of Branch-3 & Branch-3.1 for 3.2.0 pipeline

Thanks Pravin for your support. Can someone please help me merge this PR to branch-3 HIVE-26816 : Add Jenkins file for branch-3 by amanraj2520 · Pull Request #3841 · apache/hive (github.com)<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F3841&data=05%7C01%7Crajaman%40microsoft.com%7C94c1ac2c4ddd40437b5f08dad99a6017%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638061554335365489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=D7CwaRcRaQ5ubjz3Ki95HkyclN2a%2BBZ7lvTddDQpTLY%3D&reserved=0>. I do not have access to do that. Then we will start development on it.

Thanks,
Aman.


________________________________
From: Pravin Sinha <ma...@gmail.com>
Sent: Friday, December 9, 2022 1:55 AM
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Sync of Branch-3 & Branch-3.1 for 3.2.0 pipeline

[You don't often get email from mailpravinks@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Hi Aman,
 I also think that we can merge the PR to enable the test pipeline if the
change looks fine and subsequently we can fix the tests to bring it to
green state (hopefully by cherry picking a few commits from branch-3.1
which is already in green state) . Looks like currently the tests are
broken in branch-3.

Thanks,
Pravin

On Thu, Dec 8, 2022 at 3:59 PM Aman Raj <ra...@microsoft.com.invalid>
wrote:

> Hi team,
>
> For the addition of Jenkins file for branch-3, branch-3 has some existing
> tests failing which was because Jenkins was not running on branch-3. We are
> planning to merge this Jenkins file irrespective of this PR having test
> failures, since this does not change the code. We will create separate
> tasks for ensuring that branch-3 has a green build.
>
> Link to the PR : https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F3841&data=05%7C01%7Crajaman%40microsoft.com%7C94c1ac2c4ddd40437b5f08dad99a6017%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638061554335365489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=D7CwaRcRaQ5ubjz3Ki95HkyclN2a%2BBZ7lvTddDQpTLY%3D&reserved=0
>
> Fyi, branch-3.1 has a green build.
>
> Thanks,
> Aman.
> ________________________________
> From: Aman Raj <ra...@microsoft.com.INVALID>
> Sent: Wednesday, December 7, 2022 3:19 PM
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Sync of Branch-3 & Branch-3.1 for 3.2.0
> pipeline
>
> Hi Ayush,
>
> Thanks for clarifying. Will wait for it to turn green.
>
> Thanks,
> Aman.
> ________________________________
> From: Ayush Saxena <ay...@gmail.com>
> Sent: Wednesday, December 7, 2022 3:11 PM
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Sync of Branch-3 & Branch-3.1 for 3.2.0
> pipeline
>
> Hi Aman,
> The build is already running for your PR:
>
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3841%2F1%2Fpipeline&data=05%7C01%7Crajaman%40microsoft.com%7C94c1ac2c4ddd40437b5f08dad99a6017%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638061554335365489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=osIglpTld3PtOPyFhGBLTSU9Ku1FWngPMofNQXILpyM%3D&reserved=0
>
> The JenkinsFile is picked from the PR while running rather than the target
> branch.
>
> -Ayush
>
> > On 07-Dec-2022, at 3:03 PM, Aman Raj <ra...@microsoft.com.invalid>
> wrote:
> >
> > Hi Stamatis,
> >
> > How can we ensure that unless the PR is merged. Please suggest.
> > I was thinking of merging this and raising a sample PR on branch-3 to
> check whether it works or not. Is there some other way?
> >
> > Thanks,
> > Aman.
> > ________________________________
> > From: Stamatis Zampetakis <za...@gmail.com>
> > Sent: Wednesday, December 7, 2022 2:51 PM
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Sync of Branch-3 & Branch-3.1 for 3.2.0
> pipeline
> >
> > Hey Aman,
> >
> > Before checking in the PR we should ensure that it works as expected;
> i.e.,
> > having a green run in a reasonable time.
> >
> > Best,
> > Stamatis
> >
> >> On Wed, Dec 7, 2022 at 9:29 AM Aman Raj <ra...@microsoft.com.invalid>
> >> wrote:
> >>
> >> Hi Stamatis,
> >>
> >> I have raised a Pull Request for the same -
> >>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F3841&data=05%7C01%7Crajaman%40microsoft.com%7C94c1ac2c4ddd40437b5f08dad99a6017%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638061554335365489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=D7CwaRcRaQ5ubjz3Ki95HkyclN2a%2BBZ7lvTddDQpTLY%3D&reserved=0.
> Can you please check this and
> >> merge it.
> >>
> >> Thanks,
> >> Aman.
> >>
> >> ________________________________
> >> From: Aman Raj <ra...@microsoft.com.INVALID>
> >> Sent: Wednesday, December 7, 2022 1:50 PM
> >> To: dev@hive.apache.org <de...@hive.apache.org>
> >> Subject: Re: [EXTERNAL] Re: Sync of Branch-3 & Branch-3.1 for 3.2.0
> >> pipeline
> >>
> >> Hi Stamatis,
> >>
> >> Sure, thanks a lot for your help. Will make that change and update this
> >> mail thread.
> >>
> >> Thanks,
> >> Aman.
> >> ________________________________
> >> From: Stamatis Zampetakis <za...@gmail.com>
> >> Sent: Wednesday, December 7, 2022 1:42 PM
> >> To: dev@hive.apache.org <de...@hive.apache.org>
> >> Subject: [EXTERNAL] Re: Sync of Branch-3 & Branch-3.1 for 3.2.0 pipeline
> >>
> >> Hi team,
> >>
> >> I don't think you need any kind of special permissions to enable
> pre-commit
> >> tests for branch-3. I have the impression that just committing an
> >> appropriate Jenkinsfile (e.g., HIVE-24331 [1]) should do the trick.
> >>
> >> Best,
> >> Stamatis
> >>
> >> [1]
> >>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-24331&data=05%7C01%7Crajaman%40microsoft.com%7C94c1ac2c4ddd40437b5f08dad99a6017%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638061554335365489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=J57O27jG4rzcXjKMSIGD81irJTk7DoWxjswxwkNF5xg%3D&reserved=0
> >>
> >> On Wed, Dec 7, 2022 at 8:41 AM Sankar Hariappan
> >> <Sa...@microsoft.com.invalid> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> It is a blocker for us to start the Hive 3.2 release efforts. Can
> someone
> >>> help adding Jenkins pipeline for "branch-3" or pls add "sankarh",
> >> "mahesh"
> >>> as admin?
> >>>
> >>> Thanks,
> >>> Sankar
> >>>
> >>> -----Original Message-----
> >>> From: Aman Raj <ra...@microsoft.com.INVALID>
> >>> Sent: Monday, December 5, 2022 12:10 PM
> >>> To: dev@hive.apache.org
> >>> Subject: [EXTERNAL] Re: Sync of Branch-3 & Branch-3.1 for 3.2.0
> pipeline
> >>>
> >>> [You don't often get email from rajaman@microsoft.com.invalid. Learn
> why
> >>> this is important at https://aka.ms/LearnAboutSenderIdentification ]
> >>>
> >>> Hi team,
> >>>
> >>> Can someone please help me set up the Jenkins pipeline for branch-3 of
> >>> Hive. I do not have access to do the same and it is essential to raise
> >> PR's
> >>> for the 3.2.0 release. Please refer to the email thread for more
> details.
> >>> Let me know if you need anything else from my side.
> >>>
> >>> Thanks,
> >>> Aman.
> >>> ________________________________
> >>> From: Aman Raj <ra...@microsoft.com.INVALID>
> >>> Sent: Friday, December 2, 2022 9:57 AM
> >>> To: dev@hive.apache.org <de...@hive.apache.org>
> >>> Subject: [EXTERNAL] Sync of Branch-3 & Branch-3.1 for 3.2.0 pipeline
> >>>
> >>> Hi team,
> >>>
> >>> We have started working on the hive-3.2.0 release. The plan is to cut
> >>> branch-3.2 from the base branch-3. While we were working on the 3.2.0
> >>> release, we found that there are some commits (Please refer to this
> >> Parent
> >>> JIRA for the analysis -
> >>>
> >>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26752&data=05%7C01%7Crajaman%40microsoft.com%7C94c1ac2c4ddd40437b5f08dad99a6017%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638061554335365489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=n7KdxD5vKPHrkQWnTZRRPHZ%2FhRFbrp8gMLFvPLpdHdc%3D&reserved=0
> >> )
> >>> which went to branch-3.1 which doesn't exist in the branch-3, same we
> are
> >>> planning to backport to branch-3. There can be two approaches
> >>>
> >>>
> >>>
> >>>  1.  Cherry-pick missed commits and push to branch-3
> >>>  2.  Create Jira's for each commit and upload patch.
> >>>
> >>> Mostly we feel the second option will be better, any other thoughts
> will
> >>> be appreciated.
> >>>
> >>>
> >>> We also need help on creating the pre-commit pipelines for branch-3 as
> we
> >>> don't have Jenkin's access.
> >>>
> >>>
> >>>
> >>> Please feel free to add any Jira's which you want to have to 3.2.0. All
> >>> the changes for the 3.2.0 can be tracked here - Apache Hive - Agile
> >> Board -
> >>> ASF JIRA<
> >>>
> >>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2FRapidBoard.jspa%3FrapidView%3D564%26view%3Ddetail%26selectedIssue%3DHIVE-26749%26quickFilter%3D2586&data=05%7C01%7Crajaman%40microsoft.com%7C94c1ac2c4ddd40437b5f08dad99a6017%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638061554335365489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=k8cN7lUL45ddS7rinhI99GxSi8xQnR5BLFqV%2BWJMT3A%3D&reserved=0
> >>>>
> >>>
> >>> A friendly request/reminder - After this exercise if any ongoing bug
> >> fixes
> >>> on branch-3.x should also be cherry-picked to the base branch-3
> >>>
> >>> Thanks,
> >>> Aman.
> >>>
> >>>
> >>
>
T