Re: [EXTERNAL] Re: Branch-3 backports and build stability

[vihang karajgaonkar <vi...@apache.org>]

Just to update on the HoS test failures for TestMiniSparkOnYarnCliDriver, I
think I was finally able to resolve them (at least on local). I had to
revert HIVE-21044 because it was causing OOM for those tests. Also, in
order for these tests to work we will have to downgrade netty from
4.1.69.Final to 4.1.51.Final. I understand that we had upgraded netty from
4.1.17.Final to 4.1.69.Final for CVEs but the highest netty version that we
can support without breaking HoS is 4.1.51.Final. Note that 4.1.51.Final
includes many of the CVEs which affected 4.1.17.Final so we are still in a
better place than branch-3.1. Unfortunately, there is no good way to make
HoS work with a higher netty version so I think we should downgrade the
netty version to 4.1.51.Final for now and look at more options to upgrade
it 4.1.69.Final in a separate ticket.

I still need to understand why the tests which are working for me locally
don't work on the PR job. I tried running the split test classes using the
following command. Is that the right way to simulate builds from the PR
job? Let me know if anyone has more ideas.

mvn test
-Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
-Pqsplits

Thanks,
Vihang


On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <za...@gmail.com>
wrote:

> Hello,
>
> Thanks Aman for bringing this up and also for cleaning up after others (I
> saw that you raised tickets and PRs for addressing the failures).
>
> Many thanks to Vihang as well for helping out. Regarding flaky tests, yes
> we should disable them as soon as we see them.
> There have been some other discussions on how to approach flaky tests the
> more recent I could find is here [1].
>
> Best,
> Stamatis
>
> [1] https://lists.apache.org/thread/lv3bhlfoq8fwd9dwyjf7g4nx32wtrygv
>
> On Fri, Feb 17, 2023 at 4:37 AM Aman Raj <ra...@microsoft.com.invalid>
> wrote:
>
> > Hi team,
> >
> > Thanks Vihang for looking into this. I have commented on the JIRA you
> > created.
> >
> > Just to bring everyone's notice, I have seen that there has been a couple
> > of pushes to branch-3, which has lead to 5 more new test failures. The
> test
> > failures are in orc_merge1, orc_merge2, orc_merge3, orc_merge4 and
> > orc_merge10. These tests did not use to fail before. I would sincerely
> urge
> > the community to raise a PR against branch-3, so that the Jenkins
> pipeline
> > can run and then only merge things to branch-3. We had 2900+ failures
> when
> > we started 2 months back and now having brought it down to less than 15,
> > new failures again has pushed us back in this effort.
> >
> > I would like to thank everyone who has participated in this effort and
> > made it possible till this stage. Also, if the contributors can take
> > ownership of these new test case failures and fix them, it will be of
> great
> > help.
> >
> > Thanks,
> > Aman.
> > ________________________________
> > From: vihang karajgaonkar <vi...@apache.org>
> > Sent: Friday, February 17, 2023 6:10 AM
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > [You don't often get email from vihangk1@apache.org. Learn why this is
> > important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > Hi Aman,
> >
> > I created
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C7cc87475f1fe4036bcd308db107faf36%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638121912852386975%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=E7FD0nKrKQq%2F297DlTgJog365lH4Q0Xa8I2zEGgwtQY%3D&reserved=0
> > to look into
> > TestMiniSparkOnYarnCliDriver failures. I have a working theory of what
> > might be going on there. I am still investigating what is the right way
> to
> > fix it though.
> >
> > Thanks,
> > Vihang
> >
> > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj <rajaman@microsoft.com.invalid
> >
> > wrote:
> >
> > > Hi Vihang,
> > >
> > > Yes the tests are failing locally as well with the same issue.
> > >
> > > Thanks,
> > > Aman.
> > >
> > > Get Outlook for Android<
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C7cc87475f1fe4036bcd308db107faf36%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638121912852386975%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XbUx9nnHQKtIdemDWtNB8W%2BoAN9r997WjFOZlJLhBH8%3D&reserved=0
> > >
> > > ________________________________
> > > From: Vihang Karajgaonkar <vi...@databricks.com.INVALID>
> > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > >
> > > [You don't often get email from
> > vihang.karajgaonkar@databricks.com.invalid.
> > > Learn why this is important at
> > > https://aka.ms/LearnAboutSenderIdentification ]
> > >
> > > Thanks a lot Stamatis for starting this thread. I really appreciate all
> > the
> > > efforts to stabilize branch-3 to get it to a releasable state and I
> agree
> > > that we should get it to a green state before opening it for PRs not
> > > related to test failures. I can help with the effort as well.
> > >
> > > If we want to get the branch back to green state soon, have we
> considered
> > > disabling the tests which are clearly flaky? (e.g pass on some builds
> and
> > > fail on the other build with no new code changes). If we don't do that,
> > we
> > > will keep playing whack a mole with those tests. I propose for such
> tests
> > > we should disable them and create tickets to unflake them separately.
> > This
> > > will help us get back to a green state faster.
> > >
> > > Hi Aman,
> > > For TestMiniSparkOnYarnCliDriver failures, you probably should also
> look
> > > into the spark driver/application logs and see if there are
> > infrastructure
> > > errors (e.g OOMs). Are these tests failing when you run locally?
> > >
> > > Thanks,
> > > Vihang
> > >
> > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj <rajaman@microsoft.com.invalid
> >
> > > wrote:
> > >
> > > > +1,
> > > > Thanks Stamatis and Lazlo for helping in the test case fixes till
> now.
> > > >
> > > > Team,
> > > > I need help in fixing the following tests in Hive. I have tried
> > different
> > > > approaches but no luck till now.
> > > > I am facing some issues in fixing the following tests :
> > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > >
> > > > Issue :
> > > > PREHOOK: Input: default@src
> > > > PREHOOK: Output: default@src
> > > > Failed to monitor Job[-1] with exception
> > > > 'java.lang.IllegalStateException(Connection to remote Spark driver
> was
> > > > lost)' Last known state = SENT
> > > > Failed to execute spark task, with exception
> > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > FAILED: Execution Error, return code 1 from
> > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel is
> closed.
> > > >
> > > > History :
> > > > Initially the tests had failed with errors which I fixed in the
> > following
> > > > task :
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C7cc87475f1fe4036bcd308db107faf36%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638121912852386975%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qIgZVHldffGFLL7MERtkVwv8QFOPwrM49JD97BH%2Bku0%3D&reserved=0
> > > >
> > > > Does anyone know what the issue is here ? There are 6-7 failures
> > because
> > > > of this test case. Link to the failed test cases for the stacktrace :
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C7cc87475f1fe4036bcd308db107faf36%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638121912852386975%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=B4nrnCh%2B2tC2OKYwzN81y8iHb30b2OaRMcZX3gQie2Y%3D&reserved=0
> > > > Thanks,
> > > > Aman.
> > > >
> > > > ________________________________
> > > > From: László Bodor <bo...@gmail.com>
> > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > +1
> > > > also, if I merged something that I thought was for test stability
> (but
> > > > instead it was a feature), excuse me :)
> > > > for reference, the whole green test initiative is tracked under this
> > > > umbrella:
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C7cc87475f1fe4036bcd308db107faf36%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638121912852386975%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ainj7oCYknhYIHVmXITj4zBoo9466%2Bqof9ZIYkVnh44%3D&reserved=0
> > > >
> > > > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023.
> febr.
> > > 7.,
> > > > K, 12:09):
> > > >
> > > > > Hi all,
> > > > >
> > > > > The build in branch-3 is not yet green; there are ~25 test
> failures.
> > It
> > > > is
> > > > > a common practice that we shouldn't push changes on top of a broken
> > > build
> > > > > unless they are addressing test failures.
> > > > >
> > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo Bodor) are
> > > > working
> > > > > hard to stabilize the build for quite some time now. If you want to
> > > help
> > > > > out then start by reviewing, merging, and fixing things around test
> > > > > failures.
> > > > >
> > > > > It's not yet the time to bring new features, upgrades, bugs, etc.,
> in
> > > > > branch-3. I would encourage  committers to not approve such changes
> > > till
> > > > we
> > > > > get back to a stable branch.
> > > > >
> > > > > Best,
> > > > > Stamatis
> > > > >
> > > >
> > >
> >
>

Re: [EXTERNAL] Re: Branch-3 backports and build stability

[Aman Raj <ra...@microsoft.com.INVALID>]

Hi Vihang,

Only three tickets remain now to be backported :
5ea9a9ca13 HIVE-25726: Upgrade velocity to 2.3 due to CVE-2020-13936 (Sourabh Goyal via Naveen Gangam)
164486f9c6 HIVE-25468: Authorization for Create/Drop functions in HMS(Saihemanth Gantasala via Naveen Gangam)
135cfe6c2f HIVE-25547: Alter view as Select statement should create Authorizable events in HS2(Saihemanth reviewed by Naveen Gangam)

Thanks,
Aman.
________________________________
From: Vihang Karajgaonkar <vi...@gmail.com>
Sent: Wednesday, May 10, 2023 12:17 AM
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability

[You don't often get email from vihangk1@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Thanks Aman. I thought all the changes in release 3.2.0 were listed under
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26751&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dI1WgjMoBVjWAUqGwaGaXRhyxEAtIHC%2BQLAWCUJPt7I%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26751> and I saw them all
resolved. Do you know which additional tickets need to go in branch-3 after
we backport the branch-3.1 fixes in branch-3?

On Tue, May 9, 2023 at 11:20 AM Aman Raj <ra...@microsoft.com.invalid>
wrote:

> Hi Vihang,
>
> We only have 4 tickets remaining to be backported from branch-3.1 to
> branch-3. It will be completed next week.
>
> But there are a lot of new tickets that will go into release 3.2.0 on top
> of this. I was thinking of not cutting a release candidate now since it
> would mean that we only backport changes into that release candidate
> branch. This would again mean that if people commit only to branch-3 or the
> release branch, there will again be a lot of difference in these two
> branches when someone picks up the next release.
>
> Instead I am thinking that we should backport new changes to branch-3 and
> then only cut the release candidate. Please let me know your thoughts. If
> we agree that changes need to go into the new release candidate branch
> only, I am okay with that (I do not prefer it btw)
>
> Thanks,
> Aman.
>
> Get Outlook for Android<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MlHi3cH0KntRbaDbalzAZzIDfLZ8mhtWLZMewJO2iYY%3D&reserved=0<https://aka.ms/AAb9ysg>>
> ________________________________
> From: vihang karajgaonkar <vi...@apache.org>
> Sent: Monday, May 8, 2023 4:57:24 AM
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
>
> Hi Aman,
>
> I know you are backporting the branch-3.1 commits to branch-3. How close
> are you with finishing with them. Is there anything that we can help with
> to get it over the finish line?
>
> I am interested to know how close are we to cutting the branch for 3.2.0?
> Do you think we can have a release candidate this week?
>
> Thanks,
> Vihang
>
> On Thu, Mar 30, 2023 at 2:18 AM Stamatis Zampetakis <za...@gmail.com>
> wrote:
>
> > Huge thanks to everyone involved it is great to see the branch-3 in
> stable
> > state. As other people mentioned let's keep it that way!
> >
> > As far as it concerns back ports please be particularly cautious with
> > anything that touches the metastore schema and Thrift APIs.
> >
> > Best,
> > Stamatis
> >
> > On Wed, Mar 29, 2023, 4:36 AM vihang karajgaonkar <vi...@apache.org>
> > wrote:
> >
> > > Thanks a lot Aman for all your efforts on this. Really appreciate the
> > > initiative and all your hard work on this.
> > >
> > > I would like to request that all the committers should follow the merge
> > > process of master branch to merge PRs in branch-3. If there are any
> test
> > > failures which seem unrelated, please do not ignore them. One can run
> the
> > > flaky
> > > test runner <
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fjob%2Fhive-flaky-check%2F&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=W%2FRIukSnFP%2Bkhh0urf9EdMQIG%2BMdI6rBKFaMn5i20q8%3D&reserved=0<http://ci.hive.apache.org/job/hive-flaky-check/>
> <https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fjob%2Fhive-flaky-check%2F&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=W%2FRIukSnFP%2Bkhh0urf9EdMQIG%2BMdI6rBKFaMn5i20q8%3D&reserved=0<http://ci.hive.apache.org/job/hive-flaky-check/>>> to make
> > sure
> > > that test is indeed flaky. If the test is found to be flaky a
> > > ticket should be created to disable it. A separate ticket should be
> > created
> > > to deflake it and you can mention the original author or previous
> commit
> > > author who changed the test on that ticket to get help since they
> likely
> > > have the most context around that test. Once the flaky test is disabled
> > and
> > > we have a green CI job run, we should merge the PR. If others have any
> > > suggestions to improve this process please chime in.
> > >
> > > Thanks,
> > > Vihang
> > >
> > > On Tue, Mar 28, 2023 at 10:55 PM Aman Raj
> <rajaman@microsoft.com.invalid
> > >
> > > wrote:
> > >
> > > > Hi community,
> > > >
> > > > This is to notify that we have a green branch-3 now. The entire
> effort
> > of
> > > > fixing branch-3 test cases took around 4 months and as a team we
> > managed
> > > to
> > > > fix 2900+ test failures on branch-3. The entire effort can be tracked
> > > here
> > > > HIVE-26836<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ot7ShRi17W2o0%2FWtT3YrLGxuMq%2Bq79VMfitIe0wA6xU%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26836>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ot7ShRi17W2o0%2FWtT3YrLGxuMq%2Bq79VMfitIe0wA6xU%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26836>>>. We are
> > > > ready to push new features and improvements on branch-3 now.
> > > >
> > > > I really want to thank Vihang Karajgaonkar, Chris Nauroth, Lazlo
> Bodor,
> > > > Stamatis Zampetakis and Sankar Hariappan without whom this would not
> at
> > > all
> > > > have been possible. As a team we stuck together and participated in
> > > reviews
> > > > and actively suggested improvements which really helped in fixing
> some
> > > > major test failures.
> > > >
> > > > I would sincerely request that going further it should be made a
> point
> > to
> > > > merge things into branch-3 only if we have a green Jenkins pipeline.
> > > >
> > > > The next step would be to backport changes from branch-3.1 (From
> where
> > > > Hive-3.1.3 release was made) to branch-3. This would ensure that we
> do
> > > not
> > > > miss any specific ticket which went into Hive-3.1.3. I will take care
> > of
> > > > this. We can parallelly start pushing additional changes on branch-3.
> > > There
> > > > are approximately 25 tickets that need to be backported in this
> effort
> > > (Of
> > > > backporting changes from branch-3.1). I have made a note here<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fspreadsheets%2Fd%2F1K0U-vxLRZEs13oBzYBlVyK8dMMNthgXL5VEgzLRbeKs%2Fedit%3Fusp%3Dsharing&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nBliVZ8vjUY9kLSSsbQykqPt7O2OhHfeI74RhUYDABY%3D&reserved=0<https://docs.google.com/spreadsheets/d/1K0U-vxLRZEs13oBzYBlVyK8dMMNthgXL5VEgzLRbeKs/edit?usp=sharing>
> <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fspreadsheets%2Fd%2F1K0U-vxLRZEs13oBzYBlVyK8dMMNthgXL5VEgzLRbeKs%2Fedit%3Fusp%3Dsharing&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nBliVZ8vjUY9kLSSsbQykqPt7O2OhHfeI74RhUYDABY%3D&reserved=0<https://docs.google.com/spreadsheets/d/1K0U-vxLRZEs13oBzYBlVyK8dMMNthgXL5VEgzLRbeKs/edit?usp=sharing>
> >
> > > > >
> > > >
> > > > Again, thanks a lot to everyone who supported and participated in
> this
> > > > effort. Lets make this 3.2.0 Hive release happen!!
> > > >
> > > > Thanks,
> > > > Aman.
> > > >
> > > > ________________________________
> > > > From: Aman Raj <ra...@microsoft.com.INVALID>
> > > > Sent: Monday, March 20, 2023 9:21 AM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > Hi Vihang/community,
> > > >
> > > > Found the ticket which broke mm_all.q. This issue comes because of
> > > > HIVE-20182. Works in my local and on the Jenkins pipeline as well.
> > Link :
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F4127&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mz4JYd3cpAXUvltnshG82%2BtpNwLdjXys2F50X26ND9k%3D&reserved=0<https://github.com/apache/hive/pull/4127>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F4127&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mz4JYd3cpAXUvltnshG82%2BtpNwLdjXys2F50X26ND9k%3D&reserved=0<https://github.com/apache/hive/pull/4127>>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F4127&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mz4JYd3cpAXUvltnshG82%2BtpNwLdjXys2F50X26ND9k%3D&reserved=0<https://github.com/apache/hive/pull/4127>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F4127&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mz4JYd3cpAXUvltnshG82%2BtpNwLdjXys2F50X26ND9k%3D&reserved=0<https://github.com/apache/hive/pull/4127>>> Reverting this commit for
> > > now.
> > > >
> > > > Thanks,
> > > > Aman.
> > > > ________________________________
> > > > From: Aman Raj <ra...@microsoft.com.INVALID>
> > > > Sent: Monday, March 20, 2023 8:28 AM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > Sure Vihang, will look at the other ones. You can pick this up.
> > > >
> > > > Thanks,
> > > > Aman.
> > > >
> > > > Get Outlook for Android<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MlHi3cH0KntRbaDbalzAZzIDfLZ8mhtWLZMewJO2iYY%3D&reserved=0<https://aka.ms/AAb9ysg>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MlHi3cH0KntRbaDbalzAZzIDfLZ8mhtWLZMewJO2iYY%3D&reserved=0<https://aka.ms/AAb9ysg>>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MlHi3cH0KntRbaDbalzAZzIDfLZ8mhtWLZMewJO2iYY%3D&reserved=0<https://aka.ms/AAb9ysg>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MlHi3cH0KntRbaDbalzAZzIDfLZ8mhtWLZMewJO2iYY%3D&reserved=0<https://aka.ms/AAb9ysg>>>>
> > > > ________________________________
> > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > Sent: Monday, March 20, 2023 7:58:48 AM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > I think we should revert offending commits first to unblock the
> branch.
> > > We
> > > > can create followup tickets to determine if these fixes are blockers
> > for
> > > > 3.2 release and if yes, we should merge them the right way with a
> green
> > > > test run. Fixing forward always comes with the risk that it
> introduces
> > > new
> > > > test failures.
> > > >
> > > > Thanks for all your efforts on this Aman.
> > > >
> > > > I can take a look at
> > testBootstrapReplLoadRetryAfterFailureForPartitions
> > > if
> > > > you haven’t already started on it.
> > > >
> > > > Thanks,
> > > > Vihang
> > > >
> > > > On Sun, Mar 19, 2023 at 10:09 PM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hi Vihang/community,
> > > > >
> > > > > Thanks a lot Vihang for working on the major test failure. This
> > blocked
> > > > > more than 35 test cases. Now we are down to the final 4 failures. I
> > > have
> > > > > analyzed some of them and here they are  (Link :
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-4067%2F12%2Ftests&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mvG7GM8Aew%2Bw6%2FYI5UFvRjXYwO5ClXdhpb2grai6M%2Bo%3D&reserved=0<http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests>
> <
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-4067%2F12%2Ftests&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mvG7GM8Aew%2Bw6%2FYI5UFvRjXYwO5ClXdhpb2grai6M%2Bo%3D&reserved=0<http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests>
> >
> > > > )<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-4067%2F12%2Ftests&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mvG7GM8Aew%2Bw6%2FYI5UFvRjXYwO5ClXdhpb2grai6M%2Bo%3D&reserved=0<http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests>
> <
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-4067%2F12%2Ftests&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mvG7GM8Aew%2Bw6%2FYI5UFvRjXYwO5ClXdhpb2grai6M%2Bo%3D&reserved=0<http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests>
> >
> > > > >
> > > > > :
> > > > >
> > > > >   1.
> > > > > multi_in_clause - This was committed in HIVE-21685 without
> validating
> > > the
> > > > > scenario.
> > > > > This fails because Hive is not able to parse
> > > > > explain cbo
> > > > > select * from very_simple_table_for_in_test where name IN('g','r')
> > AND
> > > > > name IN('a','b')
> > > > > If we want this to work, I am able to do it in my local. We have 2
> > > > options
> > > > > :
> > > > > a. Either revert HIVE-21685 since this scenario was not validated
> > back
> > > > > then before adding this test.
> > > > > b. This fix was present in
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rU67fZ6tiWH6jqf5CCdPsXLZYfInTCmCyXPVdHL98i8%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-20718>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rU67fZ6tiWH6jqf5CCdPsXLZYfInTCmCyXPVdHL98i8%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-20718>>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rU67fZ6tiWH6jqf5CCdPsXLZYfInTCmCyXPVdHL98i8%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-20718>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rU67fZ6tiWH6jqf5CCdPsXLZYfInTCmCyXPVdHL98i8%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-20718>>> but to cherry pick
> > > this
> > > > > we need to cherry pick
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8At9kvhdXpl9eS%2Fl%2F%2FBEPiRy8ArTH9YIBo3E%2BDGNe4U%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-17040>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8At9kvhdXpl9eS%2Fl%2F%2FBEPiRy8ArTH9YIBo3E%2BDGNe4U%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-17040>>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8At9kvhdXpl9eS%2Fl%2F%2FBEPiRy8ArTH9YIBo3E%2BDGNe4U%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-17040>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8At9kvhdXpl9eS%2Fl%2F%2FBEPiRy8ArTH9YIBo3E%2BDGNe4U%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-17040>>>
> > > > > since HIVE-20718<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rU67fZ6tiWH6jqf5CCdPsXLZYfInTCmCyXPVdHL98i8%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-20718>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rU67fZ6tiWH6jqf5CCdPsXLZYfInTCmCyXPVdHL98i8%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-20718>>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rU67fZ6tiWH6jqf5CCdPsXLZYfInTCmCyXPVdHL98i8%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-20718>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870370841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rU67fZ6tiWH6jqf5CCdPsXLZYfInTCmCyXPVdHL98i8%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-20718>>>> has a
> > > > > lot of merge conflicts with  HIVE-17040<
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zNDsg6M%2Bvdww7YAeMnzjgluAwbCJcCI%2FgSm4UPnqg00%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-17040>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zNDsg6M%2Bvdww7YAeMnzjgluAwbCJcCI%2FgSm4UPnqg00%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-17040>>
> > > > ><
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zNDsg6M%2Bvdww7YAeMnzjgluAwbCJcCI%2FgSm4UPnqg00%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-17040>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zNDsg6M%2Bvdww7YAeMnzjgluAwbCJcCI%2FgSm4UPnqg00%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-17040>>>. But after cherry
> > > > > picking these we have other failures to fix.
> > > > >   2.
> > > > > current_date_timestamp.q - This breaking change was committed in
> > > > > HIVE-21388 without validation.
> > > > > The failure is because again Hive is not able to parse
> > > > > explain cbo select current_timestamp() from alltypesorc
> > > > > The solution or revert option is same as point 1.
> > > > >   3.
> > > > > testBootstrapReplLoadRetryAfterFailureForPartitions() - This I have
> > not
> > > > > investigated till now.
> > > > >   4.
> > > > > mm_all.q - This I have not investigated till now.
> > > > >
> > > > > Thanks,
> > > > > Aman.
> > > > > ________________________________
> > > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > > Sent: Friday, March 17, 2023 8:42 PM
> > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > >
> > > > > Just wanted to close the loop on the TestMiniSparkOnYarnCliDriver
> > test
> > > > > failures. We will be able to re-enable most of them back on
> branch-3.
> > > The
> > > > > ones which were disabled are being tracked separately in a
> different
> > > > ticket
> > > > > <
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tAgcVpy1v1aVIyEXZ2hRkfZsOsI%2FpWGchxI3wp3%2FfBg%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-27146>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tAgcVpy1v1aVIyEXZ2hRkfZsOsI%2FpWGchxI3wp3%2FfBg%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-27146>>
> > > > ><
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tAgcVpy1v1aVIyEXZ2hRkfZsOsI%2FpWGchxI3wp3%2FfBg%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-27146>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tAgcVpy1v1aVIyEXZ2hRkfZsOsI%2FpWGchxI3wp3%2FfBg%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-27146>>>
> > > > > but they don't look like
> > > > > a blocker.
> > > > >
> > > > > Hi Aman,
> > > > >
> > > > > Do you know how close are we to reopening branch-3?
> > > > >
> > > > > Thanks,
> > > > > Vihang
> > > > >
> > > > > On Sat, Mar 4, 2023 at 7:23 PM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > > > Or you can cd into itests and run the command you are using. Just
> > > > another
> > > > > > way I run.
> > > > > >
> > > > > > Thanks,
> > > > > > Aman.
> > > > > > Get Outlook for Android<
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dRN1S%2Bg5kyoHteRAKv3Tp8bsf%2FuHue53EEFR197eHg8%3D&reserved=0<https://aka.ms/AAb9ysg>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dRN1S%2Bg5kyoHteRAKv3Tp8bsf%2FuHue53EEFR197eHg8%3D&reserved=0<https://aka.ms/AAb9ysg>>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dRN1S%2Bg5kyoHteRAKv3Tp8bsf%2FuHue53EEFR197eHg8%3D&reserved=0<https://aka.ms/AAb9ysg>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dRN1S%2Bg5kyoHteRAKv3Tp8bsf%2FuHue53EEFR197eHg8%3D&reserved=0<https://aka.ms/AAb9ysg>>>
> > > > > >
> > > > > > ________________________________
> > > > > > From: Aman Raj <ra...@microsoft.com>
> > > > > > Sent: Saturday, March 4, 2023 7:20:36 PM
> > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> stability
> > > > > >
> > > > > > Hi Vihang,
> > > > > >
> > > > > > Thanks a lot for working on this. Can you try using
> > -Pqsplits,itests.
> > > > > > Also, I usually give a -o option after doing a clean install.
> > > > > >
> > > > > > Thanks,
> > > > > > Aman.
> > > > > >
> > > > > > Get Outlook for Android<
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA<https://aka.ms/AAb9ysg>iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dRN1S%2Bg5kyoHteRAKv3Tp8bsf%2FuHue53EEFR197eHg8%3D&reserved=0
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dRN1S%2Bg5kyoHteRAKv3Tp8bsf%2FuHue53EEFR197eHg8%3D&reserved=0<https://aka.ms/AAb9ysg>>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dRN1S%2Bg5kyoHteRAKv3Tp8bsf%2FuHue53EEFR197eHg8%3D&reserved=0<https://aka.ms/AAb9ysg>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dRN1S%2Bg5kyoHteRAKv3Tp8bsf%2FuHue53EEFR197eHg8%3D&reserved=0<https://aka.ms/AAb9ysg>>>
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > > > Sent: Saturday, 4 March, 2023, 11:35
> > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> stability
> > > > > >
> > > > > > [You don't often get email from vihangk1@apache.org. Learn why
> > this
> > > is
> > > > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > > > >
> > > > > > Just to update on the HoS test failures for
> > > > > TestMiniSparkOnYarnCliDriver, I
> > > > > > think I was finally able to resolve them (at least on local). I
> had
> > > to
> > > > > > revert HIVE-21044 because it was causing OOM for those tests.
> Also,
> > > in
> > > > > > order for these tests to work we will have to downgrade netty
> from
> > > > > > 4.1.69.Final to 4.1.51.Final. I understand that we had upgraded
> > netty
> > > > > from
> > > > > > 4.1.17.Final to 4.1.69.Final for CVEs but the highest netty
> version
> > > > that
> > > > > we
> > > > > > can support without breaking HoS is 4.1.51.Final. Note that
> > > > 4.1.51.Final
> > > > > > includes many of the CVEs which affected 4.1.17.Final so we are
> > still
> > > > in
> > > > > a
> > > > > > better place than branch-3.1. Unfortunately, there is no good way
> > to
> > > > make
> > > > > > HoS work with a higher netty version so I think we should
> downgrade
> > > the
> > > > > > netty version to 4.1.51.Final for now and look at more options to
> > > > upgrade
> > > > > > it 4.1.69.Final in a separate ticket.
> > > > > >
> > > > > > I still need to understand why the tests which are working for me
> > > > locally
> > > > > > don't work on the PR job. I tried running the split test classes
> > > using
> > > > > the
> > > > > > following command. Is that the right way to simulate builds from
> > the
> > > PR
> > > > > > job? Let me know if anyone has more ideas.
> > > > > >
> > > > > > mvn test
> > > > > >
> > -Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
> > > > > > -Pqsplits
> > > > > >
> > > > > > Thanks,
> > > > > > Vihang
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <
> > > zabetak@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > Thanks Aman for bringing this up and also for cleaning up after
> > > > others
> > > > > (I
> > > > > > > saw that you raised tickets and PRs for addressing the
> failures).
> > > > > > >
> > > > > > > Many thanks to Vihang as well for helping out. Regarding flaky
> > > tests,
> > > > > yes
> > > > > > > we should disable them as soon as we see them.
> > > > > > > There have been some other discussions on how to approach flaky
> > > tests
> > > > > the
> > > > > > > more recent I could find is here [1].
> > > > > > >
> > > > > > > Best,
> > > > > > > Stamatis
> > > > > > >
> > > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xuJ4wBNACxFIpgcLmSG%2FZItXYv4ptD9%2BVNljKeBrbgQ%3D&reserved=0<https://lists.apache.org/thread/lv3bhlfoq8fwd9dwyjf7g4nx32wtrygv>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xuJ4wBNACxFIpgcLmSG%2FZItXYv4ptD9%2BVNljKeBrbgQ%3D&reserved=0<https://lists.apache.org/thread/lv3bhlfoq8fwd9dwyjf7g4nx32wtrygv>>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xuJ4wBNACxFIpgcLmSG%2FZItXYv4ptD9%2BVNljKeBrbgQ%3D&reserved=0<https://lists.apache.org/thread/lv3bhlfoq8fwd9dwyjf7g4nx32wtrygv>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xuJ4wBNACxFIpgcLmSG%2FZItXYv4ptD9%2BVNljKeBrbgQ%3D&reserved=0<https://lists.apache.org/thread/lv3bhlfoq8fwd9dwyjf7g4nx32wtrygv>>>
> > > > > > >
> > > > > > > On Fri, Feb 17, 2023 at 4:37 AM Aman Raj
> > > > <rajaman@microsoft.com.invalid
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi team,
> > > > > > > >
> > > > > > > > Thanks Vihang for looking into this. I have commented on the
> > JIRA
> > > > you
> > > > > > > > created.
> > > > > > > >
> > > > > > > > Just to bring everyone's notice, I have seen that there has
> > been
> > > a
> > > > > > couple
> > > > > > > > of pushes to branch-3, which has lead to 5 more new test
> > > failures.
> > > > > The
> > > > > > > test
> > > > > > > > failures are in orc_merge1, orc_merge2, orc_merge3,
> orc_merge4
> > > and
> > > > > > > > orc_merge10. These tests did not use to fail before. I would
> > > > > sincerely
> > > > > > > urge
> > > > > > > > the community to raise a PR against branch-3, so that the
> > Jenkins
> > > > > > > pipeline
> > > > > > > > can run and then only merge things to branch-3. We had 2900+
> > > > failures
> > > > > > > when
> > > > > > > > we started 2 months back and now having brought it down to
> less
> > > > than
> > > > > > 15,
> > > > > > > > new failures again has pushed us back in this effort.
> > > > > > > >
> > > > > > > > I would like to thank everyone who has participated in this
> > > effort
> > > > > and
> > > > > > > > made it possible till this stage. Also, if the contributors
> can
> > > > take
> > > > > > > > ownership of these new test case failures and fix them, it
> will
> > > be
> > > > of
> > > > > > > great
> > > > > > > > help.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Aman.
> > > > > > > > ________________________________
> > > > > > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > > > > > Sent: Friday, February 17, 2023 6:10 AM
> > > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> > > stability
> > > > > > > >
> > > > > > > > [You don't often get email from vihangk1@apache.org. Learn
> why
> > > > this
> > > > > is
> > > > > > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > > > > > >
> > > > > > > > Hi Aman,
> > > > > > > >
> > > > > > > > I created
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Hl11jwWPBSez5YvxKpxVhfLPxO4TzIAaTBgEtC5n4KE%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-27087>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Hl11jwWPBSez5YvxKpxVhfLPxO4TzIAaTBgEtC5n4KE%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-27087>>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Hl11jwWPBSez5YvxKpxVhfLPxO4TzIAaTBgEtC5n4KE%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-27087>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Hl11jwWPBSez5YvxKpxVhfLPxO4TzIAaTBgEtC5n4KE%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-27087>>>
> > > > > > > > to look into
> > > > > > > > TestMiniSparkOnYarnCliDriver failures. I have a working
> theory
> > of
> > > > > what
> > > > > > > > might be going on there. I am still investigating what is the
> > > right
> > > > > way
> > > > > > > to
> > > > > > > > fix it though.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Vihang
> > > > > > > >
> > > > > > > > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj
> > > > > > <rajaman@microsoft.com.invalid
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Vihang,
> > > > > > > > >
> > > > > > > > > Yes the tests are failing locally as well with the same
> > issue.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Aman.
> > > > > > > > >
> > > > > > > > > Get Outlook for Android<
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dRN1S%2Bg5kyoHteRAKv3Tp8bsf%2FuHue53EEFR197eHg8%3D&reserved=0<https://aka.ms/AAb9ysg>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dRN1S%2Bg5kyoHteRAKv3Tp8bsf%2FuHue53EEFR197eHg8%3D&reserved=0<https://aka.ms/AAb9ysg>>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dRN1S%2Bg5kyoHteRAKv3Tp8bsf%2FuHue53EEFR197eHg8%3D&reserved=0<https://aka.ms/AAb9ysg>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dRN1S%2Bg5kyoHteRAKv3Tp8bsf%2FuHue53EEFR197eHg8%3D&reserved=0<https://aka.ms/AAb9ysg>>>
> > > > > > > > >
> > > > > > > > > ________________________________
> > > > > > > > > From: Vihang Karajgaonkar
> > > > > <vihang.karajgaonkar@databricks.com.INVALID
> > > > > > >
> > > > > > > > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> > > > stability
> > > > > > > > >
> > > > > > > > > [You don't often get email from
> > > > > > > > vihang.karajgaonkar@databricks.com.invalid.
> > > > > > > > > Learn why this is important at
> > > > > > > > > https://aka.ms/LearnAboutSenderIdentification ]
> > > > > > > > >
> > > > > > > > > Thanks a lot Stamatis for starting this thread. I really
> > > > appreciate
> > > > > > all
> > > > > > > > the
> > > > > > > > > efforts to stabilize branch-3 to get it to a releasable
> state
> > > > and I
> > > > > > > agree
> > > > > > > > > that we should get it to a green state before opening it
> for
> > > PRs
> > > > > not
> > > > > > > > > related to test failures. I can help with the effort as
> well.
> > > > > > > > >
> > > > > > > > > If we want to get the branch back to green state soon, have
> > we
> > > > > > > considered
> > > > > > > > > disabling the tests which are clearly flaky? (e.g pass on
> > some
> > > > > builds
> > > > > > > and
> > > > > > > > > fail on the other build with no new code changes). If we
> > don't
> > > do
> > > > > > that,
> > > > > > > > we
> > > > > > > > > will keep playing whack a mole with those tests. I propose
> > for
> > > > such
> > > > > > > tests
> > > > > > > > > we should disable them and create tickets to unflake them
> > > > > separately.
> > > > > > > > This
> > > > > > > > > will help us get back to a green state faster.
> > > > > > > > >
> > > > > > > > > Hi Aman,
> > > > > > > > > For TestMiniSparkOnYarnCliDriver failures, you probably
> > should
> > > > also
> > > > > > > look
> > > > > > > > > into the spark driver/application logs and see if there are
> > > > > > > > infrastructure
> > > > > > > > > errors (e.g OOMs). Are these tests failing when you run
> > > locally?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Vihang
> > > > > > > > >
> > > > > > > > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj
> > > > > > <rajaman@microsoft.com.invalid
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1,
> > > > > > > > > > Thanks Stamatis and Lazlo for helping in the test case
> > fixes
> > > > till
> > > > > > > now.
> > > > > > > > > >
> > > > > > > > > > Team,
> > > > > > > > > > I need help in fixing the following tests in Hive. I have
> > > tried
> > > > > > > > different
> > > > > > > > > > approaches but no luck till now.
> > > > > > > > > > I am facing some issues in fixing the following tests :
> > > > > > > > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > > > > > > > >
> > > > > > > > > > Issue :
> > > > > > > > > > PREHOOK: Input: default@src
> > > > > > > > > > PREHOOK: Output: default@src
> > > > > > > > > > Failed to monitor Job[-1] with exception
> > > > > > > > > > 'java.lang.IllegalStateException(Connection to remote
> Spark
> > > > > driver
> > > > > > > was
> > > > > > > > > > lost)' Last known state = SENT
> > > > > > > > > > Failed to execute spark task, with exception
> > > > > > > > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > > > > > > > FAILED: Execution Error, return code 1 from
> > > > > > > > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC
> channel
> > > is
> > > > > > > closed.
> > > > > > > > > >
> > > > > > > > > > History :
> > > > > > > > > > Initially the tests had failed with errors which I fixed
> in
> > > the
> > > > > > > > following
> > > > > > > > > > task :
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=M8%2BRlmAR%2FrDH11D30plalsuWO76o8U3OAqPh9CXhz2s%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26940>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=M8%2BRlmAR%2FrDH11D30plalsuWO76o8U3OAqPh9CXhz2s%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26940>>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=M8%2BRlmAR%2FrDH11D30plalsuWO76o8U3OAqPh9CXhz2s%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26940>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=M8%2BRlmAR%2FrDH11D30plalsuWO76o8U3OAqPh9CXhz2s%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26940>>>
> > > > > > > > > >
> > > > > > > > > > Does anyone know what the issue is here ? There are 6-7
> > > > failures
> > > > > > > > because
> > > > > > > > > > of this test case. Link to the failed test cases for the
> > > > > > stacktrace :
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VuAjpgx7Kux7SoeLysAqfkEfSRKDr%2FMTmuy3yJh30CM%3D&reserved=0<http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3949/2/tests/>
> <
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VuAjpgx7Kux7SoeLysAqfkEfSRKDr%2FMTmuy3yJh30CM%3D&reserved=0<http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3949/2/tests/>
> >
> > > > <
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VuAjpgx7Kux7SoeLysAqfkEfSRKDr%2FMTmuy3yJh30CM%3D&reserved=0<http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3949/2/tests/>
> <
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870527054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VuAjpgx7Kux7SoeLysAqfkEfSRKDr%2FMTmuy3yJh30CM%3D&reserved=0<http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3949/2/tests/>
> >
> > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Aman.
> > > > > > > > > >
> > > > > > > > > > ________________________________
> > > > > > > > > > From: László Bodor <bo...@gmail.com>
> > > > > > > > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > > > > Subject: [EXTERNAL] Re: Branch-3 backports and build
> > > stability
> > > > > > > > > >
> > > > > > > > > > +1
> > > > > > > > > > also, if I merged something that I thought was for test
> > > > stability
> > > > > > > (but
> > > > > > > > > > instead it was a feature), excuse me :)
> > > > > > > > > > for reference, the whole green test initiative is tracked
> > > under
> > > > > > this
> > > > > > > > > > umbrella:
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870683885%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=t3Rv91iUA8DX8bq2wQ7Nre9aZHIYUdI7pKk8LSFKKz4%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26836>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870683885%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=t3Rv91iUA8DX8bq2wQ7Nre9aZHIYUdI7pKk8LSFKKz4%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26836>>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870683885%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=t3Rv91iUA8DX8bq2wQ7Nre9aZHIYUdI7pKk8LSFKKz4%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26836>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C89b57ea73d084b2bb42a08db50bde915%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638192548870683885%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=t3Rv91iUA8DX8bq2wQ7Nre9aZHIYUdI7pKk8LSFKKz4%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26836>>>
> > > > > > > > > >
> > > > > > > > > > Stamatis Zampetakis <za...@gmail.com> ezt írta
> (időpont:
> > > > 2023.
> > > > > > > febr.
> > > > > > > > > 7.,
> > > > > > > > > > K, 12:09):
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > The build in branch-3 is not yet green; there are ~25
> > test
> > > > > > > failures.
> > > > > > > > It
> > > > > > > > > > is
> > > > > > > > > > > a common practice that we shouldn't push changes on top
> > of
> > > a
> > > > > > broken
> > > > > > > > > build
> > > > > > > > > > > unless they are addressing test failures.
> > > > > > > > > > >
> > > > > > > > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo
> > > > Bodor)
> > > > > > are
> > > > > > > > > > working
> > > > > > > > > > > hard to stabilize the build for quite some time now. If
> > you
> > > > > want
> > > > > > to
> > > > > > > > > help
> > > > > > > > > > > out then start by reviewing, merging, and fixing things
> > > > around
> > > > > > test
> > > > > > > > > > > failures.
> > > > > > > > > > >
> > > > > > > > > > > It's not yet the time to bring new features, upgrades,
> > > bugs,
> > > > > > etc.,
> > > > > > > in
> > > > > > > > > > > branch-3. I would encourage  committers to not approve
> > such
> > > > > > changes
> > > > > > > > > till
> > > > > > > > > > we
> > > > > > > > > > > get back to a stable branch.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Stamatis
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [EXTERNAL] Re: Branch-3 backports and build stability

[Vihang Karajgaonkar <vi...@gmail.com>]

Thanks Aman. I thought all the changes in release 3.2.0 were listed under
https://issues.apache.org/jira/browse/HIVE-26751 and I saw them all
resolved. Do you know which additional tickets need to go in branch-3 after
we backport the branch-3.1 fixes in branch-3?

On Tue, May 9, 2023 at 11:20 AM Aman Raj <ra...@microsoft.com.invalid>
wrote:

> Hi Vihang,
>
> We only have 4 tickets remaining to be backported from branch-3.1 to
> branch-3. It will be completed next week.
>
> But there are a lot of new tickets that will go into release 3.2.0 on top
> of this. I was thinking of not cutting a release candidate now since it
> would mean that we only backport changes into that release candidate
> branch. This would again mean that if people commit only to branch-3 or the
> release branch, there will again be a lot of difference in these two
> branches when someone picks up the next release.
>
> Instead I am thinking that we should backport new changes to branch-3 and
> then only cut the release candidate. Please let me know your thoughts. If
> we agree that changes need to go into the new release candidate branch
> only, I am okay with that (I do not prefer it btw)
>
> Thanks,
> Aman.
>
> Get Outlook for Android<https://aka.ms/AAb9ysg>
> ________________________________
> From: vihang karajgaonkar <vi...@apache.org>
> Sent: Monday, May 8, 2023 4:57:24 AM
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
>
> Hi Aman,
>
> I know you are backporting the branch-3.1 commits to branch-3. How close
> are you with finishing with them. Is there anything that we can help with
> to get it over the finish line?
>
> I am interested to know how close are we to cutting the branch for 3.2.0?
> Do you think we can have a release candidate this week?
>
> Thanks,
> Vihang
>
> On Thu, Mar 30, 2023 at 2:18 AM Stamatis Zampetakis <za...@gmail.com>
> wrote:
>
> > Huge thanks to everyone involved it is great to see the branch-3 in
> stable
> > state. As other people mentioned let's keep it that way!
> >
> > As far as it concerns back ports please be particularly cautious with
> > anything that touches the metastore schema and Thrift APIs.
> >
> > Best,
> > Stamatis
> >
> > On Wed, Mar 29, 2023, 4:36 AM vihang karajgaonkar <vi...@apache.org>
> > wrote:
> >
> > > Thanks a lot Aman for all your efforts on this. Really appreciate the
> > > initiative and all your hard work on this.
> > >
> > > I would like to request that all the committers should follow the merge
> > > process of master branch to merge PRs in branch-3. If there are any
> test
> > > failures which seem unrelated, please do not ignore them. One can run
> the
> > > flaky
> > > test runner <
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fjob%2Fhive-flaky-check%2F&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NFcfXybje7zooPXkkN9UnSQIexxjKJ9wQR%2B0nrXk4e0%3D&reserved=0
> <http://ci.hive.apache.org/job/hive-flaky-check/>> to make
> > sure
> > > that test is indeed flaky. If the test is found to be flaky a
> > > ticket should be created to disable it. A separate ticket should be
> > created
> > > to deflake it and you can mention the original author or previous
> commit
> > > author who changed the test on that ticket to get help since they
> likely
> > > have the most context around that test. Once the flaky test is disabled
> > and
> > > we have a green CI job run, we should merge the PR. If others have any
> > > suggestions to improve this process please chime in.
> > >
> > > Thanks,
> > > Vihang
> > >
> > > On Tue, Mar 28, 2023 at 10:55 PM Aman Raj
> <rajaman@microsoft.com.invalid
> > >
> > > wrote:
> > >
> > > > Hi community,
> > > >
> > > > This is to notify that we have a green branch-3 now. The entire
> effort
> > of
> > > > fixing branch-3 test cases took around 4 months and as a team we
> > managed
> > > to
> > > > fix 2900+ test failures on branch-3. The entire effort can be tracked
> > > here
> > > > HIVE-26836<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DXjdb%2FLcUztu6Kr2phfhuVgdP652Yvlddq1QmtTndAw%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-26836>>. We are
> > > > ready to push new features and improvements on branch-3 now.
> > > >
> > > > I really want to thank Vihang Karajgaonkar, Chris Nauroth, Lazlo
> Bodor,
> > > > Stamatis Zampetakis and Sankar Hariappan without whom this would not
> at
> > > all
> > > > have been possible. As a team we stuck together and participated in
> > > reviews
> > > > and actively suggested improvements which really helped in fixing
> some
> > > > major test failures.
> > > >
> > > > I would sincerely request that going further it should be made a
> point
> > to
> > > > merge things into branch-3 only if we have a green Jenkins pipeline.
> > > >
> > > > The next step would be to backport changes from branch-3.1 (From
> where
> > > > Hive-3.1.3 release was made) to branch-3. This would ensure that we
> do
> > > not
> > > > miss any specific ticket which went into Hive-3.1.3. I will take care
> > of
> > > > this. We can parallelly start pushing additional changes on branch-3.
> > > There
> > > > are approximately 25 tickets that need to be backported in this
> effort
> > > (Of
> > > > backporting changes from branch-3.1). I have made a note here<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fspreadsheets%2Fd%2F1K0U-vxLRZEs13oBzYBlVyK8dMMNthgXL5VEgzLRbeKs%2Fedit%3Fusp%3Dsharing&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UB3VwQkRFsNlPsZYuGuzt1uQwp20L%2BmgLAD8CvopzUw%3D&reserved=0
> <
> https://docs.google.com/spreadsheets/d/1K0U-vxLRZEs13oBzYBlVyK8dMMNthgXL5VEgzLRbeKs/edit?usp=sharing
> >
> > > > >
> > > >
> > > > Again, thanks a lot to everyone who supported and participated in
> this
> > > > effort. Lets make this 3.2.0 Hive release happen!!
> > > >
> > > > Thanks,
> > > > Aman.
> > > >
> > > > ________________________________
> > > > From: Aman Raj <ra...@microsoft.com.INVALID>
> > > > Sent: Monday, March 20, 2023 9:21 AM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > Hi Vihang/community,
> > > >
> > > > Found the ticket which broke mm_all.q. This issue comes because of
> > > > HIVE-20182. Works in my local and on the Jenkins pipeline as well.
> > Link :
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F4127&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aZRb8agmteUTddKJaBEtUC3EMd6ZibPfIu2nB9yOptk%3D&reserved=0
> <https://github.com/apache/hive/pull/4127>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F4127&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aZRb8agmteUTddKJaBEtUC3EMd6ZibPfIu2nB9yOptk%3D&reserved=0
> <https://github.com/apache/hive/pull/4127>> Reverting this commit for
> > > now.
> > > >
> > > > Thanks,
> > > > Aman.
> > > > ________________________________
> > > > From: Aman Raj <ra...@microsoft.com.INVALID>
> > > > Sent: Monday, March 20, 2023 8:28 AM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > Sure Vihang, will look at the other ones. You can pick this up.
> > > >
> > > > Thanks,
> > > > Aman.
> > > >
> > > > Get Outlook for Android<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5gG0C%2BmQpxQOV%2BCQChCGbnlMv5e9BCS7YZ5xO9y9zaw%3D&reserved=0
> <https://aka.ms/AAb9ysg>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5gG0C%2BmQpxQOV%2BCQChCGbnlMv5e9BCS7YZ5xO9y9zaw%3D&reserved=0
> <https://aka.ms/AAb9ysg>>>
> > > > ________________________________
> > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > Sent: Monday, March 20, 2023 7:58:48 AM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > I think we should revert offending commits first to unblock the
> branch.
> > > We
> > > > can create followup tickets to determine if these fixes are blockers
> > for
> > > > 3.2 release and if yes, we should merge them the right way with a
> green
> > > > test run. Fixing forward always comes with the risk that it
> introduces
> > > new
> > > > test failures.
> > > >
> > > > Thanks for all your efforts on this Aman.
> > > >
> > > > I can take a look at
> > testBootstrapReplLoadRetryAfterFailureForPartitions
> > > if
> > > > you haven’t already started on it.
> > > >
> > > > Thanks,
> > > > Vihang
> > > >
> > > > On Sun, Mar 19, 2023 at 10:09 PM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hi Vihang/community,
> > > > >
> > > > > Thanks a lot Vihang for working on the major test failure. This
> > blocked
> > > > > more than 35 test cases. Now we are down to the final 4 failures. I
> > > have
> > > > > analyzed some of them and here they are  (Link :
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-4067%2F12%2Ftests&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Vjjsob4PG0%2FU5uDRUEf23WCCfuM622SWcBLEhuX1qKo%3D&reserved=0
> <
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests
> >
> > > > )<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-4067%2F12%2Ftests&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Vjjsob4PG0%2FU5uDRUEf23WCCfuM622SWcBLEhuX1qKo%3D&reserved=0
> <
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests
> >
> > > > >
> > > > > :
> > > > >
> > > > >   1.
> > > > > multi_in_clause - This was committed in HIVE-21685 without
> validating
> > > the
> > > > > scenario.
> > > > > This fails because Hive is not able to parse
> > > > > explain cbo
> > > > > select * from very_simple_table_for_in_test where name IN('g','r')
> > AND
> > > > > name IN('a','b')
> > > > > If we want this to work, I am able to do it in my local. We have 2
> > > > options
> > > > > :
> > > > > a. Either revert HIVE-21685 since this scenario was not validated
> > back
> > > > > then before adding this test.
> > > > > b. This fix was present in
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UyGzmPpRJjFM1AciK0W8TMZ3Y%2BNrAtfQcyCEyrFu7uM%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-20718>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UyGzmPpRJjFM1AciK0W8TMZ3Y%2BNrAtfQcyCEyrFu7uM%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-20718>> but to cherry pick
> > > this
> > > > > we need to cherry pick
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pwa7HtgXaeSKVrDgb6Bh8Fg2tBYdDKmh1O6NCA0qPrw%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-17040>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pwa7HtgXaeSKVrDgb6Bh8Fg2tBYdDKmh1O6NCA0qPrw%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-17040>>
> > > > > since HIVE-20718<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UyGzmPpRJjFM1AciK0W8TMZ3Y%2BNrAtfQcyCEyrFu7uM%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-20718>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UyGzmPpRJjFM1AciK0W8TMZ3Y%2BNrAtfQcyCEyrFu7uM%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-20718>>> has a
> > > > > lot of merge conflicts with  HIVE-17040<
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pwa7HtgXaeSKVrDgb6Bh8Fg2tBYdDKmh1O6NCA0qPrw%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-17040>
> > > > ><
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pwa7HtgXaeSKVrDgb6Bh8Fg2tBYdDKmh1O6NCA0qPrw%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-17040>>. But after cherry
> > > > > picking these we have other failures to fix.
> > > > >   2.
> > > > > current_date_timestamp.q - This breaking change was committed in
> > > > > HIVE-21388 without validation.
> > > > > The failure is because again Hive is not able to parse
> > > > > explain cbo select current_timestamp() from alltypesorc
> > > > > The solution or revert option is same as point 1.
> > > > >   3.
> > > > > testBootstrapReplLoadRetryAfterFailureForPartitions() - This I have
> > not
> > > > > investigated till now.
> > > > >   4.
> > > > > mm_all.q - This I have not investigated till now.
> > > > >
> > > > > Thanks,
> > > > > Aman.
> > > > > ________________________________
> > > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > > Sent: Friday, March 17, 2023 8:42 PM
> > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > >
> > > > > Just wanted to close the loop on the TestMiniSparkOnYarnCliDriver
> > test
> > > > > failures. We will be able to re-enable most of them back on
> branch-3.
> > > The
> > > > > ones which were disabled are being tracked separately in a
> different
> > > > ticket
> > > > > <
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=kMZtW1GSTclD0RjnALCDs6KoT%2F%2B13PFbNvD8psiU%2BCQ%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-27146>
> > > > ><
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=kMZtW1GSTclD0RjnALCDs6KoT%2F%2B13PFbNvD8psiU%2BCQ%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-27146>>
> > > > > but they don't look like
> > > > > a blocker.
> > > > >
> > > > > Hi Aman,
> > > > >
> > > > > Do you know how close are we to reopening branch-3?
> > > > >
> > > > > Thanks,
> > > > > Vihang
> > > > >
> > > > > On Sat, Mar 4, 2023 at 7:23 PM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > > > Or you can cd into itests and run the command you are using. Just
> > > > another
> > > > > > way I run.
> > > > > >
> > > > > > Thanks,
> > > > > > Aman.
> > > > > > Get Outlook for Android<
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5gG0C%2BmQpxQOV%2BCQChCGbnlMv5e9BCS7YZ5xO9y9zaw%3D&reserved=0
> <https://aka.ms/AAb9ysg>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5gG0C%2BmQpxQOV%2BCQChCGbnlMv5e9BCS7YZ5xO9y9zaw%3D&reserved=0
> <https://aka.ms/AAb9ysg>>
> > > > > >
> > > > > > ________________________________
> > > > > > From: Aman Raj <ra...@microsoft.com>
> > > > > > Sent: Saturday, March 4, 2023 7:20:36 PM
> > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> stability
> > > > > >
> > > > > > Hi Vihang,
> > > > > >
> > > > > > Thanks a lot for working on this. Can you try using
> > -Pqsplits,itests.
> > > > > > Also, I usually give a -o option after doing a clean install.
> > > > > >
> > > > > > Thanks,
> > > > > > Aman.
> > > > > >
> > > > > > Get Outlook for Android<
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5gG0C%2BmQpxQOV%2BCQChCGbnlMv5e9BCS7YZ5xO9y9zaw%3D&reserved=0
> <https://aka.ms/AAb9ysg>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5gG0C%2BmQpxQOV%2BCQChCGbnlMv5e9BCS7YZ5xO9y9zaw%3D&reserved=0
> <https://aka.ms/AAb9ysg>>
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > > > Sent: Saturday, 4 March, 2023, 11:35
> > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> stability
> > > > > >
> > > > > > [You don't often get email from vihangk1@apache.org. Learn why
> > this
> > > is
> > > > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > > > >
> > > > > > Just to update on the HoS test failures for
> > > > > TestMiniSparkOnYarnCliDriver, I
> > > > > > think I was finally able to resolve them (at least on local). I
> had
> > > to
> > > > > > revert HIVE-21044 because it was causing OOM for those tests.
> Also,
> > > in
> > > > > > order for these tests to work we will have to downgrade netty
> from
> > > > > > 4.1.69.Final to 4.1.51.Final. I understand that we had upgraded
> > netty
> > > > > from
> > > > > > 4.1.17.Final to 4.1.69.Final for CVEs but the highest netty
> version
> > > > that
> > > > > we
> > > > > > can support without breaking HoS is 4.1.51.Final. Note that
> > > > 4.1.51.Final
> > > > > > includes many of the CVEs which affected 4.1.17.Final so we are
> > still
> > > > in
> > > > > a
> > > > > > better place than branch-3.1. Unfortunately, there is no good way
> > to
> > > > make
> > > > > > HoS work with a higher netty version so I think we should
> downgrade
> > > the
> > > > > > netty version to 4.1.51.Final for now and look at more options to
> > > > upgrade
> > > > > > it 4.1.69.Final in a separate ticket.
> > > > > >
> > > > > > I still need to understand why the tests which are working for me
> > > > locally
> > > > > > don't work on the PR job. I tried running the split test classes
> > > using
> > > > > the
> > > > > > following command. Is that the right way to simulate builds from
> > the
> > > PR
> > > > > > job? Let me know if anyone has more ideas.
> > > > > >
> > > > > > mvn test
> > > > > >
> > -Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
> > > > > > -Pqsplits
> > > > > >
> > > > > > Thanks,
> > > > > > Vihang
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <
> > > zabetak@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > Thanks Aman for bringing this up and also for cleaning up after
> > > > others
> > > > > (I
> > > > > > > saw that you raised tickets and PRs for addressing the
> failures).
> > > > > > >
> > > > > > > Many thanks to Vihang as well for helping out. Regarding flaky
> > > tests,
> > > > > yes
> > > > > > > we should disable them as soon as we see them.
> > > > > > > There have been some other discussions on how to approach flaky
> > > tests
> > > > > the
> > > > > > > more recent I could find is here [1].
> > > > > > >
> > > > > > > Best,
> > > > > > > Stamatis
> > > > > > >
> > > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ym1WBuGKRt%2FuTHitzCwR%2FzckJGLPO1XinHZm6kt%2BgKk%3D&reserved=0
> <https://lists.apache.org/thread/lv3bhlfoq8fwd9dwyjf7g4nx32wtrygv>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ym1WBuGKRt%2FuTHitzCwR%2FzckJGLPO1XinHZm6kt%2BgKk%3D&reserved=0
> <https://lists.apache.org/thread/lv3bhlfoq8fwd9dwyjf7g4nx32wtrygv>>
> > > > > > >
> > > > > > > On Fri, Feb 17, 2023 at 4:37 AM Aman Raj
> > > > <rajaman@microsoft.com.invalid
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi team,
> > > > > > > >
> > > > > > > > Thanks Vihang for looking into this. I have commented on the
> > JIRA
> > > > you
> > > > > > > > created.
> > > > > > > >
> > > > > > > > Just to bring everyone's notice, I have seen that there has
> > been
> > > a
> > > > > > couple
> > > > > > > > of pushes to branch-3, which has lead to 5 more new test
> > > failures.
> > > > > The
> > > > > > > test
> > > > > > > > failures are in orc_merge1, orc_merge2, orc_merge3,
> orc_merge4
> > > and
> > > > > > > > orc_merge10. These tests did not use to fail before. I would
> > > > > sincerely
> > > > > > > urge
> > > > > > > > the community to raise a PR against branch-3, so that the
> > Jenkins
> > > > > > > pipeline
> > > > > > > > can run and then only merge things to branch-3. We had 2900+
> > > > failures
> > > > > > > when
> > > > > > > > we started 2 months back and now having brought it down to
> less
> > > > than
> > > > > > 15,
> > > > > > > > new failures again has pushed us back in this effort.
> > > > > > > >
> > > > > > > > I would like to thank everyone who has participated in this
> > > effort
> > > > > and
> > > > > > > > made it possible till this stage. Also, if the contributors
> can
> > > > take
> > > > > > > > ownership of these new test case failures and fix them, it
> will
> > > be
> > > > of
> > > > > > > great
> > > > > > > > help.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Aman.
> > > > > > > > ________________________________
> > > > > > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > > > > > Sent: Friday, February 17, 2023 6:10 AM
> > > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> > > stability
> > > > > > > >
> > > > > > > > [You don't often get email from vihangk1@apache.org. Learn
> why
> > > > this
> > > > > is
> > > > > > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > > > > > >
> > > > > > > > Hi Aman,
> > > > > > > >
> > > > > > > > I created
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6vJLQ2iZLyEfQv%2By8C8se%2F%2BABdUmnOlnmCSOfTEByWg%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-27087>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6vJLQ2iZLyEfQv%2By8C8se%2F%2BABdUmnOlnmCSOfTEByWg%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-27087>>
> > > > > > > > to look into
> > > > > > > > TestMiniSparkOnYarnCliDriver failures. I have a working
> theory
> > of
> > > > > what
> > > > > > > > might be going on there. I am still investigating what is the
> > > right
> > > > > way
> > > > > > > to
> > > > > > > > fix it though.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Vihang
> > > > > > > >
> > > > > > > > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj
> > > > > > <rajaman@microsoft.com.invalid
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Vihang,
> > > > > > > > >
> > > > > > > > > Yes the tests are failing locally as well with the same
> > issue.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Aman.
> > > > > > > > >
> > > > > > > > > Get Outlook for Android<
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FU380OPfIwF6BHqGG85T8XQlWjG6QmDuQfwLytzoE6w%3D&reserved=0
> <https://aka.ms/AAb9ysg>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FU380OPfIwF6BHqGG85T8XQlWjG6QmDuQfwLytzoE6w%3D&reserved=0
> <https://aka.ms/AAb9ysg>>
> > > > > > > > >
> > > > > > > > > ________________________________
> > > > > > > > > From: Vihang Karajgaonkar
> > > > > <vihang.karajgaonkar@databricks.com.INVALID
> > > > > > >
> > > > > > > > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> > > > stability
> > > > > > > > >
> > > > > > > > > [You don't often get email from
> > > > > > > > vihang.karajgaonkar@databricks.com.invalid.
> > > > > > > > > Learn why this is important at
> > > > > > > > > https://aka.ms/LearnAboutSenderIdentification ]
> > > > > > > > >
> > > > > > > > > Thanks a lot Stamatis for starting this thread. I really
> > > > appreciate
> > > > > > all
> > > > > > > > the
> > > > > > > > > efforts to stabilize branch-3 to get it to a releasable
> state
> > > > and I
> > > > > > > agree
> > > > > > > > > that we should get it to a green state before opening it
> for
> > > PRs
> > > > > not
> > > > > > > > > related to test failures. I can help with the effort as
> well.
> > > > > > > > >
> > > > > > > > > If we want to get the branch back to green state soon, have
> > we
> > > > > > > considered
> > > > > > > > > disabling the tests which are clearly flaky? (e.g pass on
> > some
> > > > > builds
> > > > > > > and
> > > > > > > > > fail on the other build with no new code changes). If we
> > don't
> > > do
> > > > > > that,
> > > > > > > > we
> > > > > > > > > will keep playing whack a mole with those tests. I propose
> > for
> > > > such
> > > > > > > tests
> > > > > > > > > we should disable them and create tickets to unflake them
> > > > > separately.
> > > > > > > > This
> > > > > > > > > will help us get back to a green state faster.
> > > > > > > > >
> > > > > > > > > Hi Aman,
> > > > > > > > > For TestMiniSparkOnYarnCliDriver failures, you probably
> > should
> > > > also
> > > > > > > look
> > > > > > > > > into the spark driver/application logs and see if there are
> > > > > > > > infrastructure
> > > > > > > > > errors (e.g OOMs). Are these tests failing when you run
> > > locally?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Vihang
> > > > > > > > >
> > > > > > > > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj
> > > > > > <rajaman@microsoft.com.invalid
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1,
> > > > > > > > > > Thanks Stamatis and Lazlo for helping in the test case
> > fixes
> > > > till
> > > > > > > now.
> > > > > > > > > >
> > > > > > > > > > Team,
> > > > > > > > > > I need help in fixing the following tests in Hive. I have
> > > tried
> > > > > > > > different
> > > > > > > > > > approaches but no luck till now.
> > > > > > > > > > I am facing some issues in fixing the following tests :
> > > > > > > > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > > > > > > > >
> > > > > > > > > > Issue :
> > > > > > > > > > PREHOOK: Input: default@src
> > > > > > > > > > PREHOOK: Output: default@src
> > > > > > > > > > Failed to monitor Job[-1] with exception
> > > > > > > > > > 'java.lang.IllegalStateException(Connection to remote
> Spark
> > > > > driver
> > > > > > > was
> > > > > > > > > > lost)' Last known state = SENT
> > > > > > > > > > Failed to execute spark task, with exception
> > > > > > > > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > > > > > > > FAILED: Execution Error, return code 1 from
> > > > > > > > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC
> channel
> > > is
> > > > > > > closed.
> > > > > > > > > >
> > > > > > > > > > History :
> > > > > > > > > > Initially the tests had failed with errors which I fixed
> in
> > > the
> > > > > > > > following
> > > > > > > > > > task :
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=33sxNakmV8J0QuhB4tPbe2qBtO%2B6pe9ViImnh3cWN%2Fw%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-26940>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=33sxNakmV8J0QuhB4tPbe2qBtO%2B6pe9ViImnh3cWN%2Fw%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-26940>>
> > > > > > > > > >
> > > > > > > > > > Does anyone know what the issue is here ? There are 6-7
> > > > failures
> > > > > > > > because
> > > > > > > > > > of this test case. Link to the failed test cases for the
> > > > > > stacktrace :
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6YsAKbLGSDWISDgOIS8kDmsaRGQsLuMCa1Z9Mg%2FNb68%3D&reserved=0
> <
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3949/2/tests/
> >
> > > > <
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6YsAKbLGSDWISDgOIS8kDmsaRGQsLuMCa1Z9Mg%2FNb68%3D&reserved=0
> <
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3949/2/tests/
> >
> > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Aman.
> > > > > > > > > >
> > > > > > > > > > ________________________________
> > > > > > > > > > From: László Bodor <bo...@gmail.com>
> > > > > > > > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > > > > Subject: [EXTERNAL] Re: Branch-3 backports and build
> > > stability
> > > > > > > > > >
> > > > > > > > > > +1
> > > > > > > > > > also, if I merged something that I thought was for test
> > > > stability
> > > > > > > (but
> > > > > > > > > > instead it was a feature), excuse me :)
> > > > > > > > > > for reference, the whole green test initiative is tracked
> > > under
> > > > > > this
> > > > > > > > > > umbrella:
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5yJtrMl80d%2BKYqlLuoG4vLalBHflWqsJN8zpua4tplY%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-26836>
> > > > <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5yJtrMl80d%2BKYqlLuoG4vLalBHflWqsJN8zpua4tplY%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-26836>>
> > > > > > > > > >
> > > > > > > > > > Stamatis Zampetakis <za...@gmail.com> ezt írta
> (időpont:
> > > > 2023.
> > > > > > > febr.
> > > > > > > > > 7.,
> > > > > > > > > > K, 12:09):
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > The build in branch-3 is not yet green; there are ~25
> > test
> > > > > > > failures.
> > > > > > > > It
> > > > > > > > > > is
> > > > > > > > > > > a common practice that we shouldn't push changes on top
> > of
> > > a
> > > > > > broken
> > > > > > > > > build
> > > > > > > > > > > unless they are addressing test failures.
> > > > > > > > > > >
> > > > > > > > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo
> > > > Bodor)
> > > > > > are
> > > > > > > > > > working
> > > > > > > > > > > hard to stabilize the build for quite some time now. If
> > you
> > > > > want
> > > > > > to
> > > > > > > > > help
> > > > > > > > > > > out then start by reviewing, merging, and fixing things
> > > > around
> > > > > > test
> > > > > > > > > > > failures.
> > > > > > > > > > >
> > > > > > > > > > > It's not yet the time to bring new features, upgrades,
> > > bugs,
> > > > > > etc.,
> > > > > > > in
> > > > > > > > > > > branch-3. I would encourage  committers to not approve
> > such
> > > > > > changes
> > > > > > > > > till
> > > > > > > > > > we
> > > > > > > > > > > get back to a stable branch.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Stamatis
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [EXTERNAL] Re: Branch-3 backports and build stability

[Aman Raj <ra...@microsoft.com.INVALID>]

Hi Vihang,

We only have 4 tickets remaining to be backported from branch-3.1 to branch-3. It will be completed next week.

But there are a lot of new tickets that will go into release 3.2.0 on top of this. I was thinking of not cutting a release candidate now since it would mean that we only backport changes into that release candidate branch. This would again mean that if people commit only to branch-3 or the release branch, there will again be a lot of difference in these two branches when someone picks up the next release.

Instead I am thinking that we should backport new changes to branch-3 and then only cut the release candidate. Please let me know your thoughts. If we agree that changes need to go into the new release candidate branch only, I am okay with that (I do not prefer it btw)

Thanks,
Aman.

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: vihang karajgaonkar <vi...@apache.org>
Sent: Monday, May 8, 2023 4:57:24 AM
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability

Hi Aman,

I know you are backporting the branch-3.1 commits to branch-3. How close
are you with finishing with them. Is there anything that we can help with
to get it over the finish line?

I am interested to know how close are we to cutting the branch for 3.2.0?
Do you think we can have a release candidate this week?

Thanks,
Vihang

On Thu, Mar 30, 2023 at 2:18 AM Stamatis Zampetakis <za...@gmail.com>
wrote:

> Huge thanks to everyone involved it is great to see the branch-3 in stable
> state. As other people mentioned let's keep it that way!
>
> As far as it concerns back ports please be particularly cautious with
> anything that touches the metastore schema and Thrift APIs.
>
> Best,
> Stamatis
>
> On Wed, Mar 29, 2023, 4:36 AM vihang karajgaonkar <vi...@apache.org>
> wrote:
>
> > Thanks a lot Aman for all your efforts on this. Really appreciate the
> > initiative and all your hard work on this.
> >
> > I would like to request that all the committers should follow the merge
> > process of master branch to merge PRs in branch-3. If there are any test
> > failures which seem unrelated, please do not ignore them. One can run the
> > flaky
> > test runner <https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fjob%2Fhive-flaky-check%2F&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NFcfXybje7zooPXkkN9UnSQIexxjKJ9wQR%2B0nrXk4e0%3D&reserved=0<http://ci.hive.apache.org/job/hive-flaky-check/>> to make
> sure
> > that test is indeed flaky. If the test is found to be flaky a
> > ticket should be created to disable it. A separate ticket should be
> created
> > to deflake it and you can mention the original author or previous commit
> > author who changed the test on that ticket to get help since they likely
> > have the most context around that test. Once the flaky test is disabled
> and
> > we have a green CI job run, we should merge the PR. If others have any
> > suggestions to improve this process please chime in.
> >
> > Thanks,
> > Vihang
> >
> > On Tue, Mar 28, 2023 at 10:55 PM Aman Raj <rajaman@microsoft.com.invalid
> >
> > wrote:
> >
> > > Hi community,
> > >
> > > This is to notify that we have a green branch-3 now. The entire effort
> of
> > > fixing branch-3 test cases took around 4 months and as a team we
> managed
> > to
> > > fix 2900+ test failures on branch-3. The entire effort can be tracked
> > here
> > > HIVE-26836<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DXjdb%2FLcUztu6Kr2phfhuVgdP652Yvlddq1QmtTndAw%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26836>>. We are
> > > ready to push new features and improvements on branch-3 now.
> > >
> > > I really want to thank Vihang Karajgaonkar, Chris Nauroth, Lazlo Bodor,
> > > Stamatis Zampetakis and Sankar Hariappan without whom this would not at
> > all
> > > have been possible. As a team we stuck together and participated in
> > reviews
> > > and actively suggested improvements which really helped in fixing some
> > > major test failures.
> > >
> > > I would sincerely request that going further it should be made a point
> to
> > > merge things into branch-3 only if we have a green Jenkins pipeline.
> > >
> > > The next step would be to backport changes from branch-3.1 (From where
> > > Hive-3.1.3 release was made) to branch-3. This would ensure that we do
> > not
> > > miss any specific ticket which went into Hive-3.1.3. I will take care
> of
> > > this. We can parallelly start pushing additional changes on branch-3.
> > There
> > > are approximately 25 tickets that need to be backported in this effort
> > (Of
> > > backporting changes from branch-3.1). I have made a note here<
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fspreadsheets%2Fd%2F1K0U-vxLRZEs13oBzYBlVyK8dMMNthgXL5VEgzLRbeKs%2Fedit%3Fusp%3Dsharing&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UB3VwQkRFsNlPsZYuGuzt1uQwp20L%2BmgLAD8CvopzUw%3D&reserved=0<https://docs.google.com/spreadsheets/d/1K0U-vxLRZEs13oBzYBlVyK8dMMNthgXL5VEgzLRbeKs/edit?usp=sharing>
> > > >
> > >
> > > Again, thanks a lot to everyone who supported and participated in this
> > > effort. Lets make this 3.2.0 Hive release happen!!
> > >
> > > Thanks,
> > > Aman.
> > >
> > > ________________________________
> > > From: Aman Raj <ra...@microsoft.com.INVALID>
> > > Sent: Monday, March 20, 2023 9:21 AM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > >
> > > Hi Vihang/community,
> > >
> > > Found the ticket which broke mm_all.q. This issue comes because of
> > > HIVE-20182. Works in my local and on the Jenkins pipeline as well.
> Link :
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F4127&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aZRb8agmteUTddKJaBEtUC3EMd6ZibPfIu2nB9yOptk%3D&reserved=0<https://github.com/apache/hive/pull/4127>
> > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F4127&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aZRb8agmteUTddKJaBEtUC3EMd6ZibPfIu2nB9yOptk%3D&reserved=0<https://github.com/apache/hive/pull/4127>> Reverting this commit for
> > now.
> > >
> > > Thanks,
> > > Aman.
> > > ________________________________
> > > From: Aman Raj <ra...@microsoft.com.INVALID>
> > > Sent: Monday, March 20, 2023 8:28 AM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > >
> > > Sure Vihang, will look at the other ones. You can pick this up.
> > >
> > > Thanks,
> > > Aman.
> > >
> > > Get Outlook for Android<
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5gG0C%2BmQpxQOV%2BCQChCGbnlMv5e9BCS7YZ5xO9y9zaw%3D&reserved=0<https://aka.ms/AAb9ysg>
> > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5gG0C%2BmQpxQOV%2BCQChCGbnlMv5e9BCS7YZ5xO9y9zaw%3D&reserved=0<https://aka.ms/AAb9ysg>>>
> > > ________________________________
> > > From: vihang karajgaonkar <vi...@apache.org>
> > > Sent: Monday, March 20, 2023 7:58:48 AM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > >
> > > I think we should revert offending commits first to unblock the branch.
> > We
> > > can create followup tickets to determine if these fixes are blockers
> for
> > > 3.2 release and if yes, we should merge them the right way with a green
> > > test run. Fixing forward always comes with the risk that it introduces
> > new
> > > test failures.
> > >
> > > Thanks for all your efforts on this Aman.
> > >
> > > I can take a look at
> testBootstrapReplLoadRetryAfterFailureForPartitions
> > if
> > > you haven’t already started on it.
> > >
> > > Thanks,
> > > Vihang
> > >
> > > On Sun, Mar 19, 2023 at 10:09 PM Aman Raj
> <rajaman@microsoft.com.invalid
> > >
> > > wrote:
> > >
> > > > Hi Vihang/community,
> > > >
> > > > Thanks a lot Vihang for working on the major test failure. This
> blocked
> > > > more than 35 test cases. Now we are down to the final 4 failures. I
> > have
> > > > analyzed some of them and here they are  (Link :
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-4067%2F12%2Ftests&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Vjjsob4PG0%2FU5uDRUEf23WCCfuM622SWcBLEhuX1qKo%3D&reserved=0<http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests>
> > > )<
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-4067%2F12%2Ftests&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Vjjsob4PG0%2FU5uDRUEf23WCCfuM622SWcBLEhuX1qKo%3D&reserved=0<http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests>
> > > >
> > > > :
> > > >
> > > >   1.
> > > > multi_in_clause - This was committed in HIVE-21685 without validating
> > the
> > > > scenario.
> > > > This fails because Hive is not able to parse
> > > > explain cbo
> > > > select * from very_simple_table_for_in_test where name IN('g','r')
> AND
> > > > name IN('a','b')
> > > > If we want this to work, I am able to do it in my local. We have 2
> > > options
> > > > :
> > > > a. Either revert HIVE-21685 since this scenario was not validated
> back
> > > > then before adding this test.
> > > > b. This fix was present in
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UyGzmPpRJjFM1AciK0W8TMZ3Y%2BNrAtfQcyCEyrFu7uM%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-20718>
> > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UyGzmPpRJjFM1AciK0W8TMZ3Y%2BNrAtfQcyCEyrFu7uM%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-20718>> but to cherry pick
> > this
> > > > we need to cherry pick
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pwa7HtgXaeSKVrDgb6Bh8Fg2tBYdDKmh1O6NCA0qPrw%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-17040>
> > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pwa7HtgXaeSKVrDgb6Bh8Fg2tBYdDKmh1O6NCA0qPrw%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-17040>>
> > > > since HIVE-20718<
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UyGzmPpRJjFM1AciK0W8TMZ3Y%2BNrAtfQcyCEyrFu7uM%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-20718>
> > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UyGzmPpRJjFM1AciK0W8TMZ3Y%2BNrAtfQcyCEyrFu7uM%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-20718>>> has a
> > > > lot of merge conflicts with  HIVE-17040<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pwa7HtgXaeSKVrDgb6Bh8Fg2tBYdDKmh1O6NCA0qPrw%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-17040>
> > > ><https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pwa7HtgXaeSKVrDgb6Bh8Fg2tBYdDKmh1O6NCA0qPrw%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-17040>>. But after cherry
> > > > picking these we have other failures to fix.
> > > >   2.
> > > > current_date_timestamp.q - This breaking change was committed in
> > > > HIVE-21388 without validation.
> > > > The failure is because again Hive is not able to parse
> > > > explain cbo select current_timestamp() from alltypesorc
> > > > The solution or revert option is same as point 1.
> > > >   3.
> > > > testBootstrapReplLoadRetryAfterFailureForPartitions() - This I have
> not
> > > > investigated till now.
> > > >   4.
> > > > mm_all.q - This I have not investigated till now.
> > > >
> > > > Thanks,
> > > > Aman.
> > > > ________________________________
> > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > Sent: Friday, March 17, 2023 8:42 PM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > Just wanted to close the loop on the TestMiniSparkOnYarnCliDriver
> test
> > > > failures. We will be able to re-enable most of them back on branch-3.
> > The
> > > > ones which were disabled are being tracked separately in a different
> > > ticket
> > > > <
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=kMZtW1GSTclD0RjnALCDs6KoT%2F%2B13PFbNvD8psiU%2BCQ%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-27146>
> > > ><https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=kMZtW1GSTclD0RjnALCDs6KoT%2F%2B13PFbNvD8psiU%2BCQ%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-27146>>
> > > > but they don't look like
> > > > a blocker.
> > > >
> > > > Hi Aman,
> > > >
> > > > Do you know how close are we to reopening branch-3?
> > > >
> > > > Thanks,
> > > > Vihang
> > > >
> > > > On Sat, Mar 4, 2023 at 7:23 PM Aman Raj
> <rajaman@microsoft.com.invalid
> > >
> > > > wrote:
> > > >
> > > > > Or you can cd into itests and run the command you are using. Just
> > > another
> > > > > way I run.
> > > > >
> > > > > Thanks,
> > > > > Aman.
> > > > > Get Outlook for Android<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5gG0C%2BmQpxQOV%2BCQChCGbnlMv5e9BCS7YZ5xO9y9zaw%3D&reserved=0<https://aka.ms/AAb9ysg>
> > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5gG0C%2BmQpxQOV%2BCQChCGbnlMv5e9BCS7YZ5xO9y9zaw%3D&reserved=0<https://aka.ms/AAb9ysg>>
> > > > >
> > > > > ________________________________
> > > > > From: Aman Raj <ra...@microsoft.com>
> > > > > Sent: Saturday, March 4, 2023 7:20:36 PM
> > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > >
> > > > > Hi Vihang,
> > > > >
> > > > > Thanks a lot for working on this. Can you try using
> -Pqsplits,itests.
> > > > > Also, I usually give a -o option after doing a clean install.
> > > > >
> > > > > Thanks,
> > > > > Aman.
> > > > >
> > > > > Get Outlook for Android<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5gG0C%2BmQpxQOV%2BCQChCGbnlMv5e9BCS7YZ5xO9y9zaw%3D&reserved=0<https://aka.ms/AAb9ysg>
> > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5gG0C%2BmQpxQOV%2BCQChCGbnlMv5e9BCS7YZ5xO9y9zaw%3D&reserved=0<https://aka.ms/AAb9ysg>>
> > > > >
> > > > >
> > > > > ________________________________
> > > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > > Sent: Saturday, 4 March, 2023, 11:35
> > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > >
> > > > > [You don't often get email from vihangk1@apache.org. Learn why
> this
> > is
> > > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > > >
> > > > > Just to update on the HoS test failures for
> > > > TestMiniSparkOnYarnCliDriver, I
> > > > > think I was finally able to resolve them (at least on local). I had
> > to
> > > > > revert HIVE-21044 because it was causing OOM for those tests. Also,
> > in
> > > > > order for these tests to work we will have to downgrade netty from
> > > > > 4.1.69.Final to 4.1.51.Final. I understand that we had upgraded
> netty
> > > > from
> > > > > 4.1.17.Final to 4.1.69.Final for CVEs but the highest netty version
> > > that
> > > > we
> > > > > can support without breaking HoS is 4.1.51.Final. Note that
> > > 4.1.51.Final
> > > > > includes many of the CVEs which affected 4.1.17.Final so we are
> still
> > > in
> > > > a
> > > > > better place than branch-3.1. Unfortunately, there is no good way
> to
> > > make
> > > > > HoS work with a higher netty version so I think we should downgrade
> > the
> > > > > netty version to 4.1.51.Final for now and look at more options to
> > > upgrade
> > > > > it 4.1.69.Final in a separate ticket.
> > > > >
> > > > > I still need to understand why the tests which are working for me
> > > locally
> > > > > don't work on the PR job. I tried running the split test classes
> > using
> > > > the
> > > > > following command. Is that the right way to simulate builds from
> the
> > PR
> > > > > job? Let me know if anyone has more ideas.
> > > > >
> > > > > mvn test
> > > > >
> -Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
> > > > > -Pqsplits
> > > > >
> > > > > Thanks,
> > > > > Vihang
> > > > >
> > > > >
> > > > > On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <
> > zabetak@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > Thanks Aman for bringing this up and also for cleaning up after
> > > others
> > > > (I
> > > > > > saw that you raised tickets and PRs for addressing the failures).
> > > > > >
> > > > > > Many thanks to Vihang as well for helping out. Regarding flaky
> > tests,
> > > > yes
> > > > > > we should disable them as soon as we see them.
> > > > > > There have been some other discussions on how to approach flaky
> > tests
> > > > the
> > > > > > more recent I could find is here [1].
> > > > > >
> > > > > > Best,
> > > > > > Stamatis
> > > > > >
> > > > > > [1]
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ym1WBuGKRt%2FuTHitzCwR%2FzckJGLPO1XinHZm6kt%2BgKk%3D&reserved=0<https://lists.apache.org/thread/lv3bhlfoq8fwd9dwyjf7g4nx32wtrygv>
> > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988697872822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ym1WBuGKRt%2FuTHitzCwR%2FzckJGLPO1XinHZm6kt%2BgKk%3D&reserved=0<https://lists.apache.org/thread/lv3bhlfoq8fwd9dwyjf7g4nx32wtrygv>>
> > > > > >
> > > > > > On Fri, Feb 17, 2023 at 4:37 AM Aman Raj
> > > <rajaman@microsoft.com.invalid
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi team,
> > > > > > >
> > > > > > > Thanks Vihang for looking into this. I have commented on the
> JIRA
> > > you
> > > > > > > created.
> > > > > > >
> > > > > > > Just to bring everyone's notice, I have seen that there has
> been
> > a
> > > > > couple
> > > > > > > of pushes to branch-3, which has lead to 5 more new test
> > failures.
> > > > The
> > > > > > test
> > > > > > > failures are in orc_merge1, orc_merge2, orc_merge3, orc_merge4
> > and
> > > > > > > orc_merge10. These tests did not use to fail before. I would
> > > > sincerely
> > > > > > urge
> > > > > > > the community to raise a PR against branch-3, so that the
> Jenkins
> > > > > > pipeline
> > > > > > > can run and then only merge things to branch-3. We had 2900+
> > > failures
> > > > > > when
> > > > > > > we started 2 months back and now having brought it down to less
> > > than
> > > > > 15,
> > > > > > > new failures again has pushed us back in this effort.
> > > > > > >
> > > > > > > I would like to thank everyone who has participated in this
> > effort
> > > > and
> > > > > > > made it possible till this stage. Also, if the contributors can
> > > take
> > > > > > > ownership of these new test case failures and fix them, it will
> > be
> > > of
> > > > > > great
> > > > > > > help.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Aman.
> > > > > > > ________________________________
> > > > > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > > > > Sent: Friday, February 17, 2023 6:10 AM
> > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> > stability
> > > > > > >
> > > > > > > [You don't often get email from vihangk1@apache.org. Learn why
> > > this
> > > > is
> > > > > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > > > > >
> > > > > > > Hi Aman,
> > > > > > >
> > > > > > > I created
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6vJLQ2iZLyEfQv%2By8C8se%2F%2BABdUmnOlnmCSOfTEByWg%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-27087>
> > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6vJLQ2iZLyEfQv%2By8C8se%2F%2BABdUmnOlnmCSOfTEByWg%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-27087>>
> > > > > > > to look into
> > > > > > > TestMiniSparkOnYarnCliDriver failures. I have a working theory
> of
> > > > what
> > > > > > > might be going on there. I am still investigating what is the
> > right
> > > > way
> > > > > > to
> > > > > > > fix it though.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Vihang
> > > > > > >
> > > > > > > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj
> > > > > <rajaman@microsoft.com.invalid
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Vihang,
> > > > > > > >
> > > > > > > > Yes the tests are failing locally as well with the same
> issue.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Aman.
> > > > > > > >
> > > > > > > > Get Outlook for Android<
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FU380OPfIwF6BHqGG85T8XQlWjG6QmDuQfwLytzoE6w%3D&reserved=0<https://aka.ms/AAb9ysg>
> > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FU380OPfIwF6BHqGG85T8XQlWjG6QmDuQfwLytzoE6w%3D&reserved=0<https://aka.ms/AAb9ysg>>
> > > > > > > >
> > > > > > > > ________________________________
> > > > > > > > From: Vihang Karajgaonkar
> > > > <vihang.karajgaonkar@databricks.com.INVALID
> > > > > >
> > > > > > > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> > > stability
> > > > > > > >
> > > > > > > > [You don't often get email from
> > > > > > > vihang.karajgaonkar@databricks.com.invalid.
> > > > > > > > Learn why this is important at
> > > > > > > > https://aka.ms/LearnAboutSenderIdentification ]
> > > > > > > >
> > > > > > > > Thanks a lot Stamatis for starting this thread. I really
> > > appreciate
> > > > > all
> > > > > > > the
> > > > > > > > efforts to stabilize branch-3 to get it to a releasable state
> > > and I
> > > > > > agree
> > > > > > > > that we should get it to a green state before opening it for
> > PRs
> > > > not
> > > > > > > > related to test failures. I can help with the effort as well.
> > > > > > > >
> > > > > > > > If we want to get the branch back to green state soon, have
> we
> > > > > > considered
> > > > > > > > disabling the tests which are clearly flaky? (e.g pass on
> some
> > > > builds
> > > > > > and
> > > > > > > > fail on the other build with no new code changes). If we
> don't
> > do
> > > > > that,
> > > > > > > we
> > > > > > > > will keep playing whack a mole with those tests. I propose
> for
> > > such
> > > > > > tests
> > > > > > > > we should disable them and create tickets to unflake them
> > > > separately.
> > > > > > > This
> > > > > > > > will help us get back to a green state faster.
> > > > > > > >
> > > > > > > > Hi Aman,
> > > > > > > > For TestMiniSparkOnYarnCliDriver failures, you probably
> should
> > > also
> > > > > > look
> > > > > > > > into the spark driver/application logs and see if there are
> > > > > > > infrastructure
> > > > > > > > errors (e.g OOMs). Are these tests failing when you run
> > locally?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Vihang
> > > > > > > >
> > > > > > > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj
> > > > > <rajaman@microsoft.com.invalid
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1,
> > > > > > > > > Thanks Stamatis and Lazlo for helping in the test case
> fixes
> > > till
> > > > > > now.
> > > > > > > > >
> > > > > > > > > Team,
> > > > > > > > > I need help in fixing the following tests in Hive. I have
> > tried
> > > > > > > different
> > > > > > > > > approaches but no luck till now.
> > > > > > > > > I am facing some issues in fixing the following tests :
> > > > > > > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > > > > > > >
> > > > > > > > > Issue :
> > > > > > > > > PREHOOK: Input: default@src
> > > > > > > > > PREHOOK: Output: default@src
> > > > > > > > > Failed to monitor Job[-1] with exception
> > > > > > > > > 'java.lang.IllegalStateException(Connection to remote Spark
> > > > driver
> > > > > > was
> > > > > > > > > lost)' Last known state = SENT
> > > > > > > > > Failed to execute spark task, with exception
> > > > > > > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > > > > > > FAILED: Execution Error, return code 1 from
> > > > > > > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel
> > is
> > > > > > closed.
> > > > > > > > >
> > > > > > > > > History :
> > > > > > > > > Initially the tests had failed with errors which I fixed in
> > the
> > > > > > > following
> > > > > > > > > task :
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=33sxNakmV8J0QuhB4tPbe2qBtO%2B6pe9ViImnh3cWN%2Fw%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26940>
> > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=33sxNakmV8J0QuhB4tPbe2qBtO%2B6pe9ViImnh3cWN%2Fw%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26940>>
> > > > > > > > >
> > > > > > > > > Does anyone know what the issue is here ? There are 6-7
> > > failures
> > > > > > > because
> > > > > > > > > of this test case. Link to the failed test cases for the
> > > > > stacktrace :
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6YsAKbLGSDWISDgOIS8kDmsaRGQsLuMCa1Z9Mg%2FNb68%3D&reserved=0<http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3949/2/tests/>
> > > <
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6YsAKbLGSDWISDgOIS8kDmsaRGQsLuMCa1Z9Mg%2FNb68%3D&reserved=0<http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3949/2/tests/>
> > > >
> > > > > > > > > Thanks,
> > > > > > > > > Aman.
> > > > > > > > >
> > > > > > > > > ________________________________
> > > > > > > > > From: László Bodor <bo...@gmail.com>
> > > > > > > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > > > Subject: [EXTERNAL] Re: Branch-3 backports and build
> > stability
> > > > > > > > >
> > > > > > > > > +1
> > > > > > > > > also, if I merged something that I thought was for test
> > > stability
> > > > > > (but
> > > > > > > > > instead it was a feature), excuse me :)
> > > > > > > > > for reference, the whole green test initiative is tracked
> > under
> > > > > this
> > > > > > > > > umbrella:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5yJtrMl80d%2BKYqlLuoG4vLalBHflWqsJN8zpua4tplY%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26836>
> > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C9f071a96a03e491757e908db4f52a621%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638190988698028895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5yJtrMl80d%2BKYqlLuoG4vLalBHflWqsJN8zpua4tplY%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26836>>
> > > > > > > > >
> > > > > > > > > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont:
> > > 2023.
> > > > > > febr.
> > > > > > > > 7.,
> > > > > > > > > K, 12:09):
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > The build in branch-3 is not yet green; there are ~25
> test
> > > > > > failures.
> > > > > > > It
> > > > > > > > > is
> > > > > > > > > > a common practice that we shouldn't push changes on top
> of
> > a
> > > > > broken
> > > > > > > > build
> > > > > > > > > > unless they are addressing test failures.
> > > > > > > > > >
> > > > > > > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo
> > > Bodor)
> > > > > are
> > > > > > > > > working
> > > > > > > > > > hard to stabilize the build for quite some time now. If
> you
> > > > want
> > > > > to
> > > > > > > > help
> > > > > > > > > > out then start by reviewing, merging, and fixing things
> > > around
> > > > > test
> > > > > > > > > > failures.
> > > > > > > > > >
> > > > > > > > > > It's not yet the time to bring new features, upgrades,
> > bugs,
> > > > > etc.,
> > > > > > in
> > > > > > > > > > branch-3. I would encourage  committers to not approve
> such
> > > > > changes
> > > > > > > > till
> > > > > > > > > we
> > > > > > > > > > get back to a stable branch.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Stamatis
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [EXTERNAL] Re: Branch-3 backports and build stability

[vihang karajgaonkar <vi...@apache.org>]

Hi Aman,

I know you are backporting the branch-3.1 commits to branch-3. How close
are you with finishing with them. Is there anything that we can help with
to get it over the finish line?

I am interested to know how close are we to cutting the branch for 3.2.0?
Do you think we can have a release candidate this week?

Thanks,
Vihang

On Thu, Mar 30, 2023 at 2:18 AM Stamatis Zampetakis <za...@gmail.com>
wrote:

> Huge thanks to everyone involved it is great to see the branch-3 in stable
> state. As other people mentioned let's keep it that way!
>
> As far as it concerns back ports please be particularly cautious with
> anything that touches the metastore schema and Thrift APIs.
>
> Best,
> Stamatis
>
> On Wed, Mar 29, 2023, 4:36 AM vihang karajgaonkar <vi...@apache.org>
> wrote:
>
> > Thanks a lot Aman for all your efforts on this. Really appreciate the
> > initiative and all your hard work on this.
> >
> > I would like to request that all the committers should follow the merge
> > process of master branch to merge PRs in branch-3. If there are any test
> > failures which seem unrelated, please do not ignore them. One can run the
> > flaky
> > test runner <http://ci.hive.apache.org/job/hive-flaky-check/> to make
> sure
> > that test is indeed flaky. If the test is found to be flaky a
> > ticket should be created to disable it. A separate ticket should be
> created
> > to deflake it and you can mention the original author or previous commit
> > author who changed the test on that ticket to get help since they likely
> > have the most context around that test. Once the flaky test is disabled
> and
> > we have a green CI job run, we should merge the PR. If others have any
> > suggestions to improve this process please chime in.
> >
> > Thanks,
> > Vihang
> >
> > On Tue, Mar 28, 2023 at 10:55 PM Aman Raj <rajaman@microsoft.com.invalid
> >
> > wrote:
> >
> > > Hi community,
> > >
> > > This is to notify that we have a green branch-3 now. The entire effort
> of
> > > fixing branch-3 test cases took around 4 months and as a team we
> managed
> > to
> > > fix 2900+ test failures on branch-3. The entire effort can be tracked
> > here
> > > HIVE-26836<https://issues.apache.org/jira/browse/HIVE-26836>. We are
> > > ready to push new features and improvements on branch-3 now.
> > >
> > > I really want to thank Vihang Karajgaonkar, Chris Nauroth, Lazlo Bodor,
> > > Stamatis Zampetakis and Sankar Hariappan without whom this would not at
> > all
> > > have been possible. As a team we stuck together and participated in
> > reviews
> > > and actively suggested improvements which really helped in fixing some
> > > major test failures.
> > >
> > > I would sincerely request that going further it should be made a point
> to
> > > merge things into branch-3 only if we have a green Jenkins pipeline.
> > >
> > > The next step would be to backport changes from branch-3.1 (From where
> > > Hive-3.1.3 release was made) to branch-3. This would ensure that we do
> > not
> > > miss any specific ticket which went into Hive-3.1.3. I will take care
> of
> > > this. We can parallelly start pushing additional changes on branch-3.
> > There
> > > are approximately 25 tickets that need to be backported in this effort
> > (Of
> > > backporting changes from branch-3.1). I have made a note here<
> > >
> >
> https://docs.google.com/spreadsheets/d/1K0U-vxLRZEs13oBzYBlVyK8dMMNthgXL5VEgzLRbeKs/edit?usp=sharing
> > > >
> > >
> > > Again, thanks a lot to everyone who supported and participated in this
> > > effort. Lets make this 3.2.0 Hive release happen!!
> > >
> > > Thanks,
> > > Aman.
> > >
> > > ________________________________
> > > From: Aman Raj <ra...@microsoft.com.INVALID>
> > > Sent: Monday, March 20, 2023 9:21 AM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > >
> > > Hi Vihang/community,
> > >
> > > Found the ticket which broke mm_all.q. This issue comes because of
> > > HIVE-20182. Works in my local and on the Jenkins pipeline as well.
> Link :
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F4127&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XSPlEtfWDNV%2Fccv9Q33xUtMLuhvxHx3CD4kC%2F5mWj2Y%3D&reserved=0
> > > <https://github.com/apache/hive/pull/4127> Reverting this commit for
> > now.
> > >
> > > Thanks,
> > > Aman.
> > > ________________________________
> > > From: Aman Raj <ra...@microsoft.com.INVALID>
> > > Sent: Monday, March 20, 2023 8:28 AM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > >
> > > Sure Vihang, will look at the other ones. You can pick this up.
> > >
> > > Thanks,
> > > Aman.
> > >
> > > Get Outlook for Android<
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0
> > > <https://aka.ms/AAb9ysg>>
> > > ________________________________
> > > From: vihang karajgaonkar <vi...@apache.org>
> > > Sent: Monday, March 20, 2023 7:58:48 AM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > >
> > > I think we should revert offending commits first to unblock the branch.
> > We
> > > can create followup tickets to determine if these fixes are blockers
> for
> > > 3.2 release and if yes, we should merge them the right way with a green
> > > test run. Fixing forward always comes with the risk that it introduces
> > new
> > > test failures.
> > >
> > > Thanks for all your efforts on this Aman.
> > >
> > > I can take a look at
> testBootstrapReplLoadRetryAfterFailureForPartitions
> > if
> > > you haven’t already started on it.
> > >
> > > Thanks,
> > > Vihang
> > >
> > > On Sun, Mar 19, 2023 at 10:09 PM Aman Raj
> <rajaman@microsoft.com.invalid
> > >
> > > wrote:
> > >
> > > > Hi Vihang/community,
> > > >
> > > > Thanks a lot Vihang for working on the major test failure. This
> blocked
> > > > more than 35 test cases. Now we are down to the final 4 failures. I
> > have
> > > > analyzed some of them and here they are  (Link :
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-4067%2F12%2Ftests&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hiXJeNe9LPpWxhacjL2o3RUoalhcn86yog1IHz7JMHw%3D&reserved=0
> > > )<
> > >
> >
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests
> > > >
> > > > :
> > > >
> > > >   1.
> > > > multi_in_clause - This was committed in HIVE-21685 without validating
> > the
> > > > scenario.
> > > > This fails because Hive is not able to parse
> > > > explain cbo
> > > > select * from very_simple_table_for_in_test where name IN('g','r')
> AND
> > > > name IN('a','b')
> > > > If we want this to work, I am able to do it in my local. We have 2
> > > options
> > > > :
> > > > a. Either revert HIVE-21685 since this scenario was not validated
> back
> > > > then before adding this test.
> > > > b. This fix was present in
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FzY3laBCDchxxS2aFQ%2FTS3IYjOCxl%2FTTBFdQu9xBwUI%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/HIVE-20718> but to cherry pick
> > this
> > > > we need to cherry pick
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bjEWed1c5qwMjE1ZCJ1TRyHl%2FK1hADk9F05x8HO0f9A%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/HIVE-17040>
> > > > since HIVE-20718<
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FzY3laBCDchxxS2aFQ%2FTS3IYjOCxl%2FTTBFdQu9xBwUI%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/HIVE-20718>> has a
> > > > lot of merge conflicts with  HIVE-17040<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bjEWed1c5qwMjE1ZCJ1TRyHl%2FK1hADk9F05x8HO0f9A%3D&reserved=0
> > > ><https://issues.apache.org/jira/browse/HIVE-17040>. But after cherry
> > > > picking these we have other failures to fix.
> > > >   2.
> > > > current_date_timestamp.q - This breaking change was committed in
> > > > HIVE-21388 without validation.
> > > > The failure is because again Hive is not able to parse
> > > > explain cbo select current_timestamp() from alltypesorc
> > > > The solution or revert option is same as point 1.
> > > >   3.
> > > > testBootstrapReplLoadRetryAfterFailureForPartitions() - This I have
> not
> > > > investigated till now.
> > > >   4.
> > > > mm_all.q - This I have not investigated till now.
> > > >
> > > > Thanks,
> > > > Aman.
> > > > ________________________________
> > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > Sent: Friday, March 17, 2023 8:42 PM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > Just wanted to close the loop on the TestMiniSparkOnYarnCliDriver
> test
> > > > failures. We will be able to re-enable most of them back on branch-3.
> > The
> > > > ones which were disabled are being tracked separately in a different
> > > ticket
> > > > <
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CkiQWoSy7LjWvSXr0dYY%2FusSXLIKMw27KIqItvgAfCc%3D&reserved=0
> > > ><https://issues.apache.org/jira/browse/HIVE-27146>
> > > > but they don't look like
> > > > a blocker.
> > > >
> > > > Hi Aman,
> > > >
> > > > Do you know how close are we to reopening branch-3?
> > > >
> > > > Thanks,
> > > > Vihang
> > > >
> > > > On Sat, Mar 4, 2023 at 7:23 PM Aman Raj
> <rajaman@microsoft.com.invalid
> > >
> > > > wrote:
> > > >
> > > > > Or you can cd into itests and run the command you are using. Just
> > > another
> > > > > way I run.
> > > > >
> > > > > Thanks,
> > > > > Aman.
> > > > > Get Outlook for Android<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0
> > > <https://aka.ms/AAb9ysg>
> > > > >
> > > > > ________________________________
> > > > > From: Aman Raj <ra...@microsoft.com>
> > > > > Sent: Saturday, March 4, 2023 7:20:36 PM
> > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > >
> > > > > Hi Vihang,
> > > > >
> > > > > Thanks a lot for working on this. Can you try using
> -Pqsplits,itests.
> > > > > Also, I usually give a -o option after doing a clean install.
> > > > >
> > > > > Thanks,
> > > > > Aman.
> > > > >
> > > > > Get Outlook for Android<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0
> > > <https://aka.ms/AAb9ysg>
> > > > >
> > > > >
> > > > > ________________________________
> > > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > > Sent: Saturday, 4 March, 2023, 11:35
> > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > >
> > > > > [You don't often get email from vihangk1@apache.org. Learn why
> this
> > is
> > > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > > >
> > > > > Just to update on the HoS test failures for
> > > > TestMiniSparkOnYarnCliDriver, I
> > > > > think I was finally able to resolve them (at least on local). I had
> > to
> > > > > revert HIVE-21044 because it was causing OOM for those tests. Also,
> > in
> > > > > order for these tests to work we will have to downgrade netty from
> > > > > 4.1.69.Final to 4.1.51.Final. I understand that we had upgraded
> netty
> > > > from
> > > > > 4.1.17.Final to 4.1.69.Final for CVEs but the highest netty version
> > > that
> > > > we
> > > > > can support without breaking HoS is 4.1.51.Final. Note that
> > > 4.1.51.Final
> > > > > includes many of the CVEs which affected 4.1.17.Final so we are
> still
> > > in
> > > > a
> > > > > better place than branch-3.1. Unfortunately, there is no good way
> to
> > > make
> > > > > HoS work with a higher netty version so I think we should downgrade
> > the
> > > > > netty version to 4.1.51.Final for now and look at more options to
> > > upgrade
> > > > > it 4.1.69.Final in a separate ticket.
> > > > >
> > > > > I still need to understand why the tests which are working for me
> > > locally
> > > > > don't work on the PR job. I tried running the split test classes
> > using
> > > > the
> > > > > following command. Is that the right way to simulate builds from
> the
> > PR
> > > > > job? Let me know if anyone has more ideas.
> > > > >
> > > > > mvn test
> > > > >
> -Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
> > > > > -Pqsplits
> > > > >
> > > > > Thanks,
> > > > > Vihang
> > > > >
> > > > >
> > > > > On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <
> > zabetak@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > Thanks Aman for bringing this up and also for cleaning up after
> > > others
> > > > (I
> > > > > > saw that you raised tickets and PRs for addressing the failures).
> > > > > >
> > > > > > Many thanks to Vihang as well for helping out. Regarding flaky
> > tests,
> > > > yes
> > > > > > we should disable them as soon as we see them.
> > > > > > There have been some other discussions on how to approach flaky
> > tests
> > > > the
> > > > > > more recent I could find is here [1].
> > > > > >
> > > > > > Best,
> > > > > > Stamatis
> > > > > >
> > > > > > [1]
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XZWL3W%2BHTikea8Du4Ohy3%2FHNTwDZBBxwXk5ylqRX0sU%3D&reserved=0
> > > <https://lists.apache.org/thread/lv3bhlfoq8fwd9dwyjf7g4nx32wtrygv>
> > > > > >
> > > > > > On Fri, Feb 17, 2023 at 4:37 AM Aman Raj
> > > <rajaman@microsoft.com.invalid
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi team,
> > > > > > >
> > > > > > > Thanks Vihang for looking into this. I have commented on the
> JIRA
> > > you
> > > > > > > created.
> > > > > > >
> > > > > > > Just to bring everyone's notice, I have seen that there has
> been
> > a
> > > > > couple
> > > > > > > of pushes to branch-3, which has lead to 5 more new test
> > failures.
> > > > The
> > > > > > test
> > > > > > > failures are in orc_merge1, orc_merge2, orc_merge3, orc_merge4
> > and
> > > > > > > orc_merge10. These tests did not use to fail before. I would
> > > > sincerely
> > > > > > urge
> > > > > > > the community to raise a PR against branch-3, so that the
> Jenkins
> > > > > > pipeline
> > > > > > > can run and then only merge things to branch-3. We had 2900+
> > > failures
> > > > > > when
> > > > > > > we started 2 months back and now having brought it down to less
> > > than
> > > > > 15,
> > > > > > > new failures again has pushed us back in this effort.
> > > > > > >
> > > > > > > I would like to thank everyone who has participated in this
> > effort
> > > > and
> > > > > > > made it possible till this stage. Also, if the contributors can
> > > take
> > > > > > > ownership of these new test case failures and fix them, it will
> > be
> > > of
> > > > > > great
> > > > > > > help.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Aman.
> > > > > > > ________________________________
> > > > > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > > > > Sent: Friday, February 17, 2023 6:10 AM
> > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> > stability
> > > > > > >
> > > > > > > [You don't often get email from vihangk1@apache.org. Learn why
> > > this
> > > > is
> > > > > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > > > > >
> > > > > > > Hi Aman,
> > > > > > >
> > > > > > > I created
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5UlK9DVcNIyVkzyMld3%2F1deJaV1TsLMAY2lDV3kjlrQ%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/HIVE-27087>
> > > > > > > to look into
> > > > > > > TestMiniSparkOnYarnCliDriver failures. I have a working theory
> of
> > > > what
> > > > > > > might be going on there. I am still investigating what is the
> > right
> > > > way
> > > > > > to
> > > > > > > fix it though.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Vihang
> > > > > > >
> > > > > > > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj
> > > > > <rajaman@microsoft.com.invalid
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Vihang,
> > > > > > > >
> > > > > > > > Yes the tests are failing locally as well with the same
> issue.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Aman.
> > > > > > > >
> > > > > > > > Get Outlook for Android<
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0
> > > <https://aka.ms/AAb9ysg>
> > > > > > > >
> > > > > > > > ________________________________
> > > > > > > > From: Vihang Karajgaonkar
> > > > <vihang.karajgaonkar@databricks.com.INVALID
> > > > > >
> > > > > > > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> > > stability
> > > > > > > >
> > > > > > > > [You don't often get email from
> > > > > > > vihang.karajgaonkar@databricks.com.invalid.
> > > > > > > > Learn why this is important at
> > > > > > > > https://aka.ms/LearnAboutSenderIdentification ]
> > > > > > > >
> > > > > > > > Thanks a lot Stamatis for starting this thread. I really
> > > appreciate
> > > > > all
> > > > > > > the
> > > > > > > > efforts to stabilize branch-3 to get it to a releasable state
> > > and I
> > > > > > agree
> > > > > > > > that we should get it to a green state before opening it for
> > PRs
> > > > not
> > > > > > > > related to test failures. I can help with the effort as well.
> > > > > > > >
> > > > > > > > If we want to get the branch back to green state soon, have
> we
> > > > > > considered
> > > > > > > > disabling the tests which are clearly flaky? (e.g pass on
> some
> > > > builds
> > > > > > and
> > > > > > > > fail on the other build with no new code changes). If we
> don't
> > do
> > > > > that,
> > > > > > > we
> > > > > > > > will keep playing whack a mole with those tests. I propose
> for
> > > such
> > > > > > tests
> > > > > > > > we should disable them and create tickets to unflake them
> > > > separately.
> > > > > > > This
> > > > > > > > will help us get back to a green state faster.
> > > > > > > >
> > > > > > > > Hi Aman,
> > > > > > > > For TestMiniSparkOnYarnCliDriver failures, you probably
> should
> > > also
> > > > > > look
> > > > > > > > into the spark driver/application logs and see if there are
> > > > > > > infrastructure
> > > > > > > > errors (e.g OOMs). Are these tests failing when you run
> > locally?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Vihang
> > > > > > > >
> > > > > > > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj
> > > > > <rajaman@microsoft.com.invalid
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1,
> > > > > > > > > Thanks Stamatis and Lazlo for helping in the test case
> fixes
> > > till
> > > > > > now.
> > > > > > > > >
> > > > > > > > > Team,
> > > > > > > > > I need help in fixing the following tests in Hive. I have
> > tried
> > > > > > > different
> > > > > > > > > approaches but no luck till now.
> > > > > > > > > I am facing some issues in fixing the following tests :
> > > > > > > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > > > > > > >
> > > > > > > > > Issue :
> > > > > > > > > PREHOOK: Input: default@src
> > > > > > > > > PREHOOK: Output: default@src
> > > > > > > > > Failed to monitor Job[-1] with exception
> > > > > > > > > 'java.lang.IllegalStateException(Connection to remote Spark
> > > > driver
> > > > > > was
> > > > > > > > > lost)' Last known state = SENT
> > > > > > > > > Failed to execute spark task, with exception
> > > > > > > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > > > > > > FAILED: Execution Error, return code 1 from
> > > > > > > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel
> > is
> > > > > > closed.
> > > > > > > > >
> > > > > > > > > History :
> > > > > > > > > Initially the tests had failed with errors which I fixed in
> > the
> > > > > > > following
> > > > > > > > > task :
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8rdnoMslR2RT50AL3AANflY51KfU1yajCVTWEpUlyu8%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/HIVE-26940>
> > > > > > > > >
> > > > > > > > > Does anyone know what the issue is here ? There are 6-7
> > > failures
> > > > > > > because
> > > > > > > > > of this test case. Link to the failed test cases for the
> > > > > stacktrace :
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MeAJ3AqShjY4rpr82pYg1JfRSvtHRPKKWJgERVaP0fc%3D&reserved=0
> > > <
> > >
> >
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3949/2/tests/
> > > >
> > > > > > > > > Thanks,
> > > > > > > > > Aman.
> > > > > > > > >
> > > > > > > > > ________________________________
> > > > > > > > > From: László Bodor <bo...@gmail.com>
> > > > > > > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > > > Subject: [EXTERNAL] Re: Branch-3 backports and build
> > stability
> > > > > > > > >
> > > > > > > > > +1
> > > > > > > > > also, if I merged something that I thought was for test
> > > stability
> > > > > > (but
> > > > > > > > > instead it was a feature), excuse me :)
> > > > > > > > > for reference, the whole green test initiative is tracked
> > under
> > > > > this
> > > > > > > > > umbrella:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BGcj7NF8pSjr7SsYyOSe7o2VrNv2eH9YZ1ZFm4z7c6I%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/HIVE-26836>
> > > > > > > > >
> > > > > > > > > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont:
> > > 2023.
> > > > > > febr.
> > > > > > > > 7.,
> > > > > > > > > K, 12:09):
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > The build in branch-3 is not yet green; there are ~25
> test
> > > > > > failures.
> > > > > > > It
> > > > > > > > > is
> > > > > > > > > > a common practice that we shouldn't push changes on top
> of
> > a
> > > > > broken
> > > > > > > > build
> > > > > > > > > > unless they are addressing test failures.
> > > > > > > > > >
> > > > > > > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo
> > > Bodor)
> > > > > are
> > > > > > > > > working
> > > > > > > > > > hard to stabilize the build for quite some time now. If
> you
> > > > want
> > > > > to
> > > > > > > > help
> > > > > > > > > > out then start by reviewing, merging, and fixing things
> > > around
> > > > > test
> > > > > > > > > > failures.
> > > > > > > > > >
> > > > > > > > > > It's not yet the time to bring new features, upgrades,
> > bugs,
> > > > > etc.,
> > > > > > in
> > > > > > > > > > branch-3. I would encourage  committers to not approve
> such
> > > > > changes
> > > > > > > > till
> > > > > > > > > we
> > > > > > > > > > get back to a stable branch.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Stamatis
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [EXTERNAL] Re: Branch-3 backports and build stability

[Stamatis Zampetakis <za...@gmail.com>]

Huge thanks to everyone involved it is great to see the branch-3 in stable
state. As other people mentioned let's keep it that way!

As far as it concerns back ports please be particularly cautious with
anything that touches the metastore schema and Thrift APIs.

Best,
Stamatis

On Wed, Mar 29, 2023, 4:36 AM vihang karajgaonkar <vi...@apache.org>
wrote:

> Thanks a lot Aman for all your efforts on this. Really appreciate the
> initiative and all your hard work on this.
>
> I would like to request that all the committers should follow the merge
> process of master branch to merge PRs in branch-3. If there are any test
> failures which seem unrelated, please do not ignore them. One can run the
> flaky
> test runner <http://ci.hive.apache.org/job/hive-flaky-check/> to make sure
> that test is indeed flaky. If the test is found to be flaky a
> ticket should be created to disable it. A separate ticket should be created
> to deflake it and you can mention the original author or previous commit
> author who changed the test on that ticket to get help since they likely
> have the most context around that test. Once the flaky test is disabled and
> we have a green CI job run, we should merge the PR. If others have any
> suggestions to improve this process please chime in.
>
> Thanks,
> Vihang
>
> On Tue, Mar 28, 2023 at 10:55 PM Aman Raj <ra...@microsoft.com.invalid>
> wrote:
>
> > Hi community,
> >
> > This is to notify that we have a green branch-3 now. The entire effort of
> > fixing branch-3 test cases took around 4 months and as a team we managed
> to
> > fix 2900+ test failures on branch-3. The entire effort can be tracked
> here
> > HIVE-26836<https://issues.apache.org/jira/browse/HIVE-26836>. We are
> > ready to push new features and improvements on branch-3 now.
> >
> > I really want to thank Vihang Karajgaonkar, Chris Nauroth, Lazlo Bodor,
> > Stamatis Zampetakis and Sankar Hariappan without whom this would not at
> all
> > have been possible. As a team we stuck together and participated in
> reviews
> > and actively suggested improvements which really helped in fixing some
> > major test failures.
> >
> > I would sincerely request that going further it should be made a point to
> > merge things into branch-3 only if we have a green Jenkins pipeline.
> >
> > The next step would be to backport changes from branch-3.1 (From where
> > Hive-3.1.3 release was made) to branch-3. This would ensure that we do
> not
> > miss any specific ticket which went into Hive-3.1.3. I will take care of
> > this. We can parallelly start pushing additional changes on branch-3.
> There
> > are approximately 25 tickets that need to be backported in this effort
> (Of
> > backporting changes from branch-3.1). I have made a note here<
> >
> https://docs.google.com/spreadsheets/d/1K0U-vxLRZEs13oBzYBlVyK8dMMNthgXL5VEgzLRbeKs/edit?usp=sharing
> > >
> >
> > Again, thanks a lot to everyone who supported and participated in this
> > effort. Lets make this 3.2.0 Hive release happen!!
> >
> > Thanks,
> > Aman.
> >
> > ________________________________
> > From: Aman Raj <ra...@microsoft.com.INVALID>
> > Sent: Monday, March 20, 2023 9:21 AM
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > Hi Vihang/community,
> >
> > Found the ticket which broke mm_all.q. This issue comes because of
> > HIVE-20182. Works in my local and on the Jenkins pipeline as well. Link :
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F4127&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XSPlEtfWDNV%2Fccv9Q33xUtMLuhvxHx3CD4kC%2F5mWj2Y%3D&reserved=0
> > <https://github.com/apache/hive/pull/4127> Reverting this commit for
> now.
> >
> > Thanks,
> > Aman.
> > ________________________________
> > From: Aman Raj <ra...@microsoft.com.INVALID>
> > Sent: Monday, March 20, 2023 8:28 AM
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > Sure Vihang, will look at the other ones. You can pick this up.
> >
> > Thanks,
> > Aman.
> >
> > Get Outlook for Android<
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0
> > <https://aka.ms/AAb9ysg>>
> > ________________________________
> > From: vihang karajgaonkar <vi...@apache.org>
> > Sent: Monday, March 20, 2023 7:58:48 AM
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > I think we should revert offending commits first to unblock the branch.
> We
> > can create followup tickets to determine if these fixes are blockers for
> > 3.2 release and if yes, we should merge them the right way with a green
> > test run. Fixing forward always comes with the risk that it introduces
> new
> > test failures.
> >
> > Thanks for all your efforts on this Aman.
> >
> > I can take a look at testBootstrapReplLoadRetryAfterFailureForPartitions
> if
> > you haven’t already started on it.
> >
> > Thanks,
> > Vihang
> >
> > On Sun, Mar 19, 2023 at 10:09 PM Aman Raj <rajaman@microsoft.com.invalid
> >
> > wrote:
> >
> > > Hi Vihang/community,
> > >
> > > Thanks a lot Vihang for working on the major test failure. This blocked
> > > more than 35 test cases. Now we are down to the final 4 failures. I
> have
> > > analyzed some of them and here they are  (Link :
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-4067%2F12%2Ftests&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hiXJeNe9LPpWxhacjL2o3RUoalhcn86yog1IHz7JMHw%3D&reserved=0
> > )<
> >
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests
> > >
> > > :
> > >
> > >   1.
> > > multi_in_clause - This was committed in HIVE-21685 without validating
> the
> > > scenario.
> > > This fails because Hive is not able to parse
> > > explain cbo
> > > select * from very_simple_table_for_in_test where name IN('g','r') AND
> > > name IN('a','b')
> > > If we want this to work, I am able to do it in my local. We have 2
> > options
> > > :
> > > a. Either revert HIVE-21685 since this scenario was not validated back
> > > then before adding this test.
> > > b. This fix was present in
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FzY3laBCDchxxS2aFQ%2FTS3IYjOCxl%2FTTBFdQu9xBwUI%3D&reserved=0
> > <https://issues.apache.org/jira/browse/HIVE-20718> but to cherry pick
> this
> > > we need to cherry pick
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bjEWed1c5qwMjE1ZCJ1TRyHl%2FK1hADk9F05x8HO0f9A%3D&reserved=0
> > <https://issues.apache.org/jira/browse/HIVE-17040>
> > > since HIVE-20718<
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FzY3laBCDchxxS2aFQ%2FTS3IYjOCxl%2FTTBFdQu9xBwUI%3D&reserved=0
> > <https://issues.apache.org/jira/browse/HIVE-20718>> has a
> > > lot of merge conflicts with  HIVE-17040<
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bjEWed1c5qwMjE1ZCJ1TRyHl%2FK1hADk9F05x8HO0f9A%3D&reserved=0
> > ><https://issues.apache.org/jira/browse/HIVE-17040>. But after cherry
> > > picking these we have other failures to fix.
> > >   2.
> > > current_date_timestamp.q - This breaking change was committed in
> > > HIVE-21388 without validation.
> > > The failure is because again Hive is not able to parse
> > > explain cbo select current_timestamp() from alltypesorc
> > > The solution or revert option is same as point 1.
> > >   3.
> > > testBootstrapReplLoadRetryAfterFailureForPartitions() - This I have not
> > > investigated till now.
> > >   4.
> > > mm_all.q - This I have not investigated till now.
> > >
> > > Thanks,
> > > Aman.
> > > ________________________________
> > > From: vihang karajgaonkar <vi...@apache.org>
> > > Sent: Friday, March 17, 2023 8:42 PM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > >
> > > Just wanted to close the loop on the TestMiniSparkOnYarnCliDriver test
> > > failures. We will be able to re-enable most of them back on branch-3.
> The
> > > ones which were disabled are being tracked separately in a different
> > ticket
> > > <
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CkiQWoSy7LjWvSXr0dYY%2FusSXLIKMw27KIqItvgAfCc%3D&reserved=0
> > ><https://issues.apache.org/jira/browse/HIVE-27146>
> > > but they don't look like
> > > a blocker.
> > >
> > > Hi Aman,
> > >
> > > Do you know how close are we to reopening branch-3?
> > >
> > > Thanks,
> > > Vihang
> > >
> > > On Sat, Mar 4, 2023 at 7:23 PM Aman Raj <rajaman@microsoft.com.invalid
> >
> > > wrote:
> > >
> > > > Or you can cd into itests and run the command you are using. Just
> > another
> > > > way I run.
> > > >
> > > > Thanks,
> > > > Aman.
> > > > Get Outlook for Android<
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0
> > <https://aka.ms/AAb9ysg>
> > > >
> > > > ________________________________
> > > > From: Aman Raj <ra...@microsoft.com>
> > > > Sent: Saturday, March 4, 2023 7:20:36 PM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > Hi Vihang,
> > > >
> > > > Thanks a lot for working on this. Can you try using -Pqsplits,itests.
> > > > Also, I usually give a -o option after doing a clean install.
> > > >
> > > > Thanks,
> > > > Aman.
> > > >
> > > > Get Outlook for Android<
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0
> > <https://aka.ms/AAb9ysg>
> > > >
> > > >
> > > > ________________________________
> > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > Sent: Saturday, 4 March, 2023, 11:35
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > [You don't often get email from vihangk1@apache.org. Learn why this
> is
> > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > >
> > > > Just to update on the HoS test failures for
> > > TestMiniSparkOnYarnCliDriver, I
> > > > think I was finally able to resolve them (at least on local). I had
> to
> > > > revert HIVE-21044 because it was causing OOM for those tests. Also,
> in
> > > > order for these tests to work we will have to downgrade netty from
> > > > 4.1.69.Final to 4.1.51.Final. I understand that we had upgraded netty
> > > from
> > > > 4.1.17.Final to 4.1.69.Final for CVEs but the highest netty version
> > that
> > > we
> > > > can support without breaking HoS is 4.1.51.Final. Note that
> > 4.1.51.Final
> > > > includes many of the CVEs which affected 4.1.17.Final so we are still
> > in
> > > a
> > > > better place than branch-3.1. Unfortunately, there is no good way to
> > make
> > > > HoS work with a higher netty version so I think we should downgrade
> the
> > > > netty version to 4.1.51.Final for now and look at more options to
> > upgrade
> > > > it 4.1.69.Final in a separate ticket.
> > > >
> > > > I still need to understand why the tests which are working for me
> > locally
> > > > don't work on the PR job. I tried running the split test classes
> using
> > > the
> > > > following command. Is that the right way to simulate builds from the
> PR
> > > > job? Let me know if anyone has more ideas.
> > > >
> > > > mvn test
> > > > -Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
> > > > -Pqsplits
> > > >
> > > > Thanks,
> > > > Vihang
> > > >
> > > >
> > > > On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <
> zabetak@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > Thanks Aman for bringing this up and also for cleaning up after
> > others
> > > (I
> > > > > saw that you raised tickets and PRs for addressing the failures).
> > > > >
> > > > > Many thanks to Vihang as well for helping out. Regarding flaky
> tests,
> > > yes
> > > > > we should disable them as soon as we see them.
> > > > > There have been some other discussions on how to approach flaky
> tests
> > > the
> > > > > more recent I could find is here [1].
> > > > >
> > > > > Best,
> > > > > Stamatis
> > > > >
> > > > > [1]
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XZWL3W%2BHTikea8Du4Ohy3%2FHNTwDZBBxwXk5ylqRX0sU%3D&reserved=0
> > <https://lists.apache.org/thread/lv3bhlfoq8fwd9dwyjf7g4nx32wtrygv>
> > > > >
> > > > > On Fri, Feb 17, 2023 at 4:37 AM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi team,
> > > > > >
> > > > > > Thanks Vihang for looking into this. I have commented on the JIRA
> > you
> > > > > > created.
> > > > > >
> > > > > > Just to bring everyone's notice, I have seen that there has been
> a
> > > > couple
> > > > > > of pushes to branch-3, which has lead to 5 more new test
> failures.
> > > The
> > > > > test
> > > > > > failures are in orc_merge1, orc_merge2, orc_merge3, orc_merge4
> and
> > > > > > orc_merge10. These tests did not use to fail before. I would
> > > sincerely
> > > > > urge
> > > > > > the community to raise a PR against branch-3, so that the Jenkins
> > > > > pipeline
> > > > > > can run and then only merge things to branch-3. We had 2900+
> > failures
> > > > > when
> > > > > > we started 2 months back and now having brought it down to less
> > than
> > > > 15,
> > > > > > new failures again has pushed us back in this effort.
> > > > > >
> > > > > > I would like to thank everyone who has participated in this
> effort
> > > and
> > > > > > made it possible till this stage. Also, if the contributors can
> > take
> > > > > > ownership of these new test case failures and fix them, it will
> be
> > of
> > > > > great
> > > > > > help.
> > > > > >
> > > > > > Thanks,
> > > > > > Aman.
> > > > > > ________________________________
> > > > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > > > Sent: Friday, February 17, 2023 6:10 AM
> > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> stability
> > > > > >
> > > > > > [You don't often get email from vihangk1@apache.org. Learn why
> > this
> > > is
> > > > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > > > >
> > > > > > Hi Aman,
> > > > > >
> > > > > > I created
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5UlK9DVcNIyVkzyMld3%2F1deJaV1TsLMAY2lDV3kjlrQ%3D&reserved=0
> > <https://issues.apache.org/jira/browse/HIVE-27087>
> > > > > > to look into
> > > > > > TestMiniSparkOnYarnCliDriver failures. I have a working theory of
> > > what
> > > > > > might be going on there. I am still investigating what is the
> right
> > > way
> > > > > to
> > > > > > fix it though.
> > > > > >
> > > > > > Thanks,
> > > > > > Vihang
> > > > > >
> > > > > > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj
> > > > <rajaman@microsoft.com.invalid
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Vihang,
> > > > > > >
> > > > > > > Yes the tests are failing locally as well with the same issue.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Aman.
> > > > > > >
> > > > > > > Get Outlook for Android<
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0
> > <https://aka.ms/AAb9ysg>
> > > > > > >
> > > > > > > ________________________________
> > > > > > > From: Vihang Karajgaonkar
> > > <vihang.karajgaonkar@databricks.com.INVALID
> > > > >
> > > > > > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> > stability
> > > > > > >
> > > > > > > [You don't often get email from
> > > > > > vihang.karajgaonkar@databricks.com.invalid.
> > > > > > > Learn why this is important at
> > > > > > > https://aka.ms/LearnAboutSenderIdentification ]
> > > > > > >
> > > > > > > Thanks a lot Stamatis for starting this thread. I really
> > appreciate
> > > > all
> > > > > > the
> > > > > > > efforts to stabilize branch-3 to get it to a releasable state
> > and I
> > > > > agree
> > > > > > > that we should get it to a green state before opening it for
> PRs
> > > not
> > > > > > > related to test failures. I can help with the effort as well.
> > > > > > >
> > > > > > > If we want to get the branch back to green state soon, have we
> > > > > considered
> > > > > > > disabling the tests which are clearly flaky? (e.g pass on some
> > > builds
> > > > > and
> > > > > > > fail on the other build with no new code changes). If we don't
> do
> > > > that,
> > > > > > we
> > > > > > > will keep playing whack a mole with those tests. I propose for
> > such
> > > > > tests
> > > > > > > we should disable them and create tickets to unflake them
> > > separately.
> > > > > > This
> > > > > > > will help us get back to a green state faster.
> > > > > > >
> > > > > > > Hi Aman,
> > > > > > > For TestMiniSparkOnYarnCliDriver failures, you probably should
> > also
> > > > > look
> > > > > > > into the spark driver/application logs and see if there are
> > > > > > infrastructure
> > > > > > > errors (e.g OOMs). Are these tests failing when you run
> locally?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Vihang
> > > > > > >
> > > > > > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj
> > > > <rajaman@microsoft.com.invalid
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1,
> > > > > > > > Thanks Stamatis and Lazlo for helping in the test case fixes
> > till
> > > > > now.
> > > > > > > >
> > > > > > > > Team,
> > > > > > > > I need help in fixing the following tests in Hive. I have
> tried
> > > > > > different
> > > > > > > > approaches but no luck till now.
> > > > > > > > I am facing some issues in fixing the following tests :
> > > > > > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > > > > > >
> > > > > > > > Issue :
> > > > > > > > PREHOOK: Input: default@src
> > > > > > > > PREHOOK: Output: default@src
> > > > > > > > Failed to monitor Job[-1] with exception
> > > > > > > > 'java.lang.IllegalStateException(Connection to remote Spark
> > > driver
> > > > > was
> > > > > > > > lost)' Last known state = SENT
> > > > > > > > Failed to execute spark task, with exception
> > > > > > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > > > > > FAILED: Execution Error, return code 1 from
> > > > > > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel
> is
> > > > > closed.
> > > > > > > >
> > > > > > > > History :
> > > > > > > > Initially the tests had failed with errors which I fixed in
> the
> > > > > > following
> > > > > > > > task :
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8rdnoMslR2RT50AL3AANflY51KfU1yajCVTWEpUlyu8%3D&reserved=0
> > <https://issues.apache.org/jira/browse/HIVE-26940>
> > > > > > > >
> > > > > > > > Does anyone know what the issue is here ? There are 6-7
> > failures
> > > > > > because
> > > > > > > > of this test case. Link to the failed test cases for the
> > > > stacktrace :
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MeAJ3AqShjY4rpr82pYg1JfRSvtHRPKKWJgERVaP0fc%3D&reserved=0
> > <
> >
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3949/2/tests/
> > >
> > > > > > > > Thanks,
> > > > > > > > Aman.
> > > > > > > >
> > > > > > > > ________________________________
> > > > > > > > From: László Bodor <bo...@gmail.com>
> > > > > > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > > Subject: [EXTERNAL] Re: Branch-3 backports and build
> stability
> > > > > > > >
> > > > > > > > +1
> > > > > > > > also, if I merged something that I thought was for test
> > stability
> > > > > (but
> > > > > > > > instead it was a feature), excuse me :)
> > > > > > > > for reference, the whole green test initiative is tracked
> under
> > > > this
> > > > > > > > umbrella:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BGcj7NF8pSjr7SsYyOSe7o2VrNv2eH9YZ1ZFm4z7c6I%3D&reserved=0
> > <https://issues.apache.org/jira/browse/HIVE-26836>
> > > > > > > >
> > > > > > > > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont:
> > 2023.
> > > > > febr.
> > > > > > > 7.,
> > > > > > > > K, 12:09):
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > The build in branch-3 is not yet green; there are ~25 test
> > > > > failures.
> > > > > > It
> > > > > > > > is
> > > > > > > > > a common practice that we shouldn't push changes on top of
> a
> > > > broken
> > > > > > > build
> > > > > > > > > unless they are addressing test failures.
> > > > > > > > >
> > > > > > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo
> > Bodor)
> > > > are
> > > > > > > > working
> > > > > > > > > hard to stabilize the build for quite some time now. If you
> > > want
> > > > to
> > > > > > > help
> > > > > > > > > out then start by reviewing, merging, and fixing things
> > around
> > > > test
> > > > > > > > > failures.
> > > > > > > > >
> > > > > > > > > It's not yet the time to bring new features, upgrades,
> bugs,
> > > > etc.,
> > > > > in
> > > > > > > > > branch-3. I would encourage  committers to not approve such
> > > > changes
> > > > > > > till
> > > > > > > > we
> > > > > > > > > get back to a stable branch.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Stamatis
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
>

Re: [EXTERNAL] Re: Branch-3 backports and build stability

[vihang karajgaonkar <vi...@apache.org>]

Thanks a lot Aman for all your efforts on this. Really appreciate the
initiative and all your hard work on this.

I would like to request that all the committers should follow the merge
process of master branch to merge PRs in branch-3. If there are any test
failures which seem unrelated, please do not ignore them. One can run the flaky
test runner <http://ci.hive.apache.org/job/hive-flaky-check/> to make sure
that test is indeed flaky. If the test is found to be flaky a
ticket should be created to disable it. A separate ticket should be created
to deflake it and you can mention the original author or previous commit
author who changed the test on that ticket to get help since they likely
have the most context around that test. Once the flaky test is disabled and
we have a green CI job run, we should merge the PR. If others have any
suggestions to improve this process please chime in.

Thanks,
Vihang

On Tue, Mar 28, 2023 at 10:55 PM Aman Raj <ra...@microsoft.com.invalid>
wrote:

> Hi community,
>
> This is to notify that we have a green branch-3 now. The entire effort of
> fixing branch-3 test cases took around 4 months and as a team we managed to
> fix 2900+ test failures on branch-3. The entire effort can be tracked here
> HIVE-26836<https://issues.apache.org/jira/browse/HIVE-26836>. We are
> ready to push new features and improvements on branch-3 now.
>
> I really want to thank Vihang Karajgaonkar, Chris Nauroth, Lazlo Bodor,
> Stamatis Zampetakis and Sankar Hariappan without whom this would not at all
> have been possible. As a team we stuck together and participated in reviews
> and actively suggested improvements which really helped in fixing some
> major test failures.
>
> I would sincerely request that going further it should be made a point to
> merge things into branch-3 only if we have a green Jenkins pipeline.
>
> The next step would be to backport changes from branch-3.1 (From where
> Hive-3.1.3 release was made) to branch-3. This would ensure that we do not
> miss any specific ticket which went into Hive-3.1.3. I will take care of
> this. We can parallelly start pushing additional changes on branch-3. There
> are approximately 25 tickets that need to be backported in this effort (Of
> backporting changes from branch-3.1). I have made a note here<
> https://docs.google.com/spreadsheets/d/1K0U-vxLRZEs13oBzYBlVyK8dMMNthgXL5VEgzLRbeKs/edit?usp=sharing
> >
>
> Again, thanks a lot to everyone who supported and participated in this
> effort. Lets make this 3.2.0 Hive release happen!!
>
> Thanks,
> Aman.
>
> ________________________________
> From: Aman Raj <ra...@microsoft.com.INVALID>
> Sent: Monday, March 20, 2023 9:21 AM
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
>
> Hi Vihang/community,
>
> Found the ticket which broke mm_all.q. This issue comes because of
> HIVE-20182. Works in my local and on the Jenkins pipeline as well. Link :
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F4127&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XSPlEtfWDNV%2Fccv9Q33xUtMLuhvxHx3CD4kC%2F5mWj2Y%3D&reserved=0
> <https://github.com/apache/hive/pull/4127> Reverting this commit for now.
>
> Thanks,
> Aman.
> ________________________________
> From: Aman Raj <ra...@microsoft.com.INVALID>
> Sent: Monday, March 20, 2023 8:28 AM
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
>
> Sure Vihang, will look at the other ones. You can pick this up.
>
> Thanks,
> Aman.
>
> Get Outlook for Android<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0
> <https://aka.ms/AAb9ysg>>
> ________________________________
> From: vihang karajgaonkar <vi...@apache.org>
> Sent: Monday, March 20, 2023 7:58:48 AM
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
>
> I think we should revert offending commits first to unblock the branch. We
> can create followup tickets to determine if these fixes are blockers for
> 3.2 release and if yes, we should merge them the right way with a green
> test run. Fixing forward always comes with the risk that it introduces new
> test failures.
>
> Thanks for all your efforts on this Aman.
>
> I can take a look at testBootstrapReplLoadRetryAfterFailureForPartitions if
> you haven’t already started on it.
>
> Thanks,
> Vihang
>
> On Sun, Mar 19, 2023 at 10:09 PM Aman Raj <ra...@microsoft.com.invalid>
> wrote:
>
> > Hi Vihang/community,
> >
> > Thanks a lot Vihang for working on the major test failure. This blocked
> > more than 35 test cases. Now we are down to the final 4 failures. I have
> > analyzed some of them and here they are  (Link :
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-4067%2F12%2Ftests&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hiXJeNe9LPpWxhacjL2o3RUoalhcn86yog1IHz7JMHw%3D&reserved=0
> )<
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests
> >
> > :
> >
> >   1.
> > multi_in_clause - This was committed in HIVE-21685 without validating the
> > scenario.
> > This fails because Hive is not able to parse
> > explain cbo
> > select * from very_simple_table_for_in_test where name IN('g','r') AND
> > name IN('a','b')
> > If we want this to work, I am able to do it in my local. We have 2
> options
> > :
> > a. Either revert HIVE-21685 since this scenario was not validated back
> > then before adding this test.
> > b. This fix was present in
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FzY3laBCDchxxS2aFQ%2FTS3IYjOCxl%2FTTBFdQu9xBwUI%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-20718> but to cherry pick this
> > we need to cherry pick
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bjEWed1c5qwMjE1ZCJ1TRyHl%2FK1hADk9F05x8HO0f9A%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-17040>
> > since HIVE-20718<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FzY3laBCDchxxS2aFQ%2FTS3IYjOCxl%2FTTBFdQu9xBwUI%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-20718>> has a
> > lot of merge conflicts with  HIVE-17040<
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bjEWed1c5qwMjE1ZCJ1TRyHl%2FK1hADk9F05x8HO0f9A%3D&reserved=0
> ><https://issues.apache.org/jira/browse/HIVE-17040>. But after cherry
> > picking these we have other failures to fix.
> >   2.
> > current_date_timestamp.q - This breaking change was committed in
> > HIVE-21388 without validation.
> > The failure is because again Hive is not able to parse
> > explain cbo select current_timestamp() from alltypesorc
> > The solution or revert option is same as point 1.
> >   3.
> > testBootstrapReplLoadRetryAfterFailureForPartitions() - This I have not
> > investigated till now.
> >   4.
> > mm_all.q - This I have not investigated till now.
> >
> > Thanks,
> > Aman.
> > ________________________________
> > From: vihang karajgaonkar <vi...@apache.org>
> > Sent: Friday, March 17, 2023 8:42 PM
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > Just wanted to close the loop on the TestMiniSparkOnYarnCliDriver test
> > failures. We will be able to re-enable most of them back on branch-3. The
> > ones which were disabled are being tracked separately in a different
> ticket
> > <
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CkiQWoSy7LjWvSXr0dYY%2FusSXLIKMw27KIqItvgAfCc%3D&reserved=0
> ><https://issues.apache.org/jira/browse/HIVE-27146>
> > but they don't look like
> > a blocker.
> >
> > Hi Aman,
> >
> > Do you know how close are we to reopening branch-3?
> >
> > Thanks,
> > Vihang
> >
> > On Sat, Mar 4, 2023 at 7:23 PM Aman Raj <ra...@microsoft.com.invalid>
> > wrote:
> >
> > > Or you can cd into itests and run the command you are using. Just
> another
> > > way I run.
> > >
> > > Thanks,
> > > Aman.
> > > Get Outlook for Android<
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0
> <https://aka.ms/AAb9ysg>
> > >
> > > ________________________________
> > > From: Aman Raj <ra...@microsoft.com>
> > > Sent: Saturday, March 4, 2023 7:20:36 PM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > >
> > > Hi Vihang,
> > >
> > > Thanks a lot for working on this. Can you try using -Pqsplits,itests.
> > > Also, I usually give a -o option after doing a clean install.
> > >
> > > Thanks,
> > > Aman.
> > >
> > > Get Outlook for Android<
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0
> <https://aka.ms/AAb9ysg>
> > >
> > >
> > > ________________________________
> > > From: vihang karajgaonkar <vi...@apache.org>
> > > Sent: Saturday, 4 March, 2023, 11:35
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > >
> > > [You don't often get email from vihangk1@apache.org. Learn why this is
> > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > >
> > > Just to update on the HoS test failures for
> > TestMiniSparkOnYarnCliDriver, I
> > > think I was finally able to resolve them (at least on local). I had to
> > > revert HIVE-21044 because it was causing OOM for those tests. Also, in
> > > order for these tests to work we will have to downgrade netty from
> > > 4.1.69.Final to 4.1.51.Final. I understand that we had upgraded netty
> > from
> > > 4.1.17.Final to 4.1.69.Final for CVEs but the highest netty version
> that
> > we
> > > can support without breaking HoS is 4.1.51.Final. Note that
> 4.1.51.Final
> > > includes many of the CVEs which affected 4.1.17.Final so we are still
> in
> > a
> > > better place than branch-3.1. Unfortunately, there is no good way to
> make
> > > HoS work with a higher netty version so I think we should downgrade the
> > > netty version to 4.1.51.Final for now and look at more options to
> upgrade
> > > it 4.1.69.Final in a separate ticket.
> > >
> > > I still need to understand why the tests which are working for me
> locally
> > > don't work on the PR job. I tried running the split test classes using
> > the
> > > following command. Is that the right way to simulate builds from the PR
> > > job? Let me know if anyone has more ideas.
> > >
> > > mvn test
> > > -Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
> > > -Pqsplits
> > >
> > > Thanks,
> > > Vihang
> > >
> > >
> > > On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <zabetak@gmail.com
> >
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > Thanks Aman for bringing this up and also for cleaning up after
> others
> > (I
> > > > saw that you raised tickets and PRs for addressing the failures).
> > > >
> > > > Many thanks to Vihang as well for helping out. Regarding flaky tests,
> > yes
> > > > we should disable them as soon as we see them.
> > > > There have been some other discussions on how to approach flaky tests
> > the
> > > > more recent I could find is here [1].
> > > >
> > > > Best,
> > > > Stamatis
> > > >
> > > > [1]
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XZWL3W%2BHTikea8Du4Ohy3%2FHNTwDZBBxwXk5ylqRX0sU%3D&reserved=0
> <https://lists.apache.org/thread/lv3bhlfoq8fwd9dwyjf7g4nx32wtrygv>
> > > >
> > > > On Fri, Feb 17, 2023 at 4:37 AM Aman Raj
> <rajaman@microsoft.com.invalid
> > >
> > > > wrote:
> > > >
> > > > > Hi team,
> > > > >
> > > > > Thanks Vihang for looking into this. I have commented on the JIRA
> you
> > > > > created.
> > > > >
> > > > > Just to bring everyone's notice, I have seen that there has been a
> > > couple
> > > > > of pushes to branch-3, which has lead to 5 more new test failures.
> > The
> > > > test
> > > > > failures are in orc_merge1, orc_merge2, orc_merge3, orc_merge4 and
> > > > > orc_merge10. These tests did not use to fail before. I would
> > sincerely
> > > > urge
> > > > > the community to raise a PR against branch-3, so that the Jenkins
> > > > pipeline
> > > > > can run and then only merge things to branch-3. We had 2900+
> failures
> > > > when
> > > > > we started 2 months back and now having brought it down to less
> than
> > > 15,
> > > > > new failures again has pushed us back in this effort.
> > > > >
> > > > > I would like to thank everyone who has participated in this effort
> > and
> > > > > made it possible till this stage. Also, if the contributors can
> take
> > > > > ownership of these new test case failures and fix them, it will be
> of
> > > > great
> > > > > help.
> > > > >
> > > > > Thanks,
> > > > > Aman.
> > > > > ________________________________
> > > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > > Sent: Friday, February 17, 2023 6:10 AM
> > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > >
> > > > > [You don't often get email from vihangk1@apache.org. Learn why
> this
> > is
> > > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > > >
> > > > > Hi Aman,
> > > > >
> > > > > I created
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5UlK9DVcNIyVkzyMld3%2F1deJaV1TsLMAY2lDV3kjlrQ%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-27087>
> > > > > to look into
> > > > > TestMiniSparkOnYarnCliDriver failures. I have a working theory of
> > what
> > > > > might be going on there. I am still investigating what is the right
> > way
> > > > to
> > > > > fix it though.
> > > > >
> > > > > Thanks,
> > > > > Vihang
> > > > >
> > > > > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj
> > > <rajaman@microsoft.com.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Vihang,
> > > > > >
> > > > > > Yes the tests are failing locally as well with the same issue.
> > > > > >
> > > > > > Thanks,
> > > > > > Aman.
> > > > > >
> > > > > > Get Outlook for Android<
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0
> <https://aka.ms/AAb9ysg>
> > > > > >
> > > > > > ________________________________
> > > > > > From: Vihang Karajgaonkar
> > <vihang.karajgaonkar@databricks.com.INVALID
> > > >
> > > > > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build
> stability
> > > > > >
> > > > > > [You don't often get email from
> > > > > vihang.karajgaonkar@databricks.com.invalid.
> > > > > > Learn why this is important at
> > > > > > https://aka.ms/LearnAboutSenderIdentification ]
> > > > > >
> > > > > > Thanks a lot Stamatis for starting this thread. I really
> appreciate
> > > all
> > > > > the
> > > > > > efforts to stabilize branch-3 to get it to a releasable state
> and I
> > > > agree
> > > > > > that we should get it to a green state before opening it for PRs
> > not
> > > > > > related to test failures. I can help with the effort as well.
> > > > > >
> > > > > > If we want to get the branch back to green state soon, have we
> > > > considered
> > > > > > disabling the tests which are clearly flaky? (e.g pass on some
> > builds
> > > > and
> > > > > > fail on the other build with no new code changes). If we don't do
> > > that,
> > > > > we
> > > > > > will keep playing whack a mole with those tests. I propose for
> such
> > > > tests
> > > > > > we should disable them and create tickets to unflake them
> > separately.
> > > > > This
> > > > > > will help us get back to a green state faster.
> > > > > >
> > > > > > Hi Aman,
> > > > > > For TestMiniSparkOnYarnCliDriver failures, you probably should
> also
> > > > look
> > > > > > into the spark driver/application logs and see if there are
> > > > > infrastructure
> > > > > > errors (e.g OOMs). Are these tests failing when you run locally?
> > > > > >
> > > > > > Thanks,
> > > > > > Vihang
> > > > > >
> > > > > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj
> > > <rajaman@microsoft.com.invalid
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1,
> > > > > > > Thanks Stamatis and Lazlo for helping in the test case fixes
> till
> > > > now.
> > > > > > >
> > > > > > > Team,
> > > > > > > I need help in fixing the following tests in Hive. I have tried
> > > > > different
> > > > > > > approaches but no luck till now.
> > > > > > > I am facing some issues in fixing the following tests :
> > > > > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > > > > >
> > > > > > > Issue :
> > > > > > > PREHOOK: Input: default@src
> > > > > > > PREHOOK: Output: default@src
> > > > > > > Failed to monitor Job[-1] with exception
> > > > > > > 'java.lang.IllegalStateException(Connection to remote Spark
> > driver
> > > > was
> > > > > > > lost)' Last known state = SENT
> > > > > > > Failed to execute spark task, with exception
> > > > > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > > > > FAILED: Execution Error, return code 1 from
> > > > > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel is
> > > > closed.
> > > > > > >
> > > > > > > History :
> > > > > > > Initially the tests had failed with errors which I fixed in the
> > > > > following
> > > > > > > task :
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8rdnoMslR2RT50AL3AANflY51KfU1yajCVTWEpUlyu8%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-26940>
> > > > > > >
> > > > > > > Does anyone know what the issue is here ? There are 6-7
> failures
> > > > > because
> > > > > > > of this test case. Link to the failed test cases for the
> > > stacktrace :
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MeAJ3AqShjY4rpr82pYg1JfRSvtHRPKKWJgERVaP0fc%3D&reserved=0
> <
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3949/2/tests/
> >
> > > > > > > Thanks,
> > > > > > > Aman.
> > > > > > >
> > > > > > > ________________________________
> > > > > > > From: László Bodor <bo...@gmail.com>
> > > > > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > Subject: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > > > >
> > > > > > > +1
> > > > > > > also, if I merged something that I thought was for test
> stability
> > > > (but
> > > > > > > instead it was a feature), excuse me :)
> > > > > > > for reference, the whole green test initiative is tracked under
> > > this
> > > > > > > umbrella:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BGcj7NF8pSjr7SsYyOSe7o2VrNv2eH9YZ1ZFm4z7c6I%3D&reserved=0
> <https://issues.apache.org/jira/browse/HIVE-26836>
> > > > > > >
> > > > > > > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont:
> 2023.
> > > > febr.
> > > > > > 7.,
> > > > > > > K, 12:09):
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > The build in branch-3 is not yet green; there are ~25 test
> > > > failures.
> > > > > It
> > > > > > > is
> > > > > > > > a common practice that we shouldn't push changes on top of a
> > > broken
> > > > > > build
> > > > > > > > unless they are addressing test failures.
> > > > > > > >
> > > > > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo
> Bodor)
> > > are
> > > > > > > working
> > > > > > > > hard to stabilize the build for quite some time now. If you
> > want
> > > to
> > > > > > help
> > > > > > > > out then start by reviewing, merging, and fixing things
> around
> > > test
> > > > > > > > failures.
> > > > > > > >
> > > > > > > > It's not yet the time to bring new features, upgrades, bugs,
> > > etc.,
> > > > in
> > > > > > > > branch-3. I would encourage  committers to not approve such
> > > changes
> > > > > > till
> > > > > > > we
> > > > > > > > get back to a stable branch.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Stamatis
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> >
>

Re: [EXTERNAL] Re: Branch-3 backports and build stability

[Aman Raj <ra...@microsoft.com.INVALID>]

Hi community,

This is to notify that we have a green branch-3 now. The entire effort of fixing branch-3 test cases took around 4 months and as a team we managed to fix 2900+ test failures on branch-3. The entire effort can be tracked here HIVE-26836<https://issues.apache.org/jira/browse/HIVE-26836>. We are ready to push new features and improvements on branch-3 now.

I really want to thank Vihang Karajgaonkar, Chris Nauroth, Lazlo Bodor, Stamatis Zampetakis and Sankar Hariappan without whom this would not at all have been possible. As a team we stuck together and participated in reviews and actively suggested improvements which really helped in fixing some major test failures.

I would sincerely request that going further it should be made a point to merge things into branch-3 only if we have a green Jenkins pipeline.

The next step would be to backport changes from branch-3.1 (From where Hive-3.1.3 release was made) to branch-3. This would ensure that we do not miss any specific ticket which went into Hive-3.1.3. I will take care of this. We can parallelly start pushing additional changes on branch-3. There are approximately 25 tickets that need to be backported in this effort (Of backporting changes from branch-3.1). I have made a note here<https://docs.google.com/spreadsheets/d/1K0U-vxLRZEs13oBzYBlVyK8dMMNthgXL5VEgzLRbeKs/edit?usp=sharing>

Again, thanks a lot to everyone who supported and participated in this effort. Lets make this 3.2.0 Hive release happen!!

Thanks,
Aman.

________________________________
From: Aman Raj <ra...@microsoft.com.INVALID>
Sent: Monday, March 20, 2023 9:21 AM
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability

Hi Vihang/community,

Found the ticket which broke mm_all.q. This issue comes because of HIVE-20182. Works in my local and on the Jenkins pipeline as well. Link : https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F4127&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XSPlEtfWDNV%2Fccv9Q33xUtMLuhvxHx3CD4kC%2F5mWj2Y%3D&reserved=0<https://github.com/apache/hive/pull/4127> Reverting this commit for now.

Thanks,
Aman.
________________________________
From: Aman Raj <ra...@microsoft.com.INVALID>
Sent: Monday, March 20, 2023 8:28 AM
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability

Sure Vihang, will look at the other ones. You can pick this up.

Thanks,
Aman.

Get Outlook for Android<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0<https://aka.ms/AAb9ysg>>
________________________________
From: vihang karajgaonkar <vi...@apache.org>
Sent: Monday, March 20, 2023 7:58:48 AM
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability

I think we should revert offending commits first to unblock the branch. We
can create followup tickets to determine if these fixes are blockers for
3.2 release and if yes, we should merge them the right way with a green
test run. Fixing forward always comes with the risk that it introduces new
test failures.

Thanks for all your efforts on this Aman.

I can take a look at testBootstrapReplLoadRetryAfterFailureForPartitions if
you haven’t already started on it.

Thanks,
Vihang

On Sun, Mar 19, 2023 at 10:09 PM Aman Raj <ra...@microsoft.com.invalid>
wrote:

> Hi Vihang/community,
>
> Thanks a lot Vihang for working on the major test failure. This blocked
> more than 35 test cases. Now we are down to the final 4 failures. I have
> analyzed some of them and here they are  (Link :
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-4067%2F12%2Ftests&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hiXJeNe9LPpWxhacjL2o3RUoalhcn86yog1IHz7JMHw%3D&reserved=0)<http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests>
> :
>
>   1.
> multi_in_clause - This was committed in HIVE-21685 without validating the
> scenario.
> This fails because Hive is not able to parse
> explain cbo
> select * from very_simple_table_for_in_test where name IN('g','r') AND
> name IN('a','b')
> If we want this to work, I am able to do it in my local. We have 2 options
> :
> a. Either revert HIVE-21685 since this scenario was not validated back
> then before adding this test.
> b. This fix was present in
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FzY3laBCDchxxS2aFQ%2FTS3IYjOCxl%2FTTBFdQu9xBwUI%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-20718> but to cherry pick this
> we need to cherry pick https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bjEWed1c5qwMjE1ZCJ1TRyHl%2FK1hADk9F05x8HO0f9A%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-17040>
> since HIVE-20718<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FzY3laBCDchxxS2aFQ%2FTS3IYjOCxl%2FTTBFdQu9xBwUI%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-20718>> has a
> lot of merge conflicts with  HIVE-17040<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bjEWed1c5qwMjE1ZCJ1TRyHl%2FK1hADk9F05x8HO0f9A%3D&reserved=0><https://issues.apache.org/jira/browse/HIVE-17040>. But after cherry
> picking these we have other failures to fix.
>   2.
> current_date_timestamp.q - This breaking change was committed in
> HIVE-21388 without validation.
> The failure is because again Hive is not able to parse
> explain cbo select current_timestamp() from alltypesorc
> The solution or revert option is same as point 1.
>   3.
> testBootstrapReplLoadRetryAfterFailureForPartitions() - This I have not
> investigated till now.
>   4.
> mm_all.q - This I have not investigated till now.
>
> Thanks,
> Aman.
> ________________________________
> From: vihang karajgaonkar <vi...@apache.org>
> Sent: Friday, March 17, 2023 8:42 PM
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
>
> Just wanted to close the loop on the TestMiniSparkOnYarnCliDriver test
> failures. We will be able to re-enable most of them back on branch-3. The
> ones which were disabled are being tracked separately in a different ticket
> <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CkiQWoSy7LjWvSXr0dYY%2FusSXLIKMw27KIqItvgAfCc%3D&reserved=0><https://issues.apache.org/jira/browse/HIVE-27146>
> but they don't look like
> a blocker.
>
> Hi Aman,
>
> Do you know how close are we to reopening branch-3?
>
> Thanks,
> Vihang
>
> On Sat, Mar 4, 2023 at 7:23 PM Aman Raj <ra...@microsoft.com.invalid>
> wrote:
>
> > Or you can cd into itests and run the command you are using. Just another
> > way I run.
> >
> > Thanks,
> > Aman.
> > Get Outlook for Android<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0<https://aka.ms/AAb9ysg>
> >
> > ________________________________
> > From: Aman Raj <ra...@microsoft.com>
> > Sent: Saturday, March 4, 2023 7:20:36 PM
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > Hi Vihang,
> >
> > Thanks a lot for working on this. Can you try using -Pqsplits,itests.
> > Also, I usually give a -o option after doing a clean install.
> >
> > Thanks,
> > Aman.
> >
> > Get Outlook for Android<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0<https://aka.ms/AAb9ysg>
> >
> >
> > ________________________________
> > From: vihang karajgaonkar <vi...@apache.org>
> > Sent: Saturday, 4 March, 2023, 11:35
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > [You don't often get email from vihangk1@apache.org. Learn why this is
> > important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > Just to update on the HoS test failures for
> TestMiniSparkOnYarnCliDriver, I
> > think I was finally able to resolve them (at least on local). I had to
> > revert HIVE-21044 because it was causing OOM for those tests. Also, in
> > order for these tests to work we will have to downgrade netty from
> > 4.1.69.Final to 4.1.51.Final. I understand that we had upgraded netty
> from
> > 4.1.17.Final to 4.1.69.Final for CVEs but the highest netty version that
> we
> > can support without breaking HoS is 4.1.51.Final. Note that 4.1.51.Final
> > includes many of the CVEs which affected 4.1.17.Final so we are still in
> a
> > better place than branch-3.1. Unfortunately, there is no good way to make
> > HoS work with a higher netty version so I think we should downgrade the
> > netty version to 4.1.51.Final for now and look at more options to upgrade
> > it 4.1.69.Final in a separate ticket.
> >
> > I still need to understand why the tests which are working for me locally
> > don't work on the PR job. I tried running the split test classes using
> the
> > following command. Is that the right way to simulate builds from the PR
> > job? Let me know if anyone has more ideas.
> >
> > mvn test
> > -Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
> > -Pqsplits
> >
> > Thanks,
> > Vihang
> >
> >
> > On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <za...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > Thanks Aman for bringing this up and also for cleaning up after others
> (I
> > > saw that you raised tickets and PRs for addressing the failures).
> > >
> > > Many thanks to Vihang as well for helping out. Regarding flaky tests,
> yes
> > > we should disable them as soon as we see them.
> > > There have been some other discussions on how to approach flaky tests
> the
> > > more recent I could find is here [1].
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1]
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XZWL3W%2BHTikea8Du4Ohy3%2FHNTwDZBBxwXk5ylqRX0sU%3D&reserved=0<https://lists.apache.org/thread/lv3bhlfoq8fwd9dwyjf7g4nx32wtrygv>
> > >
> > > On Fri, Feb 17, 2023 at 4:37 AM Aman Raj <rajaman@microsoft.com.invalid
> >
> > > wrote:
> > >
> > > > Hi team,
> > > >
> > > > Thanks Vihang for looking into this. I have commented on the JIRA you
> > > > created.
> > > >
> > > > Just to bring everyone's notice, I have seen that there has been a
> > couple
> > > > of pushes to branch-3, which has lead to 5 more new test failures.
> The
> > > test
> > > > failures are in orc_merge1, orc_merge2, orc_merge3, orc_merge4 and
> > > > orc_merge10. These tests did not use to fail before. I would
> sincerely
> > > urge
> > > > the community to raise a PR against branch-3, so that the Jenkins
> > > pipeline
> > > > can run and then only merge things to branch-3. We had 2900+ failures
> > > when
> > > > we started 2 months back and now having brought it down to less than
> > 15,
> > > > new failures again has pushed us back in this effort.
> > > >
> > > > I would like to thank everyone who has participated in this effort
> and
> > > > made it possible till this stage. Also, if the contributors can take
> > > > ownership of these new test case failures and fix them, it will be of
> > > great
> > > > help.
> > > >
> > > > Thanks,
> > > > Aman.
> > > > ________________________________
> > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > Sent: Friday, February 17, 2023 6:10 AM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > [You don't often get email from vihangk1@apache.org. Learn why this
> is
> > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > >
> > > > Hi Aman,
> > > >
> > > > I created
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5UlK9DVcNIyVkzyMld3%2F1deJaV1TsLMAY2lDV3kjlrQ%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-27087>
> > > > to look into
> > > > TestMiniSparkOnYarnCliDriver failures. I have a working theory of
> what
> > > > might be going on there. I am still investigating what is the right
> way
> > > to
> > > > fix it though.
> > > >
> > > > Thanks,
> > > > Vihang
> > > >
> > > > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hi Vihang,
> > > > >
> > > > > Yes the tests are failing locally as well with the same issue.
> > > > >
> > > > > Thanks,
> > > > > Aman.
> > > > >
> > > > > Get Outlook for Android<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6qSGgiuKc1UyzmmYo3Tcok%2BSuOiFBdF4lfXv%2FAeuZbs%3D&reserved=0<https://aka.ms/AAb9ysg>
> > > > >
> > > > > ________________________________
> > > > > From: Vihang Karajgaonkar
> <vihang.karajgaonkar@databricks.com.INVALID
> > >
> > > > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > >
> > > > > [You don't often get email from
> > > > vihang.karajgaonkar@databricks.com.invalid.
> > > > > Learn why this is important at
> > > > > https://aka.ms/LearnAboutSenderIdentification ]
> > > > >
> > > > > Thanks a lot Stamatis for starting this thread. I really appreciate
> > all
> > > > the
> > > > > efforts to stabilize branch-3 to get it to a releasable state and I
> > > agree
> > > > > that we should get it to a green state before opening it for PRs
> not
> > > > > related to test failures. I can help with the effort as well.
> > > > >
> > > > > If we want to get the branch back to green state soon, have we
> > > considered
> > > > > disabling the tests which are clearly flaky? (e.g pass on some
> builds
> > > and
> > > > > fail on the other build with no new code changes). If we don't do
> > that,
> > > > we
> > > > > will keep playing whack a mole with those tests. I propose for such
> > > tests
> > > > > we should disable them and create tickets to unflake them
> separately.
> > > > This
> > > > > will help us get back to a green state faster.
> > > > >
> > > > > Hi Aman,
> > > > > For TestMiniSparkOnYarnCliDriver failures, you probably should also
> > > look
> > > > > into the spark driver/application logs and see if there are
> > > > infrastructure
> > > > > errors (e.g OOMs). Are these tests failing when you run locally?
> > > > >
> > > > > Thanks,
> > > > > Vihang
> > > > >
> > > > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > > > +1,
> > > > > > Thanks Stamatis and Lazlo for helping in the test case fixes till
> > > now.
> > > > > >
> > > > > > Team,
> > > > > > I need help in fixing the following tests in Hive. I have tried
> > > > different
> > > > > > approaches but no luck till now.
> > > > > > I am facing some issues in fixing the following tests :
> > > > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > > > >
> > > > > > Issue :
> > > > > > PREHOOK: Input: default@src
> > > > > > PREHOOK: Output: default@src
> > > > > > Failed to monitor Job[-1] with exception
> > > > > > 'java.lang.IllegalStateException(Connection to remote Spark
> driver
> > > was
> > > > > > lost)' Last known state = SENT
> > > > > > Failed to execute spark task, with exception
> > > > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > > > FAILED: Execution Error, return code 1 from
> > > > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel is
> > > closed.
> > > > > >
> > > > > > History :
> > > > > > Initially the tests had failed with errors which I fixed in the
> > > > following
> > > > > > task :
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8rdnoMslR2RT50AL3AANflY51KfU1yajCVTWEpUlyu8%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26940>
> > > > > >
> > > > > > Does anyone know what the issue is here ? There are 6-7 failures
> > > > because
> > > > > > of this test case. Link to the failed test cases for the
> > stacktrace :
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MeAJ3AqShjY4rpr82pYg1JfRSvtHRPKKWJgERVaP0fc%3D&reserved=0<http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3949/2/tests/>
> > > > > > Thanks,
> > > > > > Aman.
> > > > > >
> > > > > > ________________________________
> > > > > > From: László Bodor <bo...@gmail.com>
> > > > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > Subject: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > > >
> > > > > > +1
> > > > > > also, if I merged something that I thought was for test stability
> > > (but
> > > > > > instead it was a feature), excuse me :)
> > > > > > for reference, the whole green test initiative is tracked under
> > this
> > > > > > umbrella:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C043f385c28ce4867174208db28f66afd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148811080483635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BGcj7NF8pSjr7SsYyOSe7o2VrNv2eH9YZ1ZFm4z7c6I%3D&reserved=0<https://issues.apache.org/jira/browse/HIVE-26836>
> > > > > >
> > > > > > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023.
> > > febr.
> > > > > 7.,
> > > > > > K, 12:09):
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > The build in branch-3 is not yet green; there are ~25 test
> > > failures.
> > > > It
> > > > > > is
> > > > > > > a common practice that we shouldn't push changes on top of a
> > broken
> > > > > build
> > > > > > > unless they are addressing test failures.
> > > > > > >
> > > > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo Bodor)
> > are
> > > > > > working
> > > > > > > hard to stabilize the build for quite some time now. If you
> want
> > to
> > > > > help
> > > > > > > out then start by reviewing, merging, and fixing things around
> > test
> > > > > > > failures.
> > > > > > >
> > > > > > > It's not yet the time to bring new features, upgrades, bugs,
> > etc.,
> > > in
> > > > > > > branch-3. I would encourage  committers to not approve such
> > changes
> > > > > till
> > > > > > we
> > > > > > > get back to a stable branch.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stamatis
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
>

Re: [EXTERNAL] Re: Branch-3 backports and build stability

[Aman Raj <ra...@microsoft.com.INVALID>]

Hi Vihang/community,

Found the ticket which broke mm_all.q. This issue comes because of HIVE-20182. Works in my local and on the Jenkins pipeline as well. Link : https://github.com/apache/hive/pull/4127 Reverting this commit for now.

Thanks,
Aman.
________________________________
From: Aman Raj <ra...@microsoft.com.INVALID>
Sent: Monday, March 20, 2023 8:28 AM
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability

Sure Vihang, will look at the other ones. You can pick this up.

Thanks,
Aman.

Get Outlook for Android<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CfKiKqTAEMCgGkQso0X3To6mt12FMuWeHBHTbO%2F6CzY%3D&reserved=0>
________________________________
From: vihang karajgaonkar <vi...@apache.org>
Sent: Monday, March 20, 2023 7:58:48 AM
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability

I think we should revert offending commits first to unblock the branch. We
can create followup tickets to determine if these fixes are blockers for
3.2 release and if yes, we should merge them the right way with a green
test run. Fixing forward always comes with the risk that it introduces new
test failures.

Thanks for all your efforts on this Aman.

I can take a look at testBootstrapReplLoadRetryAfterFailureForPartitions if
you haven’t already started on it.

Thanks,
Vihang

On Sun, Mar 19, 2023 at 10:09 PM Aman Raj <ra...@microsoft.com.invalid>
wrote:

> Hi Vihang/community,
>
> Thanks a lot Vihang for working on the major test failure. This blocked
> more than 35 test cases. Now we are down to the final 4 failures. I have
> analyzed some of them and here they are  (Link :
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-4067%2F12%2Ftests&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u9LTxbyc%2BPjK6gWdIwVnPcDK02TIp%2FsnNgeNu%2BvHBZM%3D&reserved=0)
> :
>
>   1.
> multi_in_clause - This was committed in HIVE-21685 without validating the
> scenario.
> This fails because Hive is not able to parse
> explain cbo
> select * from very_simple_table_for_in_test where name IN('g','r') AND
> name IN('a','b')
> If we want this to work, I am able to do it in my local. We have 2 options
> :
> a. Either revert HIVE-21685 since this scenario was not validated back
> then before adding this test.
> b. This fix was present in
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qOOWnfO8K8jXEegm7uZ6tBlhAh2QtmcwTB%2Bu5r5QDSY%3D&reserved=0 but to cherry pick this
> we need to cherry pick https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2b62n0%2F0OlIskXA37qoHzRRyBcEaR%2BAhrjN9J1yAfxs%3D&reserved=0
> since HIVE-20718<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qOOWnfO8K8jXEegm7uZ6tBlhAh2QtmcwTB%2Bu5r5QDSY%3D&reserved=0> has a
> lot of merge conflicts with  HIVE-17040<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2b62n0%2F0OlIskXA37qoHzRRyBcEaR%2BAhrjN9J1yAfxs%3D&reserved=0>. But after cherry
> picking these we have other failures to fix.
>   2.
> current_date_timestamp.q - This breaking change was committed in
> HIVE-21388 without validation.
> The failure is because again Hive is not able to parse
> explain cbo select current_timestamp() from alltypesorc
> The solution or revert option is same as point 1.
>   3.
> testBootstrapReplLoadRetryAfterFailureForPartitions() - This I have not
> investigated till now.
>   4.
> mm_all.q - This I have not investigated till now.
>
> Thanks,
> Aman.
> ________________________________
> From: vihang karajgaonkar <vi...@apache.org>
> Sent: Friday, March 17, 2023 8:42 PM
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
>
> Just wanted to close the loop on the TestMiniSparkOnYarnCliDriver test
> failures. We will be able to re-enable most of them back on branch-3. The
> ones which were disabled are being tracked separately in a different ticket
> <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=smPLxPu3onfaGcaD%2Fh7ItNlE%2FvdkVN4bTdqyOUuS3MU%3D&reserved=0>
> but they don't look like
> a blocker.
>
> Hi Aman,
>
> Do you know how close are we to reopening branch-3?
>
> Thanks,
> Vihang
>
> On Sat, Mar 4, 2023 at 7:23 PM Aman Raj <ra...@microsoft.com.invalid>
> wrote:
>
> > Or you can cd into itests and run the command you are using. Just another
> > way I run.
> >
> > Thanks,
> > Aman.
> > Get Outlook for Android<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CfKiKqTAEMCgGkQso0X3To6mt12FMuWeHBHTbO%2F6CzY%3D&reserved=0
> >
> > ________________________________
> > From: Aman Raj <ra...@microsoft.com>
> > Sent: Saturday, March 4, 2023 7:20:36 PM
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > Hi Vihang,
> >
> > Thanks a lot for working on this. Can you try using -Pqsplits,itests.
> > Also, I usually give a -o option after doing a clean install.
> >
> > Thanks,
> > Aman.
> >
> > Get Outlook for Android<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CfKiKqTAEMCgGkQso0X3To6mt12FMuWeHBHTbO%2F6CzY%3D&reserved=0
> >
> >
> > ________________________________
> > From: vihang karajgaonkar <vi...@apache.org>
> > Sent: Saturday, 4 March, 2023, 11:35
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > [You don't often get email from vihangk1@apache.org. Learn why this is
> > important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > Just to update on the HoS test failures for
> TestMiniSparkOnYarnCliDriver, I
> > think I was finally able to resolve them (at least on local). I had to
> > revert HIVE-21044 because it was causing OOM for those tests. Also, in
> > order for these tests to work we will have to downgrade netty from
> > 4.1.69.Final to 4.1.51.Final. I understand that we had upgraded netty
> from
> > 4.1.17.Final to 4.1.69.Final for CVEs but the highest netty version that
> we
> > can support without breaking HoS is 4.1.51.Final. Note that 4.1.51.Final
> > includes many of the CVEs which affected 4.1.17.Final so we are still in
> a
> > better place than branch-3.1. Unfortunately, there is no good way to make
> > HoS work with a higher netty version so I think we should downgrade the
> > netty version to 4.1.51.Final for now and look at more options to upgrade
> > it 4.1.69.Final in a separate ticket.
> >
> > I still need to understand why the tests which are working for me locally
> > don't work on the PR job. I tried running the split test classes using
> the
> > following command. Is that the right way to simulate builds from the PR
> > job? Let me know if anyone has more ideas.
> >
> > mvn test
> > -Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
> > -Pqsplits
> >
> > Thanks,
> > Vihang
> >
> >
> > On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <za...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > Thanks Aman for bringing this up and also for cleaning up after others
> (I
> > > saw that you raised tickets and PRs for addressing the failures).
> > >
> > > Many thanks to Vihang as well for helping out. Regarding flaky tests,
> yes
> > > we should disable them as soon as we see them.
> > > There have been some other discussions on how to approach flaky tests
> the
> > > more recent I could find is here [1].
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1]
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zyfnyRAox8dRFuw%2Bu%2BXbMyi4hiIST4Kfgtys8H4AkXE%3D&reserved=0
> > >
> > > On Fri, Feb 17, 2023 at 4:37 AM Aman Raj <rajaman@microsoft.com.invalid
> >
> > > wrote:
> > >
> > > > Hi team,
> > > >
> > > > Thanks Vihang for looking into this. I have commented on the JIRA you
> > > > created.
> > > >
> > > > Just to bring everyone's notice, I have seen that there has been a
> > couple
> > > > of pushes to branch-3, which has lead to 5 more new test failures.
> The
> > > test
> > > > failures are in orc_merge1, orc_merge2, orc_merge3, orc_merge4 and
> > > > orc_merge10. These tests did not use to fail before. I would
> sincerely
> > > urge
> > > > the community to raise a PR against branch-3, so that the Jenkins
> > > pipeline
> > > > can run and then only merge things to branch-3. We had 2900+ failures
> > > when
> > > > we started 2 months back and now having brought it down to less than
> > 15,
> > > > new failures again has pushed us back in this effort.
> > > >
> > > > I would like to thank everyone who has participated in this effort
> and
> > > > made it possible till this stage. Also, if the contributors can take
> > > > ownership of these new test case failures and fix them, it will be of
> > > great
> > > > help.
> > > >
> > > > Thanks,
> > > > Aman.
> > > > ________________________________
> > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > Sent: Friday, February 17, 2023 6:10 AM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > [You don't often get email from vihangk1@apache.org. Learn why this
> is
> > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > >
> > > > Hi Aman,
> > > >
> > > > I created
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QyzvN3%2Ff74CNqP5hc53IVIQL9KNKZCw%2FWSHe0ldL%2F0E%3D&reserved=0
> > > > to look into
> > > > TestMiniSparkOnYarnCliDriver failures. I have a working theory of
> what
> > > > might be going on there. I am still investigating what is the right
> way
> > > to
> > > > fix it though.
> > > >
> > > > Thanks,
> > > > Vihang
> > > >
> > > > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hi Vihang,
> > > > >
> > > > > Yes the tests are failing locally as well with the same issue.
> > > > >
> > > > > Thanks,
> > > > > Aman.
> > > > >
> > > > > Get Outlook for Android<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CfKiKqTAEMCgGkQso0X3To6mt12FMuWeHBHTbO%2F6CzY%3D&reserved=0
> > > > >
> > > > > ________________________________
> > > > > From: Vihang Karajgaonkar
> <vihang.karajgaonkar@databricks.com.INVALID
> > >
> > > > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > >
> > > > > [You don't often get email from
> > > > vihang.karajgaonkar@databricks.com.invalid.
> > > > > Learn why this is important at
> > > > > https://aka.ms/LearnAboutSenderIdentification ]
> > > > >
> > > > > Thanks a lot Stamatis for starting this thread. I really appreciate
> > all
> > > > the
> > > > > efforts to stabilize branch-3 to get it to a releasable state and I
> > > agree
> > > > > that we should get it to a green state before opening it for PRs
> not
> > > > > related to test failures. I can help with the effort as well.
> > > > >
> > > > > If we want to get the branch back to green state soon, have we
> > > considered
> > > > > disabling the tests which are clearly flaky? (e.g pass on some
> builds
> > > and
> > > > > fail on the other build with no new code changes). If we don't do
> > that,
> > > > we
> > > > > will keep playing whack a mole with those tests. I propose for such
> > > tests
> > > > > we should disable them and create tickets to unflake them
> separately.
> > > > This
> > > > > will help us get back to a green state faster.
> > > > >
> > > > > Hi Aman,
> > > > > For TestMiniSparkOnYarnCliDriver failures, you probably should also
> > > look
> > > > > into the spark driver/application logs and see if there are
> > > > infrastructure
> > > > > errors (e.g OOMs). Are these tests failing when you run locally?
> > > > >
> > > > > Thanks,
> > > > > Vihang
> > > > >
> > > > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > > > +1,
> > > > > > Thanks Stamatis and Lazlo for helping in the test case fixes till
> > > now.
> > > > > >
> > > > > > Team,
> > > > > > I need help in fixing the following tests in Hive. I have tried
> > > > different
> > > > > > approaches but no luck till now.
> > > > > > I am facing some issues in fixing the following tests :
> > > > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > > > >
> > > > > > Issue :
> > > > > > PREHOOK: Input: default@src
> > > > > > PREHOOK: Output: default@src
> > > > > > Failed to monitor Job[-1] with exception
> > > > > > 'java.lang.IllegalStateException(Connection to remote Spark
> driver
> > > was
> > > > > > lost)' Last known state = SENT
> > > > > > Failed to execute spark task, with exception
> > > > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > > > FAILED: Execution Error, return code 1 from
> > > > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel is
> > > closed.
> > > > > >
> > > > > > History :
> > > > > > Initially the tests had failed with errors which I fixed in the
> > > > following
> > > > > > task :
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=b6URVGmlnYg0UmzDBFSldQAUY%2B00IZiCnwbbSfd%2FgTg%3D&reserved=0
> > > > > >
> > > > > > Does anyone know what the issue is here ? There are 6-7 failures
> > > > because
> > > > > > of this test case. Link to the failed test cases for the
> > stacktrace :
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=c3Fy8MQ31SBS2sN%2Bw7X2Jbb5Ee9aFLRpD2HIzVmKepw%3D&reserved=0
> > > > > > Thanks,
> > > > > > Aman.
> > > > > >
> > > > > > ________________________________
> > > > > > From: László Bodor <bo...@gmail.com>
> > > > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > Subject: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > > >
> > > > > > +1
> > > > > > also, if I merged something that I thought was for test stability
> > > (but
> > > > > > instead it was a feature), excuse me :)
> > > > > > for reference, the whole green test initiative is tracked under
> > this
> > > > > > umbrella:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7Cd6ea3e0148854cd10d8208db28eefb74%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148779155151838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bZ%2FZxTowuBt5pTawKkwSknIw6Pv2V%2FpfJsMZkJIf%2F4E%3D&reserved=0
> > > > > >
> > > > > > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023.
> > > febr.
> > > > > 7.,
> > > > > > K, 12:09):
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > The build in branch-3 is not yet green; there are ~25 test
> > > failures.
> > > > It
> > > > > > is
> > > > > > > a common practice that we shouldn't push changes on top of a
> > broken
> > > > > build
> > > > > > > unless they are addressing test failures.
> > > > > > >
> > > > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo Bodor)
> > are
> > > > > > working
> > > > > > > hard to stabilize the build for quite some time now. If you
> want
> > to
> > > > > help
> > > > > > > out then start by reviewing, merging, and fixing things around
> > test
> > > > > > > failures.
> > > > > > >
> > > > > > > It's not yet the time to bring new features, upgrades, bugs,
> > etc.,
> > > in
> > > > > > > branch-3. I would encourage  committers to not approve such
> > changes
> > > > > till
> > > > > > we
> > > > > > > get back to a stable branch.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stamatis
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
>

Re: [EXTERNAL] Re: Branch-3 backports and build stability

[Aman Raj <ra...@microsoft.com.INVALID>]

Sure Vihang, will look at the other ones. You can pick this up.

Thanks,
Aman.

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: vihang karajgaonkar <vi...@apache.org>
Sent: Monday, March 20, 2023 7:58:48 AM
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability

I think we should revert offending commits first to unblock the branch. We
can create followup tickets to determine if these fixes are blockers for
3.2 release and if yes, we should merge them the right way with a green
test run. Fixing forward always comes with the risk that it introduces new
test failures.

Thanks for all your efforts on this Aman.

I can take a look at testBootstrapReplLoadRetryAfterFailureForPartitions if
you haven’t already started on it.

Thanks,
Vihang

On Sun, Mar 19, 2023 at 10:09 PM Aman Raj <ra...@microsoft.com.invalid>
wrote:

> Hi Vihang/community,
>
> Thanks a lot Vihang for working on the major test failure. This blocked
> more than 35 test cases. Now we are down to the final 4 failures. I have
> analyzed some of them and here they are  (Link :
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-4067%2F12%2Ftests&data=05%7C01%7Crajaman%40microsoft.com%7C3c77d352209146ba91ec08db28eae05e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148761521049046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ju9ucPcRMofa7DERJyURmawbC5J3oIiiGOKpqFdXPG8%3D&reserved=0)
> :
>
>   1.
> multi_in_clause - This was committed in HIVE-21685 without validating the
> scenario.
> This fails because Hive is not able to parse
> explain cbo
> select * from very_simple_table_for_in_test where name IN('g','r') AND
> name IN('a','b')
> If we want this to work, I am able to do it in my local. We have 2 options
> :
> a. Either revert HIVE-21685 since this scenario was not validated back
> then before adding this test.
> b. This fix was present in
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C3c77d352209146ba91ec08db28eae05e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148761521049046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=B%2FEBhlFoOCCxepgG4dfYuTZhExHcIHBU19%2BvVYiOFhY%3D&reserved=0 but to cherry pick this
> we need to cherry pick https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C3c77d352209146ba91ec08db28eae05e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148761521049046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NrdNa%2FV%2BHoFU757IV380iIoAnAQpBdAmnOhc9Iy41gE%3D&reserved=0
> since HIVE-20718<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-20718&data=05%7C01%7Crajaman%40microsoft.com%7C3c77d352209146ba91ec08db28eae05e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148761521049046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=B%2FEBhlFoOCCxepgG4dfYuTZhExHcIHBU19%2BvVYiOFhY%3D&reserved=0> has a
> lot of merge conflicts with  HIVE-17040<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-17040&data=05%7C01%7Crajaman%40microsoft.com%7C3c77d352209146ba91ec08db28eae05e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148761521049046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NrdNa%2FV%2BHoFU757IV380iIoAnAQpBdAmnOhc9Iy41gE%3D&reserved=0>. But after cherry
> picking these we have other failures to fix.
>   2.
> current_date_timestamp.q - This breaking change was committed in
> HIVE-21388 without validation.
> The failure is because again Hive is not able to parse
> explain cbo select current_timestamp() from alltypesorc
> The solution or revert option is same as point 1.
>   3.
> testBootstrapReplLoadRetryAfterFailureForPartitions() - This I have not
> investigated till now.
>   4.
> mm_all.q - This I have not investigated till now.
>
> Thanks,
> Aman.
> ________________________________
> From: vihang karajgaonkar <vi...@apache.org>
> Sent: Friday, March 17, 2023 8:42 PM
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
>
> Just wanted to close the loop on the TestMiniSparkOnYarnCliDriver test
> failures. We will be able to re-enable most of them back on branch-3. The
> ones which were disabled are being tracked separately in a different ticket
> <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7C3c77d352209146ba91ec08db28eae05e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148761521049046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7X5VpRr%2BlHe%2FwrR19syyuFn3wtHqloC99kStdgOrelU%3D&reserved=0>
> but they don't look like
> a blocker.
>
> Hi Aman,
>
> Do you know how close are we to reopening branch-3?
>
> Thanks,
> Vihang
>
> On Sat, Mar 4, 2023 at 7:23 PM Aman Raj <ra...@microsoft.com.invalid>
> wrote:
>
> > Or you can cd into itests and run the command you are using. Just another
> > way I run.
> >
> > Thanks,
> > Aman.
> > Get Outlook for Android<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C3c77d352209146ba91ec08db28eae05e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148761521049046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2Bpf9R2HM8NYhbTiv4n4K%2B475BJglu2IAg5P8w0cxdcE%3D&reserved=0
> >
> > ________________________________
> > From: Aman Raj <ra...@microsoft.com>
> > Sent: Saturday, March 4, 2023 7:20:36 PM
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > Hi Vihang,
> >
> > Thanks a lot for working on this. Can you try using -Pqsplits,itests.
> > Also, I usually give a -o option after doing a clean install.
> >
> > Thanks,
> > Aman.
> >
> > Get Outlook for Android<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C3c77d352209146ba91ec08db28eae05e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148761521049046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2Bpf9R2HM8NYhbTiv4n4K%2B475BJglu2IAg5P8w0cxdcE%3D&reserved=0
> >
> >
> > ________________________________
> > From: vihang karajgaonkar <vi...@apache.org>
> > Sent: Saturday, 4 March, 2023, 11:35
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > [You don't often get email from vihangk1@apache.org. Learn why this is
> > important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > Just to update on the HoS test failures for
> TestMiniSparkOnYarnCliDriver, I
> > think I was finally able to resolve them (at least on local). I had to
> > revert HIVE-21044 because it was causing OOM for those tests. Also, in
> > order for these tests to work we will have to downgrade netty from
> > 4.1.69.Final to 4.1.51.Final. I understand that we had upgraded netty
> from
> > 4.1.17.Final to 4.1.69.Final for CVEs but the highest netty version that
> we
> > can support without breaking HoS is 4.1.51.Final. Note that 4.1.51.Final
> > includes many of the CVEs which affected 4.1.17.Final so we are still in
> a
> > better place than branch-3.1. Unfortunately, there is no good way to make
> > HoS work with a higher netty version so I think we should downgrade the
> > netty version to 4.1.51.Final for now and look at more options to upgrade
> > it 4.1.69.Final in a separate ticket.
> >
> > I still need to understand why the tests which are working for me locally
> > don't work on the PR job. I tried running the split test classes using
> the
> > following command. Is that the right way to simulate builds from the PR
> > job? Let me know if anyone has more ideas.
> >
> > mvn test
> > -Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
> > -Pqsplits
> >
> > Thanks,
> > Vihang
> >
> >
> > On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <za...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > Thanks Aman for bringing this up and also for cleaning up after others
> (I
> > > saw that you raised tickets and PRs for addressing the failures).
> > >
> > > Many thanks to Vihang as well for helping out. Regarding flaky tests,
> yes
> > > we should disable them as soon as we see them.
> > > There have been some other discussions on how to approach flaky tests
> the
> > > more recent I could find is here [1].
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1]
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C3c77d352209146ba91ec08db28eae05e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148761521049046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=axMTbO1ru%2B4cW0Buw9Zq2JNzG%2FqxVrgVMbao7Ej1u4A%3D&reserved=0
> > >
> > > On Fri, Feb 17, 2023 at 4:37 AM Aman Raj <rajaman@microsoft.com.invalid
> >
> > > wrote:
> > >
> > > > Hi team,
> > > >
> > > > Thanks Vihang for looking into this. I have commented on the JIRA you
> > > > created.
> > > >
> > > > Just to bring everyone's notice, I have seen that there has been a
> > couple
> > > > of pushes to branch-3, which has lead to 5 more new test failures.
> The
> > > test
> > > > failures are in orc_merge1, orc_merge2, orc_merge3, orc_merge4 and
> > > > orc_merge10. These tests did not use to fail before. I would
> sincerely
> > > urge
> > > > the community to raise a PR against branch-3, so that the Jenkins
> > > pipeline
> > > > can run and then only merge things to branch-3. We had 2900+ failures
> > > when
> > > > we started 2 months back and now having brought it down to less than
> > 15,
> > > > new failures again has pushed us back in this effort.
> > > >
> > > > I would like to thank everyone who has participated in this effort
> and
> > > > made it possible till this stage. Also, if the contributors can take
> > > > ownership of these new test case failures and fix them, it will be of
> > > great
> > > > help.
> > > >
> > > > Thanks,
> > > > Aman.
> > > > ________________________________
> > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > Sent: Friday, February 17, 2023 6:10 AM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > [You don't often get email from vihangk1@apache.org. Learn why this
> is
> > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > >
> > > > Hi Aman,
> > > >
> > > > I created
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C3c77d352209146ba91ec08db28eae05e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148761521049046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fxfHjGkxeC9kRPCRTtdLNK7mJMSX6g7xCfBN2Iu3bGA%3D&reserved=0
> > > > to look into
> > > > TestMiniSparkOnYarnCliDriver failures. I have a working theory of
> what
> > > > might be going on there. I am still investigating what is the right
> way
> > > to
> > > > fix it though.
> > > >
> > > > Thanks,
> > > > Vihang
> > > >
> > > > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hi Vihang,
> > > > >
> > > > > Yes the tests are failing locally as well with the same issue.
> > > > >
> > > > > Thanks,
> > > > > Aman.
> > > > >
> > > > > Get Outlook for Android<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C3c77d352209146ba91ec08db28eae05e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148761521049046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2Bpf9R2HM8NYhbTiv4n4K%2B475BJglu2IAg5P8w0cxdcE%3D&reserved=0
> > > > >
> > > > > ________________________________
> > > > > From: Vihang Karajgaonkar
> <vihang.karajgaonkar@databricks.com.INVALID
> > >
> > > > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > >
> > > > > [You don't often get email from
> > > > vihang.karajgaonkar@databricks.com.invalid.
> > > > > Learn why this is important at
> > > > > https://aka.ms/LearnAboutSenderIdentification ]
> > > > >
> > > > > Thanks a lot Stamatis for starting this thread. I really appreciate
> > all
> > > > the
> > > > > efforts to stabilize branch-3 to get it to a releasable state and I
> > > agree
> > > > > that we should get it to a green state before opening it for PRs
> not
> > > > > related to test failures. I can help with the effort as well.
> > > > >
> > > > > If we want to get the branch back to green state soon, have we
> > > considered
> > > > > disabling the tests which are clearly flaky? (e.g pass on some
> builds
> > > and
> > > > > fail on the other build with no new code changes). If we don't do
> > that,
> > > > we
> > > > > will keep playing whack a mole with those tests. I propose for such
> > > tests
> > > > > we should disable them and create tickets to unflake them
> separately.
> > > > This
> > > > > will help us get back to a green state faster.
> > > > >
> > > > > Hi Aman,
> > > > > For TestMiniSparkOnYarnCliDriver failures, you probably should also
> > > look
> > > > > into the spark driver/application logs and see if there are
> > > > infrastructure
> > > > > errors (e.g OOMs). Are these tests failing when you run locally?
> > > > >
> > > > > Thanks,
> > > > > Vihang
> > > > >
> > > > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > > > +1,
> > > > > > Thanks Stamatis and Lazlo for helping in the test case fixes till
> > > now.
> > > > > >
> > > > > > Team,
> > > > > > I need help in fixing the following tests in Hive. I have tried
> > > > different
> > > > > > approaches but no luck till now.
> > > > > > I am facing some issues in fixing the following tests :
> > > > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > > > >
> > > > > > Issue :
> > > > > > PREHOOK: Input: default@src
> > > > > > PREHOOK: Output: default@src
> > > > > > Failed to monitor Job[-1] with exception
> > > > > > 'java.lang.IllegalStateException(Connection to remote Spark
> driver
> > > was
> > > > > > lost)' Last known state = SENT
> > > > > > Failed to execute spark task, with exception
> > > > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > > > FAILED: Execution Error, return code 1 from
> > > > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel is
> > > closed.
> > > > > >
> > > > > > History :
> > > > > > Initially the tests had failed with errors which I fixed in the
> > > > following
> > > > > > task :
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C3c77d352209146ba91ec08db28eae05e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148761521049046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BjUB874gg7OVYBqF3NoUCWGY8LjCzg0tzteuEu9t1Cw%3D&reserved=0
> > > > > >
> > > > > > Does anyone know what the issue is here ? There are 6-7 failures
> > > > because
> > > > > > of this test case. Link to the failed test cases for the
> > stacktrace :
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C3c77d352209146ba91ec08db28eae05e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148761521049046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9W%2FVM8oE1Uz%2FSVOKOsCT10pkZt2fRdbJnDnZBRr2LBs%3D&reserved=0
> > > > > > Thanks,
> > > > > > Aman.
> > > > > >
> > > > > > ________________________________
> > > > > > From: László Bodor <bo...@gmail.com>
> > > > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > Subject: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > > >
> > > > > > +1
> > > > > > also, if I merged something that I thought was for test stability
> > > (but
> > > > > > instead it was a feature), excuse me :)
> > > > > > for reference, the whole green test initiative is tracked under
> > this
> > > > > > umbrella:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C3c77d352209146ba91ec08db28eae05e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638148761521049046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UjJtD9Z8piukwlZB6PC8unoRvSbQlDPx2X6e5JcPZh4%3D&reserved=0
> > > > > >
> > > > > > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023.
> > > febr.
> > > > > 7.,
> > > > > > K, 12:09):
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > The build in branch-3 is not yet green; there are ~25 test
> > > failures.
> > > > It
> > > > > > is
> > > > > > > a common practice that we shouldn't push changes on top of a
> > broken
> > > > > build
> > > > > > > unless they are addressing test failures.
> > > > > > >
> > > > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo Bodor)
> > are
> > > > > > working
> > > > > > > hard to stabilize the build for quite some time now. If you
> want
> > to
> > > > > help
> > > > > > > out then start by reviewing, merging, and fixing things around
> > test
> > > > > > > failures.
> > > > > > >
> > > > > > > It's not yet the time to bring new features, upgrades, bugs,
> > etc.,
> > > in
> > > > > > > branch-3. I would encourage  committers to not approve such
> > changes
> > > > > till
> > > > > > we
> > > > > > > get back to a stable branch.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stamatis
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
>

Re: [EXTERNAL] Re: Branch-3 backports and build stability

[vihang karajgaonkar <vi...@apache.org>]

I think we should revert offending commits first to unblock the branch. We
can create followup tickets to determine if these fixes are blockers for
3.2 release and if yes, we should merge them the right way with a green
test run. Fixing forward always comes with the risk that it introduces new
test failures.

Thanks for all your efforts on this Aman.

I can take a look at testBootstrapReplLoadRetryAfterFailureForPartitions if
you haven’t already started on it.

Thanks,
Vihang

On Sun, Mar 19, 2023 at 10:09 PM Aman Raj <ra...@microsoft.com.invalid>
wrote:

> Hi Vihang/community,
>
> Thanks a lot Vihang for working on the major test failure. This blocked
> more than 35 test cases. Now we are down to the final 4 failures. I have
> analyzed some of them and here they are  (Link :
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests)
> :
>
>   1.
> multi_in_clause - This was committed in HIVE-21685 without validating the
> scenario.
> This fails because Hive is not able to parse
> explain cbo
> select * from very_simple_table_for_in_test where name IN('g','r') AND
> name IN('a','b')
> If we want this to work, I am able to do it in my local. We have 2 options
> :
> a. Either revert HIVE-21685 since this scenario was not validated back
> then before adding this test.
> b. This fix was present in
> https://issues.apache.org/jira/browse/HIVE-20718 but to cherry pick this
> we need to cherry pick https://issues.apache.org/jira/browse/HIVE-17040
> since HIVE-20718<https://issues.apache.org/jira/browse/HIVE-20718> has a
> lot of merge conflicts with  HIVE-17040<
> https://issues.apache.org/jira/browse/HIVE-17040>. But after cherry
> picking these we have other failures to fix.
>   2.
> current_date_timestamp.q - This breaking change was committed in
> HIVE-21388 without validation.
> The failure is because again Hive is not able to parse
> explain cbo select current_timestamp() from alltypesorc
> The solution or revert option is same as point 1.
>   3.
> testBootstrapReplLoadRetryAfterFailureForPartitions() - This I have not
> investigated till now.
>   4.
> mm_all.q - This I have not investigated till now.
>
> Thanks,
> Aman.
> ________________________________
> From: vihang karajgaonkar <vi...@apache.org>
> Sent: Friday, March 17, 2023 8:42 PM
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
>
> Just wanted to close the loop on the TestMiniSparkOnYarnCliDriver test
> failures. We will be able to re-enable most of them back on branch-3. The
> ones which were disabled are being tracked separately in a different ticket
> <
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=C19is4AtBNH04Dm1F1bwp4wVw6erFn736e47p6STrzE%3D&reserved=0>
> but they don't look like
> a blocker.
>
> Hi Aman,
>
> Do you know how close are we to reopening branch-3?
>
> Thanks,
> Vihang
>
> On Sat, Mar 4, 2023 at 7:23 PM Aman Raj <ra...@microsoft.com.invalid>
> wrote:
>
> > Or you can cd into itests and run the command you are using. Just another
> > way I run.
> >
> > Thanks,
> > Aman.
> > Get Outlook for Android<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nAL14KzxAWwQAV5WJmfkBgaJh0M0wPwq5qORrXcQ6fk%3D&reserved=0
> >
> > ________________________________
> > From: Aman Raj <ra...@microsoft.com>
> > Sent: Saturday, March 4, 2023 7:20:36 PM
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > Hi Vihang,
> >
> > Thanks a lot for working on this. Can you try using -Pqsplits,itests.
> > Also, I usually give a -o option after doing a clean install.
> >
> > Thanks,
> > Aman.
> >
> > Get Outlook for Android<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nAL14KzxAWwQAV5WJmfkBgaJh0M0wPwq5qORrXcQ6fk%3D&reserved=0
> >
> >
> > ________________________________
> > From: vihang karajgaonkar <vi...@apache.org>
> > Sent: Saturday, 4 March, 2023, 11:35
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > [You don't often get email from vihangk1@apache.org. Learn why this is
> > important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > Just to update on the HoS test failures for
> TestMiniSparkOnYarnCliDriver, I
> > think I was finally able to resolve them (at least on local). I had to
> > revert HIVE-21044 because it was causing OOM for those tests. Also, in
> > order for these tests to work we will have to downgrade netty from
> > 4.1.69.Final to 4.1.51.Final. I understand that we had upgraded netty
> from
> > 4.1.17.Final to 4.1.69.Final for CVEs but the highest netty version that
> we
> > can support without breaking HoS is 4.1.51.Final. Note that 4.1.51.Final
> > includes many of the CVEs which affected 4.1.17.Final so we are still in
> a
> > better place than branch-3.1. Unfortunately, there is no good way to make
> > HoS work with a higher netty version so I think we should downgrade the
> > netty version to 4.1.51.Final for now and look at more options to upgrade
> > it 4.1.69.Final in a separate ticket.
> >
> > I still need to understand why the tests which are working for me locally
> > don't work on the PR job. I tried running the split test classes using
> the
> > following command. Is that the right way to simulate builds from the PR
> > job? Let me know if anyone has more ideas.
> >
> > mvn test
> > -Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
> > -Pqsplits
> >
> > Thanks,
> > Vihang
> >
> >
> > On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <za...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > Thanks Aman for bringing this up and also for cleaning up after others
> (I
> > > saw that you raised tickets and PRs for addressing the failures).
> > >
> > > Many thanks to Vihang as well for helping out. Regarding flaky tests,
> yes
> > > we should disable them as soon as we see them.
> > > There have been some other discussions on how to approach flaky tests
> the
> > > more recent I could find is here [1].
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1]
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mIYO5QQf%2Fnt7A%2FfB9v5WxYVWKzzrlu75GYWVcRu%2BJMU%3D&reserved=0
> > >
> > > On Fri, Feb 17, 2023 at 4:37 AM Aman Raj <rajaman@microsoft.com.invalid
> >
> > > wrote:
> > >
> > > > Hi team,
> > > >
> > > > Thanks Vihang for looking into this. I have commented on the JIRA you
> > > > created.
> > > >
> > > > Just to bring everyone's notice, I have seen that there has been a
> > couple
> > > > of pushes to branch-3, which has lead to 5 more new test failures.
> The
> > > test
> > > > failures are in orc_merge1, orc_merge2, orc_merge3, orc_merge4 and
> > > > orc_merge10. These tests did not use to fail before. I would
> sincerely
> > > urge
> > > > the community to raise a PR against branch-3, so that the Jenkins
> > > pipeline
> > > > can run and then only merge things to branch-3. We had 2900+ failures
> > > when
> > > > we started 2 months back and now having brought it down to less than
> > 15,
> > > > new failures again has pushed us back in this effort.
> > > >
> > > > I would like to thank everyone who has participated in this effort
> and
> > > > made it possible till this stage. Also, if the contributors can take
> > > > ownership of these new test case failures and fix them, it will be of
> > > great
> > > > help.
> > > >
> > > > Thanks,
> > > > Aman.
> > > > ________________________________
> > > > From: vihang karajgaonkar <vi...@apache.org>
> > > > Sent: Friday, February 17, 2023 6:10 AM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > [You don't often get email from vihangk1@apache.org. Learn why this
> is
> > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > >
> > > > Hi Aman,
> > > >
> > > > I created
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ty%2BvnDmVyTiOXtgoH1YdVYfROcX9pKsx%2FhF6C6pPPaA%3D&reserved=0
> > > > to look into
> > > > TestMiniSparkOnYarnCliDriver failures. I have a working theory of
> what
> > > > might be going on there. I am still investigating what is the right
> way
> > > to
> > > > fix it though.
> > > >
> > > > Thanks,
> > > > Vihang
> > > >
> > > > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hi Vihang,
> > > > >
> > > > > Yes the tests are failing locally as well with the same issue.
> > > > >
> > > > > Thanks,
> > > > > Aman.
> > > > >
> > > > > Get Outlook for Android<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nAL14KzxAWwQAV5WJmfkBgaJh0M0wPwq5qORrXcQ6fk%3D&reserved=0
> > > > >
> > > > > ________________________________
> > > > > From: Vihang Karajgaonkar
> <vihang.karajgaonkar@databricks.com.INVALID
> > >
> > > > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > >
> > > > > [You don't often get email from
> > > > vihang.karajgaonkar@databricks.com.invalid.
> > > > > Learn why this is important at
> > > > > https://aka.ms/LearnAboutSenderIdentification ]
> > > > >
> > > > > Thanks a lot Stamatis for starting this thread. I really appreciate
> > all
> > > > the
> > > > > efforts to stabilize branch-3 to get it to a releasable state and I
> > > agree
> > > > > that we should get it to a green state before opening it for PRs
> not
> > > > > related to test failures. I can help with the effort as well.
> > > > >
> > > > > If we want to get the branch back to green state soon, have we
> > > considered
> > > > > disabling the tests which are clearly flaky? (e.g pass on some
> builds
> > > and
> > > > > fail on the other build with no new code changes). If we don't do
> > that,
> > > > we
> > > > > will keep playing whack a mole with those tests. I propose for such
> > > tests
> > > > > we should disable them and create tickets to unflake them
> separately.
> > > > This
> > > > > will help us get back to a green state faster.
> > > > >
> > > > > Hi Aman,
> > > > > For TestMiniSparkOnYarnCliDriver failures, you probably should also
> > > look
> > > > > into the spark driver/application logs and see if there are
> > > > infrastructure
> > > > > errors (e.g OOMs). Are these tests failing when you run locally?
> > > > >
> > > > > Thanks,
> > > > > Vihang
> > > > >
> > > > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > > > +1,
> > > > > > Thanks Stamatis and Lazlo for helping in the test case fixes till
> > > now.
> > > > > >
> > > > > > Team,
> > > > > > I need help in fixing the following tests in Hive. I have tried
> > > > different
> > > > > > approaches but no luck till now.
> > > > > > I am facing some issues in fixing the following tests :
> > > > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > > > >
> > > > > > Issue :
> > > > > > PREHOOK: Input: default@src
> > > > > > PREHOOK: Output: default@src
> > > > > > Failed to monitor Job[-1] with exception
> > > > > > 'java.lang.IllegalStateException(Connection to remote Spark
> driver
> > > was
> > > > > > lost)' Last known state = SENT
> > > > > > Failed to execute spark task, with exception
> > > > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > > > FAILED: Execution Error, return code 1 from
> > > > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel is
> > > closed.
> > > > > >
> > > > > > History :
> > > > > > Initially the tests had failed with errors which I fixed in the
> > > > following
> > > > > > task :
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XJvVVYRbp2h8M2f%2BeAZdY5T1jwym5h3522kGS7tZWic%3D&reserved=0
> > > > > >
> > > > > > Does anyone know what the issue is here ? There are 6-7 failures
> > > > because
> > > > > > of this test case. Link to the failed test cases for the
> > stacktrace :
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=gVJNSjFUhvUUMiKghSW%2F6OMVRgxtjQxm5BJ2h0pTv2s%3D&reserved=0
> > > > > > Thanks,
> > > > > > Aman.
> > > > > >
> > > > > > ________________________________
> > > > > > From: László Bodor <bo...@gmail.com>
> > > > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > Subject: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > > >
> > > > > > +1
> > > > > > also, if I merged something that I thought was for test stability
> > > (but
> > > > > > instead it was a feature), excuse me :)
> > > > > > for reference, the whole green test initiative is tracked under
> > this
> > > > > > umbrella:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wIr0EHWrRcXh0D0lLvu8g5r0sxpdFkfn2pFu6Ag%2BJ38%3D&reserved=0
> > > > > >
> > > > > > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023.
> > > febr.
> > > > > 7.,
> > > > > > K, 12:09):
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > The build in branch-3 is not yet green; there are ~25 test
> > > failures.
> > > > It
> > > > > > is
> > > > > > > a common practice that we shouldn't push changes on top of a
> > broken
> > > > > build
> > > > > > > unless they are addressing test failures.
> > > > > > >
> > > > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo Bodor)
> > are
> > > > > > working
> > > > > > > hard to stabilize the build for quite some time now. If you
> want
> > to
> > > > > help
> > > > > > > out then start by reviewing, merging, and fixing things around
> > test
> > > > > > > failures.
> > > > > > >
> > > > > > > It's not yet the time to bring new features, upgrades, bugs,
> > etc.,
> > > in
> > > > > > > branch-3. I would encourage  committers to not approve such
> > changes
> > > > > till
> > > > > > we
> > > > > > > get back to a stable branch.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stamatis
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
>

Re: [EXTERNAL] Re: Branch-3 backports and build stability

[Aman Raj <ra...@microsoft.com.INVALID>]

Hi Vihang/community,

Thanks a lot Vihang for working on the major test failure. This blocked more than 35 test cases. Now we are down to the final 4 failures. I have analyzed some of them and here they are  (Link : http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4067/12/tests) :

  1.
multi_in_clause - This was committed in HIVE-21685 without validating the scenario.
This fails because Hive is not able to parse
explain cbo
select * from very_simple_table_for_in_test where name IN('g','r') AND name IN('a','b')
If we want this to work, I am able to do it in my local. We have 2 options :
a. Either revert HIVE-21685 since this scenario was not validated back then before adding this test.
b. This fix was present in https://issues.apache.org/jira/browse/HIVE-20718 but to cherry pick this we need to cherry pick https://issues.apache.org/jira/browse/HIVE-17040 since HIVE-20718<https://issues.apache.org/jira/browse/HIVE-20718> has a lot of merge conflicts with  HIVE-17040<https://issues.apache.org/jira/browse/HIVE-17040>. But after cherry picking these we have other failures to fix.
  2.
current_date_timestamp.q - This breaking change was committed in HIVE-21388 without validation.
The failure is because again Hive is not able to parse
explain cbo select current_timestamp() from alltypesorc
The solution or revert option is same as point 1.
  3.
testBootstrapReplLoadRetryAfterFailureForPartitions() - This I have not investigated till now.
  4.
mm_all.q - This I have not investigated till now.

Thanks,
Aman.
________________________________
From: vihang karajgaonkar <vi...@apache.org>
Sent: Friday, March 17, 2023 8:42 PM
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability

Just wanted to close the loop on the TestMiniSparkOnYarnCliDriver test
failures. We will be able to re-enable most of them back on branch-3. The
ones which were disabled are being tracked separately in a different ticket
<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27146&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=C19is4AtBNH04Dm1F1bwp4wVw6erFn736e47p6STrzE%3D&reserved=0> but they don't look like
a blocker.

Hi Aman,

Do you know how close are we to reopening branch-3?

Thanks,
Vihang

On Sat, Mar 4, 2023 at 7:23 PM Aman Raj <ra...@microsoft.com.invalid>
wrote:

> Or you can cd into itests and run the command you are using. Just another
> way I run.
>
> Thanks,
> Aman.
> Get Outlook for Android<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nAL14KzxAWwQAV5WJmfkBgaJh0M0wPwq5qORrXcQ6fk%3D&reserved=0>
> ________________________________
> From: Aman Raj <ra...@microsoft.com>
> Sent: Saturday, March 4, 2023 7:20:36 PM
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
>
> Hi Vihang,
>
> Thanks a lot for working on this. Can you try using -Pqsplits,itests.
> Also, I usually give a -o option after doing a clean install.
>
> Thanks,
> Aman.
>
> Get Outlook for Android<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nAL14KzxAWwQAV5WJmfkBgaJh0M0wPwq5qORrXcQ6fk%3D&reserved=0>
>
> ________________________________
> From: vihang karajgaonkar <vi...@apache.org>
> Sent: Saturday, 4 March, 2023, 11:35
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
>
> [You don't often get email from vihangk1@apache.org. Learn why this is
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> Just to update on the HoS test failures for TestMiniSparkOnYarnCliDriver, I
> think I was finally able to resolve them (at least on local). I had to
> revert HIVE-21044 because it was causing OOM for those tests. Also, in
> order for these tests to work we will have to downgrade netty from
> 4.1.69.Final to 4.1.51.Final. I understand that we had upgraded netty from
> 4.1.17.Final to 4.1.69.Final for CVEs but the highest netty version that we
> can support without breaking HoS is 4.1.51.Final. Note that 4.1.51.Final
> includes many of the CVEs which affected 4.1.17.Final so we are still in a
> better place than branch-3.1. Unfortunately, there is no good way to make
> HoS work with a higher netty version so I think we should downgrade the
> netty version to 4.1.51.Final for now and look at more options to upgrade
> it 4.1.69.Final in a separate ticket.
>
> I still need to understand why the tests which are working for me locally
> don't work on the PR job. I tried running the split test classes using the
> following command. Is that the right way to simulate builds from the PR
> job? Let me know if anyone has more ideas.
>
> mvn test
> -Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
> -Pqsplits
>
> Thanks,
> Vihang
>
>
> On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <za...@gmail.com>
> wrote:
>
> > Hello,
> >
> > Thanks Aman for bringing this up and also for cleaning up after others (I
> > saw that you raised tickets and PRs for addressing the failures).
> >
> > Many thanks to Vihang as well for helping out. Regarding flaky tests, yes
> > we should disable them as soon as we see them.
> > There have been some other discussions on how to approach flaky tests the
> > more recent I could find is here [1].
> >
> > Best,
> > Stamatis
> >
> > [1]
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mIYO5QQf%2Fnt7A%2FfB9v5WxYVWKzzrlu75GYWVcRu%2BJMU%3D&reserved=0
> >
> > On Fri, Feb 17, 2023 at 4:37 AM Aman Raj <ra...@microsoft.com.invalid>
> > wrote:
> >
> > > Hi team,
> > >
> > > Thanks Vihang for looking into this. I have commented on the JIRA you
> > > created.
> > >
> > > Just to bring everyone's notice, I have seen that there has been a
> couple
> > > of pushes to branch-3, which has lead to 5 more new test failures. The
> > test
> > > failures are in orc_merge1, orc_merge2, orc_merge3, orc_merge4 and
> > > orc_merge10. These tests did not use to fail before. I would sincerely
> > urge
> > > the community to raise a PR against branch-3, so that the Jenkins
> > pipeline
> > > can run and then only merge things to branch-3. We had 2900+ failures
> > when
> > > we started 2 months back and now having brought it down to less than
> 15,
> > > new failures again has pushed us back in this effort.
> > >
> > > I would like to thank everyone who has participated in this effort and
> > > made it possible till this stage. Also, if the contributors can take
> > > ownership of these new test case failures and fix them, it will be of
> > great
> > > help.
> > >
> > > Thanks,
> > > Aman.
> > > ________________________________
> > > From: vihang karajgaonkar <vi...@apache.org>
> > > Sent: Friday, February 17, 2023 6:10 AM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > >
> > > [You don't often get email from vihangk1@apache.org. Learn why this is
> > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > >
> > > Hi Aman,
> > >
> > > I created
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ty%2BvnDmVyTiOXtgoH1YdVYfROcX9pKsx%2FhF6C6pPPaA%3D&reserved=0
> > > to look into
> > > TestMiniSparkOnYarnCliDriver failures. I have a working theory of what
> > > might be going on there. I am still investigating what is the right way
> > to
> > > fix it though.
> > >
> > > Thanks,
> > > Vihang
> > >
> > > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj
> <rajaman@microsoft.com.invalid
> > >
> > > wrote:
> > >
> > > > Hi Vihang,
> > > >
> > > > Yes the tests are failing locally as well with the same issue.
> > > >
> > > > Thanks,
> > > > Aman.
> > > >
> > > > Get Outlook for Android<
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nAL14KzxAWwQAV5WJmfkBgaJh0M0wPwq5qORrXcQ6fk%3D&reserved=0
> > > >
> > > > ________________________________
> > > > From: Vihang Karajgaonkar <vihang.karajgaonkar@databricks.com.INVALID
> >
> > > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > [You don't often get email from
> > > vihang.karajgaonkar@databricks.com.invalid.
> > > > Learn why this is important at
> > > > https://aka.ms/LearnAboutSenderIdentification ]
> > > >
> > > > Thanks a lot Stamatis for starting this thread. I really appreciate
> all
> > > the
> > > > efforts to stabilize branch-3 to get it to a releasable state and I
> > agree
> > > > that we should get it to a green state before opening it for PRs not
> > > > related to test failures. I can help with the effort as well.
> > > >
> > > > If we want to get the branch back to green state soon, have we
> > considered
> > > > disabling the tests which are clearly flaky? (e.g pass on some builds
> > and
> > > > fail on the other build with no new code changes). If we don't do
> that,
> > > we
> > > > will keep playing whack a mole with those tests. I propose for such
> > tests
> > > > we should disable them and create tickets to unflake them separately.
> > > This
> > > > will help us get back to a green state faster.
> > > >
> > > > Hi Aman,
> > > > For TestMiniSparkOnYarnCliDriver failures, you probably should also
> > look
> > > > into the spark driver/application logs and see if there are
> > > infrastructure
> > > > errors (e.g OOMs). Are these tests failing when you run locally?
> > > >
> > > > Thanks,
> > > > Vihang
> > > >
> > > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj
> <rajaman@microsoft.com.invalid
> > >
> > > > wrote:
> > > >
> > > > > +1,
> > > > > Thanks Stamatis and Lazlo for helping in the test case fixes till
> > now.
> > > > >
> > > > > Team,
> > > > > I need help in fixing the following tests in Hive. I have tried
> > > different
> > > > > approaches but no luck till now.
> > > > > I am facing some issues in fixing the following tests :
> > > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > > >
> > > > > Issue :
> > > > > PREHOOK: Input: default@src
> > > > > PREHOOK: Output: default@src
> > > > > Failed to monitor Job[-1] with exception
> > > > > 'java.lang.IllegalStateException(Connection to remote Spark driver
> > was
> > > > > lost)' Last known state = SENT
> > > > > Failed to execute spark task, with exception
> > > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > > FAILED: Execution Error, return code 1 from
> > > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel is
> > closed.
> > > > >
> > > > > History :
> > > > > Initially the tests had failed with errors which I fixed in the
> > > following
> > > > > task :
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XJvVVYRbp2h8M2f%2BeAZdY5T1jwym5h3522kGS7tZWic%3D&reserved=0
> > > > >
> > > > > Does anyone know what the issue is here ? There are 6-7 failures
> > > because
> > > > > of this test case. Link to the failed test cases for the
> stacktrace :
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=gVJNSjFUhvUUMiKghSW%2F6OMVRgxtjQxm5BJ2h0pTv2s%3D&reserved=0
> > > > > Thanks,
> > > > > Aman.
> > > > >
> > > > > ________________________________
> > > > > From: László Bodor <bo...@gmail.com>
> > > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > Subject: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > >
> > > > > +1
> > > > > also, if I merged something that I thought was for test stability
> > (but
> > > > > instead it was a feature), excuse me :)
> > > > > for reference, the whole green test initiative is tracked under
> this
> > > > > umbrella:
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7Cfe96faae91f8418ecaa108db26fa0a5e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638146627636747901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wIr0EHWrRcXh0D0lLvu8g5r0sxpdFkfn2pFu6Ag%2BJ38%3D&reserved=0
> > > > >
> > > > > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023.
> > febr.
> > > > 7.,
> > > > > K, 12:09):
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > The build in branch-3 is not yet green; there are ~25 test
> > failures.
> > > It
> > > > > is
> > > > > > a common practice that we shouldn't push changes on top of a
> broken
> > > > build
> > > > > > unless they are addressing test failures.
> > > > > >
> > > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo Bodor)
> are
> > > > > working
> > > > > > hard to stabilize the build for quite some time now. If you want
> to
> > > > help
> > > > > > out then start by reviewing, merging, and fixing things around
> test
> > > > > > failures.
> > > > > >
> > > > > > It's not yet the time to bring new features, upgrades, bugs,
> etc.,
> > in
> > > > > > branch-3. I would encourage  committers to not approve such
> changes
> > > > till
> > > > > we
> > > > > > get back to a stable branch.
> > > > > >
> > > > > > Best,
> > > > > > Stamatis
> > > > > >
> > > > >
> > > >
> > >
> >
>
>

Re: [EXTERNAL] Re: Branch-3 backports and build stability

[vihang karajgaonkar <vi...@apache.org>]

Just wanted to close the loop on the TestMiniSparkOnYarnCliDriver test
failures. We will be able to re-enable most of them back on branch-3. The
ones which were disabled are being tracked separately in a different ticket
<https://issues.apache.org/jira/browse/HIVE-27146> but they don't look like
a blocker.

Hi Aman,

Do you know how close are we to reopening branch-3?

Thanks,
Vihang

On Sat, Mar 4, 2023 at 7:23 PM Aman Raj <ra...@microsoft.com.invalid>
wrote:

> Or you can cd into itests and run the command you are using. Just another
> way I run.
>
> Thanks,
> Aman.
> Get Outlook for Android<https://aka.ms/AAb9ysg>
> ________________________________
> From: Aman Raj <ra...@microsoft.com>
> Sent: Saturday, March 4, 2023 7:20:36 PM
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
>
> Hi Vihang,
>
> Thanks a lot for working on this. Can you try using -Pqsplits,itests.
> Also, I usually give a -o option after doing a clean install.
>
> Thanks,
> Aman.
>
> Get Outlook for Android<https://aka.ms/AAb9ysg>
>
> ________________________________
> From: vihang karajgaonkar <vi...@apache.org>
> Sent: Saturday, 4 March, 2023, 11:35
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
>
> [You don't often get email from vihangk1@apache.org. Learn why this is
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> Just to update on the HoS test failures for TestMiniSparkOnYarnCliDriver, I
> think I was finally able to resolve them (at least on local). I had to
> revert HIVE-21044 because it was causing OOM for those tests. Also, in
> order for these tests to work we will have to downgrade netty from
> 4.1.69.Final to 4.1.51.Final. I understand that we had upgraded netty from
> 4.1.17.Final to 4.1.69.Final for CVEs but the highest netty version that we
> can support without breaking HoS is 4.1.51.Final. Note that 4.1.51.Final
> includes many of the CVEs which affected 4.1.17.Final so we are still in a
> better place than branch-3.1. Unfortunately, there is no good way to make
> HoS work with a higher netty version so I think we should downgrade the
> netty version to 4.1.51.Final for now and look at more options to upgrade
> it 4.1.69.Final in a separate ticket.
>
> I still need to understand why the tests which are working for me locally
> don't work on the PR job. I tried running the split test classes using the
> following command. Is that the right way to simulate builds from the PR
> job? Let me know if anyone has more ideas.
>
> mvn test
> -Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
> -Pqsplits
>
> Thanks,
> Vihang
>
>
> On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <za...@gmail.com>
> wrote:
>
> > Hello,
> >
> > Thanks Aman for bringing this up and also for cleaning up after others (I
> > saw that you raised tickets and PRs for addressing the failures).
> >
> > Many thanks to Vihang as well for helping out. Regarding flaky tests, yes
> > we should disable them as soon as we see them.
> > There have been some other discussions on how to approach flaky tests the
> > more recent I could find is here [1].
> >
> > Best,
> > Stamatis
> >
> > [1]
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=vB4E9RakrfYFCHGsxque1mnx9gb06JEXuuW2LJTzttM%3D&reserved=0
> >
> > On Fri, Feb 17, 2023 at 4:37 AM Aman Raj <ra...@microsoft.com.invalid>
> > wrote:
> >
> > > Hi team,
> > >
> > > Thanks Vihang for looking into this. I have commented on the JIRA you
> > > created.
> > >
> > > Just to bring everyone's notice, I have seen that there has been a
> couple
> > > of pushes to branch-3, which has lead to 5 more new test failures. The
> > test
> > > failures are in orc_merge1, orc_merge2, orc_merge3, orc_merge4 and
> > > orc_merge10. These tests did not use to fail before. I would sincerely
> > urge
> > > the community to raise a PR against branch-3, so that the Jenkins
> > pipeline
> > > can run and then only merge things to branch-3. We had 2900+ failures
> > when
> > > we started 2 months back and now having brought it down to less than
> 15,
> > > new failures again has pushed us back in this effort.
> > >
> > > I would like to thank everyone who has participated in this effort and
> > > made it possible till this stage. Also, if the contributors can take
> > > ownership of these new test case failures and fix them, it will be of
> > great
> > > help.
> > >
> > > Thanks,
> > > Aman.
> > > ________________________________
> > > From: vihang karajgaonkar <vi...@apache.org>
> > > Sent: Friday, February 17, 2023 6:10 AM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > >
> > > [You don't often get email from vihangk1@apache.org. Learn why this is
> > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > >
> > > Hi Aman,
> > >
> > > I created
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=AxFvKZaLesnxQ9o3hITgazLHWK7dcxl47JhVcBs0uKQ%3D&reserved=0
> > > to look into
> > > TestMiniSparkOnYarnCliDriver failures. I have a working theory of what
> > > might be going on there. I am still investigating what is the right way
> > to
> > > fix it though.
> > >
> > > Thanks,
> > > Vihang
> > >
> > > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj
> <rajaman@microsoft.com.invalid
> > >
> > > wrote:
> > >
> > > > Hi Vihang,
> > > >
> > > > Yes the tests are failing locally as well with the same issue.
> > > >
> > > > Thanks,
> > > > Aman.
> > > >
> > > > Get Outlook for Android<
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=O5l3TzSJJrjDJgqIdxUlB1VI7%2BcvXZxEq%2F0l9wvvY2s%3D&reserved=0
> > > >
> > > > ________________________________
> > > > From: Vihang Karajgaonkar <vihang.karajgaonkar@databricks.com.INVALID
> >
> > > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > [You don't often get email from
> > > vihang.karajgaonkar@databricks.com.invalid.
> > > > Learn why this is important at
> > > > https://aka.ms/LearnAboutSenderIdentification ]
> > > >
> > > > Thanks a lot Stamatis for starting this thread. I really appreciate
> all
> > > the
> > > > efforts to stabilize branch-3 to get it to a releasable state and I
> > agree
> > > > that we should get it to a green state before opening it for PRs not
> > > > related to test failures. I can help with the effort as well.
> > > >
> > > > If we want to get the branch back to green state soon, have we
> > considered
> > > > disabling the tests which are clearly flaky? (e.g pass on some builds
> > and
> > > > fail on the other build with no new code changes). If we don't do
> that,
> > > we
> > > > will keep playing whack a mole with those tests. I propose for such
> > tests
> > > > we should disable them and create tickets to unflake them separately.
> > > This
> > > > will help us get back to a green state faster.
> > > >
> > > > Hi Aman,
> > > > For TestMiniSparkOnYarnCliDriver failures, you probably should also
> > look
> > > > into the spark driver/application logs and see if there are
> > > infrastructure
> > > > errors (e.g OOMs). Are these tests failing when you run locally?
> > > >
> > > > Thanks,
> > > > Vihang
> > > >
> > > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj
> <rajaman@microsoft.com.invalid
> > >
> > > > wrote:
> > > >
> > > > > +1,
> > > > > Thanks Stamatis and Lazlo for helping in the test case fixes till
> > now.
> > > > >
> > > > > Team,
> > > > > I need help in fixing the following tests in Hive. I have tried
> > > different
> > > > > approaches but no luck till now.
> > > > > I am facing some issues in fixing the following tests :
> > > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > > >
> > > > > Issue :
> > > > > PREHOOK: Input: default@src
> > > > > PREHOOK: Output: default@src
> > > > > Failed to monitor Job[-1] with exception
> > > > > 'java.lang.IllegalStateException(Connection to remote Spark driver
> > was
> > > > > lost)' Last known state = SENT
> > > > > Failed to execute spark task, with exception
> > > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > > FAILED: Execution Error, return code 1 from
> > > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel is
> > closed.
> > > > >
> > > > > History :
> > > > > Initially the tests had failed with errors which I fixed in the
> > > following
> > > > > task :
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=dRsV1sHLgLxon8eBYh%2BX6kG3YaR%2F8Lqd4aZGj4cFjs4%3D&reserved=0
> > > > >
> > > > > Does anyone know what the issue is here ? There are 6-7 failures
> > > because
> > > > > of this test case. Link to the failed test cases for the
> stacktrace :
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SGCWDnQ5QUiy5ycAZWv1V4jXdQHh4zPMi4vtHwP1slU%3D&reserved=0
> > > > > Thanks,
> > > > > Aman.
> > > > >
> > > > > ________________________________
> > > > > From: László Bodor <bo...@gmail.com>
> > > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > Subject: [EXTERNAL] Re: Branch-3 backports and build stability
> > > > >
> > > > > +1
> > > > > also, if I merged something that I thought was for test stability
> > (but
> > > > > instead it was a feature), excuse me :)
> > > > > for reference, the whole green test initiative is tracked under
> this
> > > > > umbrella:
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=yHtzz3SnJq8iJdgDw50qU6KxXYfwEeVCvtHP1C9sFdg%3D&reserved=0
> > > > >
> > > > > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023.
> > febr.
> > > > 7.,
> > > > > K, 12:09):
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > The build in branch-3 is not yet green; there are ~25 test
> > failures.
> > > It
> > > > > is
> > > > > > a common practice that we shouldn't push changes on top of a
> broken
> > > > build
> > > > > > unless they are addressing test failures.
> > > > > >
> > > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo Bodor)
> are
> > > > > working
> > > > > > hard to stabilize the build for quite some time now. If you want
> to
> > > > help
> > > > > > out then start by reviewing, merging, and fixing things around
> test
> > > > > > failures.
> > > > > >
> > > > > > It's not yet the time to bring new features, upgrades, bugs,
> etc.,
> > in
> > > > > > branch-3. I would encourage  committers to not approve such
> changes
> > > > till
> > > > > we
> > > > > > get back to a stable branch.
> > > > > >
> > > > > > Best,
> > > > > > Stamatis
> > > > > >
> > > > >
> > > >
> > >
> >
>
>

Re: [EXTERNAL] Re: Branch-3 backports and build stability

[Aman Raj <ra...@microsoft.com.INVALID>]

Or you can cd into itests and run the command you are using. Just another way I run.

Thanks,
Aman.
Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Aman Raj <ra...@microsoft.com>
Sent: Saturday, March 4, 2023 7:20:36 PM
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability

Hi Vihang,

Thanks a lot for working on this. Can you try using -Pqsplits,itests. Also, I usually give a -o option after doing a clean install.

Thanks,
Aman.

Get Outlook for Android<https://aka.ms/AAb9ysg>

________________________________
From: vihang karajgaonkar <vi...@apache.org>
Sent: Saturday, 4 March, 2023, 11:35
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability

[You don't often get email from vihangk1@apache.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Just to update on the HoS test failures for TestMiniSparkOnYarnCliDriver, I
think I was finally able to resolve them (at least on local). I had to
revert HIVE-21044 because it was causing OOM for those tests. Also, in
order for these tests to work we will have to downgrade netty from
4.1.69.Final to 4.1.51.Final. I understand that we had upgraded netty from
4.1.17.Final to 4.1.69.Final for CVEs but the highest netty version that we
can support without breaking HoS is 4.1.51.Final. Note that 4.1.51.Final
includes many of the CVEs which affected 4.1.17.Final so we are still in a
better place than branch-3.1. Unfortunately, there is no good way to make
HoS work with a higher netty version so I think we should downgrade the
netty version to 4.1.51.Final for now and look at more options to upgrade
it 4.1.69.Final in a separate ticket.

I still need to understand why the tests which are working for me locally
don't work on the PR job. I tried running the split test classes using the
following command. Is that the right way to simulate builds from the PR
job? Let me know if anyone has more ideas.

mvn test
-Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
-Pqsplits

Thanks,
Vihang


On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <za...@gmail.com>
wrote:

> Hello,
>
> Thanks Aman for bringing this up and also for cleaning up after others (I
> saw that you raised tickets and PRs for addressing the failures).
>
> Many thanks to Vihang as well for helping out. Regarding flaky tests, yes
> we should disable them as soon as we see them.
> There have been some other discussions on how to approach flaky tests the
> more recent I could find is here [1].
>
> Best,
> Stamatis
>
> [1] https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=vB4E9RakrfYFCHGsxque1mnx9gb06JEXuuW2LJTzttM%3D&reserved=0
>
> On Fri, Feb 17, 2023 at 4:37 AM Aman Raj <ra...@microsoft.com.invalid>
> wrote:
>
> > Hi team,
> >
> > Thanks Vihang for looking into this. I have commented on the JIRA you
> > created.
> >
> > Just to bring everyone's notice, I have seen that there has been a couple
> > of pushes to branch-3, which has lead to 5 more new test failures. The
> test
> > failures are in orc_merge1, orc_merge2, orc_merge3, orc_merge4 and
> > orc_merge10. These tests did not use to fail before. I would sincerely
> urge
> > the community to raise a PR against branch-3, so that the Jenkins
> pipeline
> > can run and then only merge things to branch-3. We had 2900+ failures
> when
> > we started 2 months back and now having brought it down to less than 15,
> > new failures again has pushed us back in this effort.
> >
> > I would like to thank everyone who has participated in this effort and
> > made it possible till this stage. Also, if the contributors can take
> > ownership of these new test case failures and fix them, it will be of
> great
> > help.
> >
> > Thanks,
> > Aman.
> > ________________________________
> > From: vihang karajgaonkar <vi...@apache.org>
> > Sent: Friday, February 17, 2023 6:10 AM
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > [You don't often get email from vihangk1@apache.org. Learn why this is
> > important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > Hi Aman,
> >
> > I created
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=AxFvKZaLesnxQ9o3hITgazLHWK7dcxl47JhVcBs0uKQ%3D&reserved=0
> > to look into
> > TestMiniSparkOnYarnCliDriver failures. I have a working theory of what
> > might be going on there. I am still investigating what is the right way
> to
> > fix it though.
> >
> > Thanks,
> > Vihang
> >
> > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj <rajaman@microsoft.com.invalid
> >
> > wrote:
> >
> > > Hi Vihang,
> > >
> > > Yes the tests are failing locally as well with the same issue.
> > >
> > > Thanks,
> > > Aman.
> > >
> > > Get Outlook for Android<
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=O5l3TzSJJrjDJgqIdxUlB1VI7%2BcvXZxEq%2F0l9wvvY2s%3D&reserved=0
> > >
> > > ________________________________
> > > From: Vihang Karajgaonkar <vi...@databricks.com.INVALID>
> > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > >
> > > [You don't often get email from
> > vihang.karajgaonkar@databricks.com.invalid.
> > > Learn why this is important at
> > > https://aka.ms/LearnAboutSenderIdentification ]
> > >
> > > Thanks a lot Stamatis for starting this thread. I really appreciate all
> > the
> > > efforts to stabilize branch-3 to get it to a releasable state and I
> agree
> > > that we should get it to a green state before opening it for PRs not
> > > related to test failures. I can help with the effort as well.
> > >
> > > If we want to get the branch back to green state soon, have we
> considered
> > > disabling the tests which are clearly flaky? (e.g pass on some builds
> and
> > > fail on the other build with no new code changes). If we don't do that,
> > we
> > > will keep playing whack a mole with those tests. I propose for such
> tests
> > > we should disable them and create tickets to unflake them separately.
> > This
> > > will help us get back to a green state faster.
> > >
> > > Hi Aman,
> > > For TestMiniSparkOnYarnCliDriver failures, you probably should also
> look
> > > into the spark driver/application logs and see if there are
> > infrastructure
> > > errors (e.g OOMs). Are these tests failing when you run locally?
> > >
> > > Thanks,
> > > Vihang
> > >
> > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj <rajaman@microsoft.com.invalid
> >
> > > wrote:
> > >
> > > > +1,
> > > > Thanks Stamatis and Lazlo for helping in the test case fixes till
> now.
> > > >
> > > > Team,
> > > > I need help in fixing the following tests in Hive. I have tried
> > different
> > > > approaches but no luck till now.
> > > > I am facing some issues in fixing the following tests :
> > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > >
> > > > Issue :
> > > > PREHOOK: Input: default@src
> > > > PREHOOK: Output: default@src
> > > > Failed to monitor Job[-1] with exception
> > > > 'java.lang.IllegalStateException(Connection to remote Spark driver
> was
> > > > lost)' Last known state = SENT
> > > > Failed to execute spark task, with exception
> > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > FAILED: Execution Error, return code 1 from
> > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel is
> closed.
> > > >
> > > > History :
> > > > Initially the tests had failed with errors which I fixed in the
> > following
> > > > task :
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=dRsV1sHLgLxon8eBYh%2BX6kG3YaR%2F8Lqd4aZGj4cFjs4%3D&reserved=0
> > > >
> > > > Does anyone know what the issue is here ? There are 6-7 failures
> > because
> > > > of this test case. Link to the failed test cases for the stacktrace :
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SGCWDnQ5QUiy5ycAZWv1V4jXdQHh4zPMi4vtHwP1slU%3D&reserved=0
> > > > Thanks,
> > > > Aman.
> > > >
> > > > ________________________________
> > > > From: László Bodor <bo...@gmail.com>
> > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > +1
> > > > also, if I merged something that I thought was for test stability
> (but
> > > > instead it was a feature), excuse me :)
> > > > for reference, the whole green test initiative is tracked under this
> > > > umbrella:
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=yHtzz3SnJq8iJdgDw50qU6KxXYfwEeVCvtHP1C9sFdg%3D&reserved=0
> > > >
> > > > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023.
> febr.
> > > 7.,
> > > > K, 12:09):
> > > >
> > > > > Hi all,
> > > > >
> > > > > The build in branch-3 is not yet green; there are ~25 test
> failures.
> > It
> > > > is
> > > > > a common practice that we shouldn't push changes on top of a broken
> > > build
> > > > > unless they are addressing test failures.
> > > > >
> > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo Bodor) are
> > > > working
> > > > > hard to stabilize the build for quite some time now. If you want to
> > > help
> > > > > out then start by reviewing, merging, and fixing things around test
> > > > > failures.
> > > > >
> > > > > It's not yet the time to bring new features, upgrades, bugs, etc.,
> in
> > > > > branch-3. I would encourage  committers to not approve such changes
> > > till
> > > > we
> > > > > get back to a stable branch.
> > > > >
> > > > > Best,
> > > > > Stamatis
> > > > >
> > > >
> > >
> >
>

Re: [EXTERNAL] Re: Branch-3 backports and build stability

[Aman Raj <ra...@microsoft.com.INVALID>]

Hi Vihang,

Thanks a lot for working on this. Can you try using -Pqsplits,itests. Also, I usually give a -o option after doing a clean install.

Thanks,
Aman.

Get Outlook for Android<https://aka.ms/AAb9ysg>

________________________________
From: vihang karajgaonkar <vi...@apache.org>
Sent: Saturday, 4 March, 2023, 11:35
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability

[You don't often get email from vihangk1@apache.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Just to update on the HoS test failures for TestMiniSparkOnYarnCliDriver, I
think I was finally able to resolve them (at least on local). I had to
revert HIVE-21044 because it was causing OOM for those tests. Also, in
order for these tests to work we will have to downgrade netty from
4.1.69.Final to 4.1.51.Final. I understand that we had upgraded netty from
4.1.17.Final to 4.1.69.Final for CVEs but the highest netty version that we
can support without breaking HoS is 4.1.51.Final. Note that 4.1.51.Final
includes many of the CVEs which affected 4.1.17.Final so we are still in a
better place than branch-3.1. Unfortunately, there is no good way to make
HoS work with a higher netty version so I think we should downgrade the
netty version to 4.1.51.Final for now and look at more options to upgrade
it 4.1.69.Final in a separate ticket.

I still need to understand why the tests which are working for me locally
don't work on the PR job. I tried running the split test classes using the
following command. Is that the right way to simulate builds from the PR
job? Let me know if anyone has more ideas.

mvn test
-Dtest=org.apache.hadoop.hive.cli.split2.TestMiniSparkOnYarnCliDriver
-Pqsplits

Thanks,
Vihang


On Fri, Feb 17, 2023 at 4:01 AM Stamatis Zampetakis <za...@gmail.com>
wrote:

> Hello,
>
> Thanks Aman for bringing this up and also for cleaning up after others (I
> saw that you raised tickets and PRs for addressing the failures).
>
> Many thanks to Vihang as well for helping out. Regarding flaky tests, yes
> we should disable them as soon as we see them.
> There have been some other discussions on how to approach flaky tests the
> more recent I could find is here [1].
>
> Best,
> Stamatis
>
> [1] https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Flv3bhlfoq8fwd9dwyjf7g4nx32wtrygv&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=vB4E9RakrfYFCHGsxque1mnx9gb06JEXuuW2LJTzttM%3D&reserved=0
>
> On Fri, Feb 17, 2023 at 4:37 AM Aman Raj <ra...@microsoft.com.invalid>
> wrote:
>
> > Hi team,
> >
> > Thanks Vihang for looking into this. I have commented on the JIRA you
> > created.
> >
> > Just to bring everyone's notice, I have seen that there has been a couple
> > of pushes to branch-3, which has lead to 5 more new test failures. The
> test
> > failures are in orc_merge1, orc_merge2, orc_merge3, orc_merge4 and
> > orc_merge10. These tests did not use to fail before. I would sincerely
> urge
> > the community to raise a PR against branch-3, so that the Jenkins
> pipeline
> > can run and then only merge things to branch-3. We had 2900+ failures
> when
> > we started 2 months back and now having brought it down to less than 15,
> > new failures again has pushed us back in this effort.
> >
> > I would like to thank everyone who has participated in this effort and
> > made it possible till this stage. Also, if the contributors can take
> > ownership of these new test case failures and fix them, it will be of
> great
> > help.
> >
> > Thanks,
> > Aman.
> > ________________________________
> > From: vihang karajgaonkar <vi...@apache.org>
> > Sent: Friday, February 17, 2023 6:10 AM
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> >
> > [You don't often get email from vihangk1@apache.org. Learn why this is
> > important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > Hi Aman,
> >
> > I created
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=AxFvKZaLesnxQ9o3hITgazLHWK7dcxl47JhVcBs0uKQ%3D&reserved=0
> > to look into
> > TestMiniSparkOnYarnCliDriver failures. I have a working theory of what
> > might be going on there. I am still investigating what is the right way
> to
> > fix it though.
> >
> > Thanks,
> > Vihang
> >
> > On Fri, Feb 10, 2023 at 10:26 AM Aman Raj <rajaman@microsoft.com.invalid
> >
> > wrote:
> >
> > > Hi Vihang,
> > >
> > > Yes the tests are failing locally as well with the same issue.
> > >
> > > Thanks,
> > > Aman.
> > >
> > > Get Outlook for Android<
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=O5l3TzSJJrjDJgqIdxUlB1VI7%2BcvXZxEq%2F0l9wvvY2s%3D&reserved=0
> > >
> > > ________________________________
> > > From: Vihang Karajgaonkar <vi...@databricks.com.INVALID>
> > > Sent: Friday, February 10, 2023 11:22:15 PM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Branch-3 backports and build stability
> > >
> > > [You don't often get email from
> > vihang.karajgaonkar@databricks.com.invalid.
> > > Learn why this is important at
> > > https://aka.ms/LearnAboutSenderIdentification ]
> > >
> > > Thanks a lot Stamatis for starting this thread. I really appreciate all
> > the
> > > efforts to stabilize branch-3 to get it to a releasable state and I
> agree
> > > that we should get it to a green state before opening it for PRs not
> > > related to test failures. I can help with the effort as well.
> > >
> > > If we want to get the branch back to green state soon, have we
> considered
> > > disabling the tests which are clearly flaky? (e.g pass on some builds
> and
> > > fail on the other build with no new code changes). If we don't do that,
> > we
> > > will keep playing whack a mole with those tests. I propose for such
> tests
> > > we should disable them and create tickets to unflake them separately.
> > This
> > > will help us get back to a green state faster.
> > >
> > > Hi Aman,
> > > For TestMiniSparkOnYarnCliDriver failures, you probably should also
> look
> > > into the spark driver/application logs and see if there are
> > infrastructure
> > > errors (e.g OOMs). Are these tests failing when you run locally?
> > >
> > > Thanks,
> > > Vihang
> > >
> > > On Tue, Feb 7, 2023 at 10:05 PM Aman Raj <rajaman@microsoft.com.invalid
> >
> > > wrote:
> > >
> > > > +1,
> > > > Thanks Stamatis and Lazlo for helping in the test case fixes till
> now.
> > > >
> > > > Team,
> > > > I need help in fixing the following tests in Hive. I have tried
> > different
> > > > approaches but no luck till now.
> > > > I am facing some issues in fixing the following tests :
> > > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
> > > >
> > > > Issue :
> > > > PREHOOK: Input: default@src
> > > > PREHOOK: Output: default@src
> > > > Failed to monitor Job[-1] with exception
> > > > 'java.lang.IllegalStateException(Connection to remote Spark driver
> was
> > > > lost)' Last known state = SENT
> > > > Failed to execute spark task, with exception
> > > > 'java.lang.IllegalStateException(RPC channel is closed.)'
> > > > FAILED: Execution Error, return code 1 from
> > > > org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel is
> closed.
> > > >
> > > > History :
> > > > Initially the tests had failed with errors which I fixed in the
> > following
> > > > task :
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=dRsV1sHLgLxon8eBYh%2BX6kG3YaR%2F8Lqd4aZGj4cFjs4%3D&reserved=0
> > > >
> > > > Does anyone know what the issue is here ? There are 6-7 failures
> > because
> > > > of this test case. Link to the failed test cases for the stacktrace :
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3949%2F2%2Ftests%2F&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SGCWDnQ5QUiy5ycAZWv1V4jXdQHh4zPMi4vtHwP1slU%3D&reserved=0
> > > > Thanks,
> > > > Aman.
> > > >
> > > > ________________________________
> > > > From: László Bodor <bo...@gmail.com>
> > > > Sent: Tuesday, February 7, 2023 4:46 PM
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: [EXTERNAL] Re: Branch-3 backports and build stability
> > > >
> > > > +1
> > > > also, if I merged something that I thought was for test stability
> (but
> > > > instead it was a feature), excuse me :)
> > > > for reference, the whole green test initiative is tracked under this
> > > > umbrella:
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26836&data=05%7C01%7Crajaman%40microsoft.com%7C24312f2572754c8a428908db1c76210e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638135067023705364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=yHtzz3SnJq8iJdgDw50qU6KxXYfwEeVCvtHP1C9sFdg%3D&reserved=0
> > > >
> > > > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023.
> febr.
> > > 7.,
> > > > K, 12:09):
> > > >
> > > > > Hi all,
> > > > >
> > > > > The build in branch-3 is not yet green; there are ~25 test
> failures.
> > It
> > > > is
> > > > > a common practice that we shouldn't push changes on top of a broken
> > > build
> > > > > unless they are addressing test failures.
> > > > >
> > > > > Some people (mainly Aman Raj, Chris Nauroth, and Laszlo Bodor) are
> > > > working
> > > > > hard to stabilize the build for quite some time now. If you want to
> > > help
> > > > > out then start by reviewing, merging, and fixing things around test
> > > > > failures.
> > > > >
> > > > > It's not yet the time to bring new features, upgrades, bugs, etc.,
> in
> > > > > branch-3. I would encourage  committers to not approve such changes
> > > till
> > > > we
> > > > > get back to a stable branch.
> > > > >
> > > > > Best,
> > > > > Stamatis
> > > > >
> > > >
> > >
> >
>