Proposal to deprecate Hive on Spark from branch-3

[Aman Raj <ra...@microsoft.com.INVALID>]

Hi team,

We have been trying to fix Hive on Spark test failures for a long time. As of now, branch-3 has less than 12 test failures (whose fix have not been identified). 8 of them are related to Hive on Spark. I had mailed about the failures in my previous mail threads. Thanks to Vihang for working on them as well. But we have not been able to identify the root cause till now. These fixes can be tracked in the following tickets : [HIVE-27087] Fix TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF JIRA (apache.org)<https://issues.apache.org/jira/browse/HIVE-27087> and [HIVE-26940] Backport of HIVE-19882 : Fix QTestUtil session lifecycle - ASF JIRA (apache.org)<https://issues.apache.org/jira/browse/HIVE-26940>

Until we have a green branch-3, we cannot go ahead to push new features for the Hive-3.2.0 release. This is kind of a blocker for this release. Already bringing the test fixes to the current state took more than 2 months.

I wanted to bring up a proposal to deprecate Hive on Spark from branch-3 altogether. This would ensure that branch-3 is aligned with the master as done in https://issues.apache.org/jira/browse/HIVE-26134. Just wanted to have a vote on this in parallel working on the test fixes. If we have the approval from the community, we can deprecate it altogether.

Please feel free to suggest any concerns or suggestions you have. Also, I welcome any possible fix suggestion for the test failures.

Thanks,
Aman.

Re: Proposal to deprecate Hive on Spark from branch-3

[Stamatis Zampetakis <za...@gmail.com>]

I am +1 on removing/deprecating Hive on Spark since we've done this on
master and apparently nobody is interested in this feature.

Best,
Stamatis

On Wed, Feb 22, 2023 at 8:18 AM Aman Raj <ra...@microsoft.com.invalid>
wrote:

> Hi team,
>
> We have been trying to fix Hive on Spark test failures for a long time. As
> of now, branch-3 has less than 12 test failures (whose fix have not been
> identified). 8 of them are related to Hive on Spark. I had mailed about the
> failures in my previous mail threads. Thanks to Vihang for working on them
> as well. But we have not been able to identify the root cause till now.
> These fixes can be tracked in the following tickets : [HIVE-27087] Fix
> TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF JIRA (
> apache.org)<https://issues.apache.org/jira/browse/HIVE-27087> and
> [HIVE-26940] Backport of HIVE-19882 : Fix QTestUtil session lifecycle - ASF
> JIRA (apache.org)<https://issues.apache.org/jira/browse/HIVE-26940>
>
> Until we have a green branch-3, we cannot go ahead to push new features
> for the Hive-3.2.0 release. This is kind of a blocker for this release.
> Already bringing the test fixes to the current state took more than 2
> months.
>
> I wanted to bring up a proposal to deprecate Hive on Spark from branch-3
> altogether. This would ensure that branch-3 is aligned with the master as
> done in https://issues.apache.org/jira/browse/HIVE-26134. Just wanted to
> have a vote on this in parallel working on the test fixes. If we have the
> approval from the community, we can deprecate it altogether.
>
> Please feel free to suggest any concerns or suggestions you have. Also, I
> welcome any possible fix suggestion for the test failures.
>
> Thanks,
> Aman.
>

Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3

[Ayush Saxena <ay...@gmail.com>]

Regarding deprecation, I think the general meaning is "this will be removed
in the future releases", it nowhere means it is broken or has issues. I
don't think if we are deprecating it, we can just ignore the failures if
they are indeed bugs or issues with the prod code. We shouldn't release
broken code under the label: "It is deprecated"

3.2.0 indeed is a minor release, which as Stamatis mentioned now might
already have some breaking changes, that is my guess as well, the way
thirdparty libs are being upgraded, a bunch of them are well known for poor
backward compatibility. And these library compatibility usually is one of
the core reasons which pushes other projects from upgrading...

So, If we are taking it that way, we should avoid any such breaking changes
which aren't in scope of a minor release. Don't pull in any major features
or any change which have major user level impacts and beyond the scope of a
minor release.

BTW. A fix even if committed to the branch, if it isn't in line with the
scope of a "maintenance release" can not be considered "accepted". It can
be vetoed during the release vote and will push everything back, So, either
have a prior agreement or not push any breaking change via hive code or
coming via any thirdparty lib upgrades. It is just gonna create troubles
and might fetch vetoes during the release vote, just my 2 cents...

-Ayush

On Tue, 28 Feb 2023 at 04:08, vihang karajgaonkar <vi...@apache.org>
wrote:

> I think 3.2.0 is seen as a minor release, not a maintenance release. Eg.
> 3.2.1 would be a maintenance release which typically includes bug fixes,
> minor usability improvements and security fixes. If we block new features
> from going into minor releases that would be a step back in my opinion.
> I am pretty sure we have released small features in minor releases before
> (e.g 2.2.0 release notes
> <
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12335837&styleName=Text&projectId=12310843
> >).
> I would request not to club new features and breaking features together in
> the same decision.
>
> For breaking changes, I don't think we have published guidelines before so
> I am open to that. Especially for the HoS tests which are failing, some of
> the failures are related to dependency upgrades (done for security reasons)
> while some have no clear root-cause yet. For a feature which has been
> removed in master branch, I think it is okay to deprecate it and disable
> the tests given that we have spent considerable time on them already and
> none of the original HoS contributors are available to help.
>
> Thanks,
> Vihang
>
> On Mon, Feb 27, 2023 at 1:24 PM Stamatis Zampetakis <za...@gmail.com>
> wrote:
>
> > Some people raised a valid point that branch-3 is a maintenance branch.
> If
> > we really aim 3.2.0 to be a maintenance release then we should minimize
> > breaking changes and prohibit new features. In this case Spark cannot go
> > away and the only thing we can do is deprecate it. It also means that we
> > should fix the tests cause failures typically indicate breaking changes
> > which again are not tolerable for a maintenance release.
> >
> > On the other hand,
> > I got the impression that some people were interested for getting new
> > features in 3.2.0 (  some may be in already). Furthermore, some
> dependency
> > upgrades may also lead to breaking changes/different behavior so we
> should
> > definitely agree on what is acceptable and what is not for branch-3.
> >
> > Summing up the question boils down to the following. Do we allow breaking
> > changes and new features in branch-3 or not?
> >
> > Best,
> > Stamatis
> >
> > On Fri, Feb 24, 2023, 10:41 AM Aman Raj <ra...@microsoft.com.invalid>
> > wrote:
> >
> > > Hi Laszlo,
> > >
> > > I am perfectly fine with disabling the Hive on Spark tests. In fact, I
> > > prefer that. I agree with Vihang and you on this. I had proposed this
> > idea
> > > long back (of disabling the test cases) and then we had discussed on
> the
> > > community that either we fix the Hive on Spark test cases or remove
> Hive
> > on
> > > Spark. Therefore, I initiated this thread of removing Hive on Spark
> since
> > > we are not still able to resolve the test cases since the past couple
> of
> > > months.
> > >
> > > Thanks,
> > > Aman.
> > >
> > > ________________________________
> > > From: László Bodor <bo...@gmail.com>
> > > Sent: Friday, February 24, 2023 2:57 PM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from
> > > branch-3
> > >
> > > +1 on Vihang's suggestion
> > > I remember that spark removal was a debated thing even on master, so
> > > completely removing it backwards from a "maintenance" branch-3 line is
> > not
> > > really acceptable (actually, I'm surprised it's not -1ed yet by hive on
> > > spark folks), but it depends on what *deprecation* really means: I mean
> > > disabling some spark tests to stabilize precommit is completely fine in
> > the
> > > absence of community aspiration to fix them properly
> > >
> > > regarding the motivation: "This would ensure that branch-3 is aligned
> > with
> > > the master as done in ..."  <-- I don't think we're targeting this, we
> > are
> > > about to make 3.x releases as simply as possible
> > >
> > > I'm hoping/assuming that most of the +1s so far are in line with
> Vihang's
> > > suggestion
> > >
> > > vihang karajgaonkar <vi...@apache.org> ezt írta (időpont: 2023.
> febr.
> > > 23., Cs, 16:37):
> > >
> > > > +1 to deprecate Hive on Spark.
> > > >
> > > > I feel directly removing it in a minor release is probably a bad
> idea.
> > > Most
> > > > users will upgrade to 3.2 first and go to 4.0 later. If we deprecate
> it
> > > in
> > > > 3.2 it transitions well into its removal as users upgrade to 4.0
> > > > eventually.
> > > >
> > > > If the goal to stabilize the branch-3, we can disable the failing
> tests
> > > on
> > > > Hive on Spark.
> > > >
> > > > Thanks,
> > > > Vihang
> > > >
> > > > On Thu, Feb 23, 2023 at 12:32 AM Alessandro Solimando <
> > > > alessandro.solimando@gmail.com> wrote:
> > > >
> > > > > +1 from me too
> > > > >
> > > > > On Thu, 23 Feb 2023 at 06:09, Ayush Saxena <ay...@gmail.com>
> > wrote:
> > > > >
> > > > > > +1 on removing Hive on Spark from branch-3
> > > > > >
> > > > > > -Ayush
> > > > > >
> > > > > > > On 23-Feb-2023, at 6:40 AM, Wang, Yuming
> > <yumwang@ebay.com.invalid
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > +1.
> > > > > > >
> > > > > > > From: Naresh P R <na...@apache.org>
> > > > > > > Date: Thursday, February 23, 2023 at 02:49
> > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark
> > > from
> > > > > > branch-3
> > > > > > > External Email
> > > > > > >
> > > > > > > +1 to remove Hive on Spark in branch-3
> > > > > > > ---
> > > > > > > Regards,
> > > > > > > Naresh P R
> > > > > > >
> > > > > > >> On Wed, Feb 22, 2023 at 5:37 AM Sankar Hariappan
> > > > > > >> <Sa...@microsoft.com.invalid> wrote:
> > > > > > >>
> > > > > > >> +1, to remove Hive on Spark in branch-3.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Sankar
> > > > > > >>
> > > > > > >> -----Original Message-----
> > > > > > >> From: Rajesh Balamohan <rb...@apache.org>
> > > > > > >> Sent: Wednesday, February 22, 2023 6:58 PM
> > > > > > >> To: dev@hive.apache.org
> > > > > > >> Subject: [EXTERNAL] Re: Proposal to deprecate Hive on Spark
> from
> > > > > > branch-3
> > > > > > >>
> > > > > > >> +1 on removing Hive on Spark in branch-3.
> > > > > > >>
> > > > > > >> It was not done earlier since it was removing a feature in the
> > > > branch.
> > > > > > But
> > > > > > >> if there is enough consensus, we should consider removing it.
> > > > > > >>
> > > > > > >> ~Rajesh.B
> > > > > > >>
> > > > > > >> On Wed, Feb 22, 2023 at 12:48 PM Aman Raj
> > > > > <rajaman@microsoft.com.invalid
> > > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Hi team,
> > > > > > >>>
> > > > > > >>> We have been trying to fix Hive on Spark test failures for a
> > long
> > > > > > >>> time. As of now, branch-3 has less than 12 test failures
> (whose
> > > fix
> > > > > > >>> have not been identified). 8 of them are related to Hive on
> > > Spark.
> > > > I
> > > > > > >>> had mailed about the failures in my previous mail threads.
> > Thanks
> > > > to
> > > > > > >>> Vihang for working on them as well. But we have not been able
> > to
> > > > > > >> identify the root cause till now.
> > > > > > >>> These fixes can be tracked in the following tickets :
> > > [HIVE-27087]
> > > > > Fix
> > > > > > >>> TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF
> > > JIRA (
> > > > > > >>> apache.org)<
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%2525&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KORvfBkmdpqIFOoWr6J4X%2BqAQO6jcykzjY3%2FU0pq0y4%3D&reserved=0
> > > > > > >>> 3A%2F%2Fissues.apache.org
> > > > > %2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7C
> > > > > > >>> Sankar.Hariappan%40microsoft.com
> > > > > %7C687a6a4dbd41454568e008db14d8cc23%7C
> > > > > > >>>
> > > > >
> > 72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknow
> > > > > > >>>
> > > > >
> > n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> > > > > > >>>
> > > > >
> > JXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RbAqrwK6fQFDStufXYfpusNc81EzjtpiaHm
> > > > > > >>> qv5CFiAs%3D&reserved=0> and [HIVE-26940] Backport of
> > HIVE-19882 :
> > > > Fix
> > > > > > >>> QTestUtil session lifecycle - ASF JIRA
> > > > > > >>> (apache.org)<
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttps%2F&data=05%7C01%7Crajaman%40microsoft.com%7C7342dd78ed7f424b36a608db16495672%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638128276496747003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rokHWpxWQ0xYFDr%2BC2p%2BFv4gjLtv%2FCXeZGxdcCSFcVM%3D&reserved=0
> > > > > > >>> %3A%2F%2Fissues.apache.org
> > > > > %2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7
> > > > > > >>> CSankar.Hariappan%40microsoft.com
> > > > > %7C687a6a4dbd41454568e008db14d8cc23%7
> > > > > > >>>
> > > > >
> > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnkno
> > > > > > >>>
> > > > >
> > wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> > > > > > >>>
> > > > >
> > CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PaDtEZD569Sv0ER9sC4l6q1ZxyoBeER3zn
> > > > > > >>> Bsc51PWI8%3D&reserved=0>
> > > > > > >>>
> > > > > > >>> Until we have a green branch-3, we cannot go ahead to push
> new
> > > > > > >>> features for the Hive-3.2.0 release. This is kind of a
> blocker
> > > for
> > > > > this
> > > > > > >> release.
> > > > > > >>> Already bringing the test fixes to the current state took
> more
> > > > than 2
> > > > > > >>> months.
> > > > > > >>>
> > > > > > >>> I wanted to bring up a proposal to deprecate Hive on Spark
> from
> > > > > > >>> branch-3 altogether. This would ensure that branch-3 is
> aligned
> > > > with
> > > > > > >>> the master as done in
> > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu%2F&data=05%7C01%7Crajaman%40microsoft.com%7C7342dd78ed7f424b36a608db16495672%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638128276496747003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ajkc6Ev5ts7dQobFo1gwBDRvTt9ggRFTasOz%2B5k3Ocg%3D&reserved=0
> > > > > > >>> es.apache.org
> > > > > > >> %2Fjira%2Fbrowse%2FHIVE-26134&data=05%7C01%7CSankar.Hariappan%
> > > > > > >> 40microsoft.com
> > > > > >
> > > > >
> > > >
> > >
> >
> %7C687a6a4dbd41454568e008db14d8cc23%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j%2F%2BYTakSvfk1Mm%2Fq8kI84gKm6s%2Fg2iA1abIPzY7t%2Bsg%3D&reserved=0.
> > > > > > >> Just wanted to have a vote on this in parallel working on the
> > test
> > > > > > fixes.
> > > > > > >> If we have the approval from the community, we can deprecate
> it
> > > > > > altogether.
> > > > > > >>>
> > > > > > >>> Please feel free to suggest any concerns or suggestions you
> > have.
> > > > > > >>> Also, I welcome any possible fix suggestion for the test
> > > failures.
> > > > > > >>>
> > > > > > >>> Thanks,
> > > > > > >>> Aman.
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3

[vihang karajgaonkar <vi...@apache.org>]

I think 3.2.0 is seen as a minor release, not a maintenance release. Eg.
3.2.1 would be a maintenance release which typically includes bug fixes,
minor usability improvements and security fixes. If we block new features
from going into minor releases that would be a step back in my opinion.
I am pretty sure we have released small features in minor releases before
(e.g 2.2.0 release notes
<https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12335837&styleName=Text&projectId=12310843>).
I would request not to club new features and breaking features together in
the same decision.

For breaking changes, I don't think we have published guidelines before so
I am open to that. Especially for the HoS tests which are failing, some of
the failures are related to dependency upgrades (done for security reasons)
while some have no clear root-cause yet. For a feature which has been
removed in master branch, I think it is okay to deprecate it and disable
the tests given that we have spent considerable time on them already and
none of the original HoS contributors are available to help.

Thanks,
Vihang

On Mon, Feb 27, 2023 at 1:24 PM Stamatis Zampetakis <za...@gmail.com>
wrote:

> Some people raised a valid point that branch-3 is a maintenance branch. If
> we really aim 3.2.0 to be a maintenance release then we should minimize
> breaking changes and prohibit new features. In this case Spark cannot go
> away and the only thing we can do is deprecate it. It also means that we
> should fix the tests cause failures typically indicate breaking changes
> which again are not tolerable for a maintenance release.
>
> On the other hand,
> I got the impression that some people were interested for getting new
> features in 3.2.0 (  some may be in already). Furthermore, some dependency
> upgrades may also lead to breaking changes/different behavior so we should
> definitely agree on what is acceptable and what is not for branch-3.
>
> Summing up the question boils down to the following. Do we allow breaking
> changes and new features in branch-3 or not?
>
> Best,
> Stamatis
>
> On Fri, Feb 24, 2023, 10:41 AM Aman Raj <ra...@microsoft.com.invalid>
> wrote:
>
> > Hi Laszlo,
> >
> > I am perfectly fine with disabling the Hive on Spark tests. In fact, I
> > prefer that. I agree with Vihang and you on this. I had proposed this
> idea
> > long back (of disabling the test cases) and then we had discussed on the
> > community that either we fix the Hive on Spark test cases or remove Hive
> on
> > Spark. Therefore, I initiated this thread of removing Hive on Spark since
> > we are not still able to resolve the test cases since the past couple of
> > months.
> >
> > Thanks,
> > Aman.
> >
> > ________________________________
> > From: László Bodor <bo...@gmail.com>
> > Sent: Friday, February 24, 2023 2:57 PM
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from
> > branch-3
> >
> > +1 on Vihang's suggestion
> > I remember that spark removal was a debated thing even on master, so
> > completely removing it backwards from a "maintenance" branch-3 line is
> not
> > really acceptable (actually, I'm surprised it's not -1ed yet by hive on
> > spark folks), but it depends on what *deprecation* really means: I mean
> > disabling some spark tests to stabilize precommit is completely fine in
> the
> > absence of community aspiration to fix them properly
> >
> > regarding the motivation: "This would ensure that branch-3 is aligned
> with
> > the master as done in ..."  <-- I don't think we're targeting this, we
> are
> > about to make 3.x releases as simply as possible
> >
> > I'm hoping/assuming that most of the +1s so far are in line with Vihang's
> > suggestion
> >
> > vihang karajgaonkar <vi...@apache.org> ezt írta (időpont: 2023. febr.
> > 23., Cs, 16:37):
> >
> > > +1 to deprecate Hive on Spark.
> > >
> > > I feel directly removing it in a minor release is probably a bad idea.
> > Most
> > > users will upgrade to 3.2 first and go to 4.0 later. If we deprecate it
> > in
> > > 3.2 it transitions well into its removal as users upgrade to 4.0
> > > eventually.
> > >
> > > If the goal to stabilize the branch-3, we can disable the failing tests
> > on
> > > Hive on Spark.
> > >
> > > Thanks,
> > > Vihang
> > >
> > > On Thu, Feb 23, 2023 at 12:32 AM Alessandro Solimando <
> > > alessandro.solimando@gmail.com> wrote:
> > >
> > > > +1 from me too
> > > >
> > > > On Thu, 23 Feb 2023 at 06:09, Ayush Saxena <ay...@gmail.com>
> wrote:
> > > >
> > > > > +1 on removing Hive on Spark from branch-3
> > > > >
> > > > > -Ayush
> > > > >
> > > > > > On 23-Feb-2023, at 6:40 AM, Wang, Yuming
> <yumwang@ebay.com.invalid
> > >
> > > > > wrote:
> > > > > >
> > > > > > +1.
> > > > > >
> > > > > > From: Naresh P R <na...@apache.org>
> > > > > > Date: Thursday, February 23, 2023 at 02:49
> > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark
> > from
> > > > > branch-3
> > > > > > External Email
> > > > > >
> > > > > > +1 to remove Hive on Spark in branch-3
> > > > > > ---
> > > > > > Regards,
> > > > > > Naresh P R
> > > > > >
> > > > > >> On Wed, Feb 22, 2023 at 5:37 AM Sankar Hariappan
> > > > > >> <Sa...@microsoft.com.invalid> wrote:
> > > > > >>
> > > > > >> +1, to remove Hive on Spark in branch-3.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Sankar
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: Rajesh Balamohan <rb...@apache.org>
> > > > > >> Sent: Wednesday, February 22, 2023 6:58 PM
> > > > > >> To: dev@hive.apache.org
> > > > > >> Subject: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from
> > > > > branch-3
> > > > > >>
> > > > > >> +1 on removing Hive on Spark in branch-3.
> > > > > >>
> > > > > >> It was not done earlier since it was removing a feature in the
> > > branch.
> > > > > But
> > > > > >> if there is enough consensus, we should consider removing it.
> > > > > >>
> > > > > >> ~Rajesh.B
> > > > > >>
> > > > > >> On Wed, Feb 22, 2023 at 12:48 PM Aman Raj
> > > > <rajaman@microsoft.com.invalid
> > > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hi team,
> > > > > >>>
> > > > > >>> We have been trying to fix Hive on Spark test failures for a
> long
> > > > > >>> time. As of now, branch-3 has less than 12 test failures (whose
> > fix
> > > > > >>> have not been identified). 8 of them are related to Hive on
> > Spark.
> > > I
> > > > > >>> had mailed about the failures in my previous mail threads.
> Thanks
> > > to
> > > > > >>> Vihang for working on them as well. But we have not been able
> to
> > > > > >> identify the root cause till now.
> > > > > >>> These fixes can be tracked in the following tickets :
> > [HIVE-27087]
> > > > Fix
> > > > > >>> TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF
> > JIRA (
> > > > > >>> apache.org)<
> > > > >
> > > >
> > >
> >
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%2525&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KORvfBkmdpqIFOoWr6J4X%2BqAQO6jcykzjY3%2FU0pq0y4%3D&reserved=0
> > > > > >>> 3A%2F%2Fissues.apache.org
> > > > %2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7C
> > > > > >>> Sankar.Hariappan%40microsoft.com
> > > > %7C687a6a4dbd41454568e008db14d8cc23%7C
> > > > > >>>
> > > >
> 72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknow
> > > > > >>>
> > > >
> n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> > > > > >>>
> > > >
> JXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RbAqrwK6fQFDStufXYfpusNc81EzjtpiaHm
> > > > > >>> qv5CFiAs%3D&reserved=0> and [HIVE-26940] Backport of
> HIVE-19882 :
> > > Fix
> > > > > >>> QTestUtil session lifecycle - ASF JIRA
> > > > > >>> (apache.org)<
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttps%2F&data=05%7C01%7Crajaman%40microsoft.com%7C7342dd78ed7f424b36a608db16495672%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638128276496747003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rokHWpxWQ0xYFDr%2BC2p%2BFv4gjLtv%2FCXeZGxdcCSFcVM%3D&reserved=0
> > > > > >>> %3A%2F%2Fissues.apache.org
> > > > %2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7
> > > > > >>> CSankar.Hariappan%40microsoft.com
> > > > %7C687a6a4dbd41454568e008db14d8cc23%7
> > > > > >>>
> > > >
> C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnkno
> > > > > >>>
> > > >
> wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> > > > > >>>
> > > >
> CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PaDtEZD569Sv0ER9sC4l6q1ZxyoBeER3zn
> > > > > >>> Bsc51PWI8%3D&reserved=0>
> > > > > >>>
> > > > > >>> Until we have a green branch-3, we cannot go ahead to push new
> > > > > >>> features for the Hive-3.2.0 release. This is kind of a blocker
> > for
> > > > this
> > > > > >> release.
> > > > > >>> Already bringing the test fixes to the current state took more
> > > than 2
> > > > > >>> months.
> > > > > >>>
> > > > > >>> I wanted to bring up a proposal to deprecate Hive on Spark from
> > > > > >>> branch-3 altogether. This would ensure that branch-3 is aligned
> > > with
> > > > > >>> the master as done in
> > > > > >>>
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu%2F&data=05%7C01%7Crajaman%40microsoft.com%7C7342dd78ed7f424b36a608db16495672%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638128276496747003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ajkc6Ev5ts7dQobFo1gwBDRvTt9ggRFTasOz%2B5k3Ocg%3D&reserved=0
> > > > > >>> es.apache.org
> > > > > >> %2Fjira%2Fbrowse%2FHIVE-26134&data=05%7C01%7CSankar.Hariappan%
> > > > > >> 40microsoft.com
> > > > >
> > > >
> > >
> >
> %7C687a6a4dbd41454568e008db14d8cc23%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j%2F%2BYTakSvfk1Mm%2Fq8kI84gKm6s%2Fg2iA1abIPzY7t%2Bsg%3D&reserved=0.
> > > > > >> Just wanted to have a vote on this in parallel working on the
> test
> > > > > fixes.
> > > > > >> If we have the approval from the community, we can deprecate it
> > > > > altogether.
> > > > > >>>
> > > > > >>> Please feel free to suggest any concerns or suggestions you
> have.
> > > > > >>> Also, I welcome any possible fix suggestion for the test
> > failures.
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>> Aman.
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3

[Stamatis Zampetakis <za...@gmail.com>]

Some people raised a valid point that branch-3 is a maintenance branch. If
we really aim 3.2.0 to be a maintenance release then we should minimize
breaking changes and prohibit new features. In this case Spark cannot go
away and the only thing we can do is deprecate it. It also means that we
should fix the tests cause failures typically indicate breaking changes
which again are not tolerable for a maintenance release.

On the other hand,
I got the impression that some people were interested for getting new
features in 3.2.0 (  some may be in already). Furthermore, some dependency
upgrades may also lead to breaking changes/different behavior so we should
definitely agree on what is acceptable and what is not for branch-3.

Summing up the question boils down to the following. Do we allow breaking
changes and new features in branch-3 or not?

Best,
Stamatis

On Fri, Feb 24, 2023, 10:41 AM Aman Raj <ra...@microsoft.com.invalid>
wrote:

> Hi Laszlo,
>
> I am perfectly fine with disabling the Hive on Spark tests. In fact, I
> prefer that. I agree with Vihang and you on this. I had proposed this idea
> long back (of disabling the test cases) and then we had discussed on the
> community that either we fix the Hive on Spark test cases or remove Hive on
> Spark. Therefore, I initiated this thread of removing Hive on Spark since
> we are not still able to resolve the test cases since the past couple of
> months.
>
> Thanks,
> Aman.
>
> ________________________________
> From: László Bodor <bo...@gmail.com>
> Sent: Friday, February 24, 2023 2:57 PM
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from
> branch-3
>
> +1 on Vihang's suggestion
> I remember that spark removal was a debated thing even on master, so
> completely removing it backwards from a "maintenance" branch-3 line is not
> really acceptable (actually, I'm surprised it's not -1ed yet by hive on
> spark folks), but it depends on what *deprecation* really means: I mean
> disabling some spark tests to stabilize precommit is completely fine in the
> absence of community aspiration to fix them properly
>
> regarding the motivation: "This would ensure that branch-3 is aligned with
> the master as done in ..."  <-- I don't think we're targeting this, we are
> about to make 3.x releases as simply as possible
>
> I'm hoping/assuming that most of the +1s so far are in line with Vihang's
> suggestion
>
> vihang karajgaonkar <vi...@apache.org> ezt írta (időpont: 2023. febr.
> 23., Cs, 16:37):
>
> > +1 to deprecate Hive on Spark.
> >
> > I feel directly removing it in a minor release is probably a bad idea.
> Most
> > users will upgrade to 3.2 first and go to 4.0 later. If we deprecate it
> in
> > 3.2 it transitions well into its removal as users upgrade to 4.0
> > eventually.
> >
> > If the goal to stabilize the branch-3, we can disable the failing tests
> on
> > Hive on Spark.
> >
> > Thanks,
> > Vihang
> >
> > On Thu, Feb 23, 2023 at 12:32 AM Alessandro Solimando <
> > alessandro.solimando@gmail.com> wrote:
> >
> > > +1 from me too
> > >
> > > On Thu, 23 Feb 2023 at 06:09, Ayush Saxena <ay...@gmail.com> wrote:
> > >
> > > > +1 on removing Hive on Spark from branch-3
> > > >
> > > > -Ayush
> > > >
> > > > > On 23-Feb-2023, at 6:40 AM, Wang, Yuming <yumwang@ebay.com.invalid
> >
> > > > wrote:
> > > > >
> > > > > +1.
> > > > >
> > > > > From: Naresh P R <na...@apache.org>
> > > > > Date: Thursday, February 23, 2023 at 02:49
> > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark
> from
> > > > branch-3
> > > > > External Email
> > > > >
> > > > > +1 to remove Hive on Spark in branch-3
> > > > > ---
> > > > > Regards,
> > > > > Naresh P R
> > > > >
> > > > >> On Wed, Feb 22, 2023 at 5:37 AM Sankar Hariappan
> > > > >> <Sa...@microsoft.com.invalid> wrote:
> > > > >>
> > > > >> +1, to remove Hive on Spark in branch-3.
> > > > >>
> > > > >> Thanks,
> > > > >> Sankar
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: Rajesh Balamohan <rb...@apache.org>
> > > > >> Sent: Wednesday, February 22, 2023 6:58 PM
> > > > >> To: dev@hive.apache.org
> > > > >> Subject: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from
> > > > branch-3
> > > > >>
> > > > >> +1 on removing Hive on Spark in branch-3.
> > > > >>
> > > > >> It was not done earlier since it was removing a feature in the
> > branch.
> > > > But
> > > > >> if there is enough consensus, we should consider removing it.
> > > > >>
> > > > >> ~Rajesh.B
> > > > >>
> > > > >> On Wed, Feb 22, 2023 at 12:48 PM Aman Raj
> > > <rajaman@microsoft.com.invalid
> > > > >
> > > > >> wrote:
> > > > >>
> > > > >>> Hi team,
> > > > >>>
> > > > >>> We have been trying to fix Hive on Spark test failures for a long
> > > > >>> time. As of now, branch-3 has less than 12 test failures (whose
> fix
> > > > >>> have not been identified). 8 of them are related to Hive on
> Spark.
> > I
> > > > >>> had mailed about the failures in my previous mail threads. Thanks
> > to
> > > > >>> Vihang for working on them as well. But we have not been able to
> > > > >> identify the root cause till now.
> > > > >>> These fixes can be tracked in the following tickets :
> [HIVE-27087]
> > > Fix
> > > > >>> TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF
> JIRA (
> > > > >>> apache.org)<
> > > >
> > >
> >
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%2525&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KORvfBkmdpqIFOoWr6J4X%2BqAQO6jcykzjY3%2FU0pq0y4%3D&reserved=0
> > > > >>> 3A%2F%2Fissues.apache.org
> > > %2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7C
> > > > >>> Sankar.Hariappan%40microsoft.com
> > > %7C687a6a4dbd41454568e008db14d8cc23%7C
> > > > >>>
> > > 72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknow
> > > > >>>
> > > n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> > > > >>>
> > > JXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RbAqrwK6fQFDStufXYfpusNc81EzjtpiaHm
> > > > >>> qv5CFiAs%3D&reserved=0> and [HIVE-26940] Backport of HIVE-19882 :
> > Fix
> > > > >>> QTestUtil session lifecycle - ASF JIRA
> > > > >>> (apache.org)<
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttps%2F&data=05%7C01%7Crajaman%40microsoft.com%7C7342dd78ed7f424b36a608db16495672%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638128276496747003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rokHWpxWQ0xYFDr%2BC2p%2BFv4gjLtv%2FCXeZGxdcCSFcVM%3D&reserved=0
> > > > >>> %3A%2F%2Fissues.apache.org
> > > %2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7
> > > > >>> CSankar.Hariappan%40microsoft.com
> > > %7C687a6a4dbd41454568e008db14d8cc23%7
> > > > >>>
> > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnkno
> > > > >>>
> > > wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> > > > >>>
> > > CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PaDtEZD569Sv0ER9sC4l6q1ZxyoBeER3zn
> > > > >>> Bsc51PWI8%3D&reserved=0>
> > > > >>>
> > > > >>> Until we have a green branch-3, we cannot go ahead to push new
> > > > >>> features for the Hive-3.2.0 release. This is kind of a blocker
> for
> > > this
> > > > >> release.
> > > > >>> Already bringing the test fixes to the current state took more
> > than 2
> > > > >>> months.
> > > > >>>
> > > > >>> I wanted to bring up a proposal to deprecate Hive on Spark from
> > > > >>> branch-3 altogether. This would ensure that branch-3 is aligned
> > with
> > > > >>> the master as done in
> > > > >>>
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu%2F&data=05%7C01%7Crajaman%40microsoft.com%7C7342dd78ed7f424b36a608db16495672%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638128276496747003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ajkc6Ev5ts7dQobFo1gwBDRvTt9ggRFTasOz%2B5k3Ocg%3D&reserved=0
> > > > >>> es.apache.org
> > > > >> %2Fjira%2Fbrowse%2FHIVE-26134&data=05%7C01%7CSankar.Hariappan%
> > > > >> 40microsoft.com
> > > >
> > >
> >
> %7C687a6a4dbd41454568e008db14d8cc23%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j%2F%2BYTakSvfk1Mm%2Fq8kI84gKm6s%2Fg2iA1abIPzY7t%2Bsg%3D&reserved=0.
> > > > >> Just wanted to have a vote on this in parallel working on the test
> > > > fixes.
> > > > >> If we have the approval from the community, we can deprecate it
> > > > altogether.
> > > > >>>
> > > > >>> Please feel free to suggest any concerns or suggestions you have.
> > > > >>> Also, I welcome any possible fix suggestion for the test
> failures.
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Aman.
> > > > >>>
> > > > >>
> > > >
> > >
> >
>

Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3

[Aman Raj <ra...@microsoft.com.INVALID>]

Hi Laszlo,

I am perfectly fine with disabling the Hive on Spark tests. In fact, I prefer that. I agree with Vihang and you on this. I had proposed this idea long back (of disabling the test cases) and then we had discussed on the community that either we fix the Hive on Spark test cases or remove Hive on Spark. Therefore, I initiated this thread of removing Hive on Spark since we are not still able to resolve the test cases since the past couple of months.

Thanks,
Aman.

________________________________
From: László Bodor <bo...@gmail.com>
Sent: Friday, February 24, 2023 2:57 PM
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3

+1 on Vihang's suggestion
I remember that spark removal was a debated thing even on master, so
completely removing it backwards from a "maintenance" branch-3 line is not
really acceptable (actually, I'm surprised it's not -1ed yet by hive on
spark folks), but it depends on what *deprecation* really means: I mean
disabling some spark tests to stabilize precommit is completely fine in the
absence of community aspiration to fix them properly

regarding the motivation: "This would ensure that branch-3 is aligned with
the master as done in ..."  <-- I don't think we're targeting this, we are
about to make 3.x releases as simply as possible

I'm hoping/assuming that most of the +1s so far are in line with Vihang's
suggestion

vihang karajgaonkar <vi...@apache.org> ezt írta (időpont: 2023. febr.
23., Cs, 16:37):

> +1 to deprecate Hive on Spark.
>
> I feel directly removing it in a minor release is probably a bad idea. Most
> users will upgrade to 3.2 first and go to 4.0 later. If we deprecate it in
> 3.2 it transitions well into its removal as users upgrade to 4.0
> eventually.
>
> If the goal to stabilize the branch-3, we can disable the failing tests on
> Hive on Spark.
>
> Thanks,
> Vihang
>
> On Thu, Feb 23, 2023 at 12:32 AM Alessandro Solimando <
> alessandro.solimando@gmail.com> wrote:
>
> > +1 from me too
> >
> > On Thu, 23 Feb 2023 at 06:09, Ayush Saxena <ay...@gmail.com> wrote:
> >
> > > +1 on removing Hive on Spark from branch-3
> > >
> > > -Ayush
> > >
> > > > On 23-Feb-2023, at 6:40 AM, Wang, Yuming <yu...@ebay.com.invalid>
> > > wrote:
> > > >
> > > > +1.
> > > >
> > > > From: Naresh P R <na...@apache.org>
> > > > Date: Thursday, February 23, 2023 at 02:49
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from
> > > branch-3
> > > > External Email
> > > >
> > > > +1 to remove Hive on Spark in branch-3
> > > > ---
> > > > Regards,
> > > > Naresh P R
> > > >
> > > >> On Wed, Feb 22, 2023 at 5:37 AM Sankar Hariappan
> > > >> <Sa...@microsoft.com.invalid> wrote:
> > > >>
> > > >> +1, to remove Hive on Spark in branch-3.
> > > >>
> > > >> Thanks,
> > > >> Sankar
> > > >>
> > > >> -----Original Message-----
> > > >> From: Rajesh Balamohan <rb...@apache.org>
> > > >> Sent: Wednesday, February 22, 2023 6:58 PM
> > > >> To: dev@hive.apache.org
> > > >> Subject: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from
> > > branch-3
> > > >>
> > > >> +1 on removing Hive on Spark in branch-3.
> > > >>
> > > >> It was not done earlier since it was removing a feature in the
> branch.
> > > But
> > > >> if there is enough consensus, we should consider removing it.
> > > >>
> > > >> ~Rajesh.B
> > > >>
> > > >> On Wed, Feb 22, 2023 at 12:48 PM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > >> wrote:
> > > >>
> > > >>> Hi team,
> > > >>>
> > > >>> We have been trying to fix Hive on Spark test failures for a long
> > > >>> time. As of now, branch-3 has less than 12 test failures (whose fix
> > > >>> have not been identified). 8 of them are related to Hive on Spark.
> I
> > > >>> had mailed about the failures in my previous mail threads. Thanks
> to
> > > >>> Vihang for working on them as well. But we have not been able to
> > > >> identify the root cause till now.
> > > >>> These fixes can be tracked in the following tickets : [HIVE-27087]
> > Fix
> > > >>> TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF JIRA (
> > > >>> apache.org)<
> > >
> >
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%2525&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KORvfBkmdpqIFOoWr6J4X%2BqAQO6jcykzjY3%2FU0pq0y4%3D&reserved=0
> > > >>> 3A%2F%2Fissues.apache.org
> > %2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7C
> > > >>> Sankar.Hariappan%40microsoft.com
> > %7C687a6a4dbd41454568e008db14d8cc23%7C
> > > >>>
> > 72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknow
> > > >>>
> > n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> > > >>>
> > JXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RbAqrwK6fQFDStufXYfpusNc81EzjtpiaHm
> > > >>> qv5CFiAs%3D&reserved=0> and [HIVE-26940] Backport of HIVE-19882 :
> Fix
> > > >>> QTestUtil session lifecycle - ASF JIRA
> > > >>> (apache.org)<
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttps%2F&data=05%7C01%7Crajaman%40microsoft.com%7C7342dd78ed7f424b36a608db16495672%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638128276496747003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rokHWpxWQ0xYFDr%2BC2p%2BFv4gjLtv%2FCXeZGxdcCSFcVM%3D&reserved=0
> > > >>> %3A%2F%2Fissues.apache.org
> > %2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7
> > > >>> CSankar.Hariappan%40microsoft.com
> > %7C687a6a4dbd41454568e008db14d8cc23%7
> > > >>>
> > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnkno
> > > >>>
> > wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> > > >>>
> > CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PaDtEZD569Sv0ER9sC4l6q1ZxyoBeER3zn
> > > >>> Bsc51PWI8%3D&reserved=0>
> > > >>>
> > > >>> Until we have a green branch-3, we cannot go ahead to push new
> > > >>> features for the Hive-3.2.0 release. This is kind of a blocker for
> > this
> > > >> release.
> > > >>> Already bringing the test fixes to the current state took more
> than 2
> > > >>> months.
> > > >>>
> > > >>> I wanted to bring up a proposal to deprecate Hive on Spark from
> > > >>> branch-3 altogether. This would ensure that branch-3 is aligned
> with
> > > >>> the master as done in
> > > >>>
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu%2F&data=05%7C01%7Crajaman%40microsoft.com%7C7342dd78ed7f424b36a608db16495672%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638128276496747003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ajkc6Ev5ts7dQobFo1gwBDRvTt9ggRFTasOz%2B5k3Ocg%3D&reserved=0
> > > >>> es.apache.org
> > > >> %2Fjira%2Fbrowse%2FHIVE-26134&data=05%7C01%7CSankar.Hariappan%
> > > >> 40microsoft.com
> > >
> >
> %7C687a6a4dbd41454568e008db14d8cc23%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j%2F%2BYTakSvfk1Mm%2Fq8kI84gKm6s%2Fg2iA1abIPzY7t%2Bsg%3D&reserved=0.
> > > >> Just wanted to have a vote on this in parallel working on the test
> > > fixes.
> > > >> If we have the approval from the community, we can deprecate it
> > > altogether.
> > > >>>
> > > >>> Please feel free to suggest any concerns or suggestions you have.
> > > >>> Also, I welcome any possible fix suggestion for the test failures.
> > > >>>
> > > >>> Thanks,
> > > >>> Aman.
> > > >>>
> > > >>
> > >
> >
>

Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3

[László Bodor <bo...@gmail.com>]

+1 on Vihang's suggestion
I remember that spark removal was a debated thing even on master, so
completely removing it backwards from a "maintenance" branch-3 line is not
really acceptable (actually, I'm surprised it's not -1ed yet by hive on
spark folks), but it depends on what *deprecation* really means: I mean
disabling some spark tests to stabilize precommit is completely fine in the
absence of community aspiration to fix them properly

regarding the motivation: "This would ensure that branch-3 is aligned with
the master as done in ..."  <-- I don't think we're targeting this, we are
about to make 3.x releases as simply as possible

I'm hoping/assuming that most of the +1s so far are in line with Vihang's
suggestion

vihang karajgaonkar <vi...@apache.org> ezt írta (időpont: 2023. febr.
23., Cs, 16:37):

> +1 to deprecate Hive on Spark.
>
> I feel directly removing it in a minor release is probably a bad idea. Most
> users will upgrade to 3.2 first and go to 4.0 later. If we deprecate it in
> 3.2 it transitions well into its removal as users upgrade to 4.0
> eventually.
>
> If the goal to stabilize the branch-3, we can disable the failing tests on
> Hive on Spark.
>
> Thanks,
> Vihang
>
> On Thu, Feb 23, 2023 at 12:32 AM Alessandro Solimando <
> alessandro.solimando@gmail.com> wrote:
>
> > +1 from me too
> >
> > On Thu, 23 Feb 2023 at 06:09, Ayush Saxena <ay...@gmail.com> wrote:
> >
> > > +1 on removing Hive on Spark from branch-3
> > >
> > > -Ayush
> > >
> > > > On 23-Feb-2023, at 6:40 AM, Wang, Yuming <yu...@ebay.com.invalid>
> > > wrote:
> > > >
> > > > +1.
> > > >
> > > > From: Naresh P R <na...@apache.org>
> > > > Date: Thursday, February 23, 2023 at 02:49
> > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from
> > > branch-3
> > > > External Email
> > > >
> > > > +1 to remove Hive on Spark in branch-3
> > > > ---
> > > > Regards,
> > > > Naresh P R
> > > >
> > > >> On Wed, Feb 22, 2023 at 5:37 AM Sankar Hariappan
> > > >> <Sa...@microsoft.com.invalid> wrote:
> > > >>
> > > >> +1, to remove Hive on Spark in branch-3.
> > > >>
> > > >> Thanks,
> > > >> Sankar
> > > >>
> > > >> -----Original Message-----
> > > >> From: Rajesh Balamohan <rb...@apache.org>
> > > >> Sent: Wednesday, February 22, 2023 6:58 PM
> > > >> To: dev@hive.apache.org
> > > >> Subject: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from
> > > branch-3
> > > >>
> > > >> +1 on removing Hive on Spark in branch-3.
> > > >>
> > > >> It was not done earlier since it was removing a feature in the
> branch.
> > > But
> > > >> if there is enough consensus, we should consider removing it.
> > > >>
> > > >> ~Rajesh.B
> > > >>
> > > >> On Wed, Feb 22, 2023 at 12:48 PM Aman Raj
> > <rajaman@microsoft.com.invalid
> > > >
> > > >> wrote:
> > > >>
> > > >>> Hi team,
> > > >>>
> > > >>> We have been trying to fix Hive on Spark test failures for a long
> > > >>> time. As of now, branch-3 has less than 12 test failures (whose fix
> > > >>> have not been identified). 8 of them are related to Hive on Spark.
> I
> > > >>> had mailed about the failures in my previous mail threads. Thanks
> to
> > > >>> Vihang for working on them as well. But we have not been able to
> > > >> identify the root cause till now.
> > > >>> These fixes can be tracked in the following tickets : [HIVE-27087]
> > Fix
> > > >>> TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF JIRA (
> > > >>> apache.org)<
> > >
> >
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%2525&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KORvfBkmdpqIFOoWr6J4X%2BqAQO6jcykzjY3%2FU0pq0y4%3D&reserved=0
> > > >>> 3A%2F%2Fissues.apache.org
> > %2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7C
> > > >>> Sankar.Hariappan%40microsoft.com
> > %7C687a6a4dbd41454568e008db14d8cc23%7C
> > > >>>
> > 72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknow
> > > >>>
> > n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> > > >>>
> > JXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RbAqrwK6fQFDStufXYfpusNc81EzjtpiaHm
> > > >>> qv5CFiAs%3D&reserved=0> and [HIVE-26940] Backport of HIVE-19882 :
> Fix
> > > >>> QTestUtil session lifecycle - ASF JIRA
> > > >>> (apache.org)<
> > >
> >
> https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttps%2F&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nmosJ%2FQ2JM1UBDuightSWLL9haNQFuc24Zkvo4RnUh4%3D&reserved=0
> > > >>> %3A%2F%2Fissues.apache.org
> > %2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7
> > > >>> CSankar.Hariappan%40microsoft.com
> > %7C687a6a4dbd41454568e008db14d8cc23%7
> > > >>>
> > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnkno
> > > >>>
> > wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> > > >>>
> > CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PaDtEZD569Sv0ER9sC4l6q1ZxyoBeER3zn
> > > >>> Bsc51PWI8%3D&reserved=0>
> > > >>>
> > > >>> Until we have a green branch-3, we cannot go ahead to push new
> > > >>> features for the Hive-3.2.0 release. This is kind of a blocker for
> > this
> > > >> release.
> > > >>> Already bringing the test fixes to the current state took more
> than 2
> > > >>> months.
> > > >>>
> > > >>> I wanted to bring up a proposal to deprecate Hive on Spark from
> > > >>> branch-3 altogether. This would ensure that branch-3 is aligned
> with
> > > >>> the master as done in
> > > >>>
> > >
> >
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu%2F&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=n51cF5fFuwSFFhX%2B0S828W3jYN3G3YwRwJWne1AMGtg%3D&reserved=0
> > > >>> es.apache.org
> > > >> %2Fjira%2Fbrowse%2FHIVE-26134&data=05%7C01%7CSankar.Hariappan%
> > > >> 40microsoft.com
> > >
> >
> %7C687a6a4dbd41454568e008db14d8cc23%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j%2F%2BYTakSvfk1Mm%2Fq8kI84gKm6s%2Fg2iA1abIPzY7t%2Bsg%3D&reserved=0.
> > > >> Just wanted to have a vote on this in parallel working on the test
> > > fixes.
> > > >> If we have the approval from the community, we can deprecate it
> > > altogether.
> > > >>>
> > > >>> Please feel free to suggest any concerns or suggestions you have.
> > > >>> Also, I welcome any possible fix suggestion for the test failures.
> > > >>>
> > > >>> Thanks,
> > > >>> Aman.
> > > >>>
> > > >>
> > >
> >
>

Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3

[vihang karajgaonkar <vi...@apache.org>]

+1 to deprecate Hive on Spark.

I feel directly removing it in a minor release is probably a bad idea. Most
users will upgrade to 3.2 first and go to 4.0 later. If we deprecate it in
3.2 it transitions well into its removal as users upgrade to 4.0 eventually.

If the goal to stabilize the branch-3, we can disable the failing tests on
Hive on Spark.

Thanks,
Vihang

On Thu, Feb 23, 2023 at 12:32 AM Alessandro Solimando <
alessandro.solimando@gmail.com> wrote:

> +1 from me too
>
> On Thu, 23 Feb 2023 at 06:09, Ayush Saxena <ay...@gmail.com> wrote:
>
> > +1 on removing Hive on Spark from branch-3
> >
> > -Ayush
> >
> > > On 23-Feb-2023, at 6:40 AM, Wang, Yuming <yu...@ebay.com.invalid>
> > wrote:
> > >
> > > +1.
> > >
> > > From: Naresh P R <na...@apache.org>
> > > Date: Thursday, February 23, 2023 at 02:49
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from
> > branch-3
> > > External Email
> > >
> > > +1 to remove Hive on Spark in branch-3
> > > ---
> > > Regards,
> > > Naresh P R
> > >
> > >> On Wed, Feb 22, 2023 at 5:37 AM Sankar Hariappan
> > >> <Sa...@microsoft.com.invalid> wrote:
> > >>
> > >> +1, to remove Hive on Spark in branch-3.
> > >>
> > >> Thanks,
> > >> Sankar
> > >>
> > >> -----Original Message-----
> > >> From: Rajesh Balamohan <rb...@apache.org>
> > >> Sent: Wednesday, February 22, 2023 6:58 PM
> > >> To: dev@hive.apache.org
> > >> Subject: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from
> > branch-3
> > >>
> > >> +1 on removing Hive on Spark in branch-3.
> > >>
> > >> It was not done earlier since it was removing a feature in the branch.
> > But
> > >> if there is enough consensus, we should consider removing it.
> > >>
> > >> ~Rajesh.B
> > >>
> > >> On Wed, Feb 22, 2023 at 12:48 PM Aman Raj
> <rajaman@microsoft.com.invalid
> > >
> > >> wrote:
> > >>
> > >>> Hi team,
> > >>>
> > >>> We have been trying to fix Hive on Spark test failures for a long
> > >>> time. As of now, branch-3 has less than 12 test failures (whose fix
> > >>> have not been identified). 8 of them are related to Hive on Spark. I
> > >>> had mailed about the failures in my previous mail threads. Thanks to
> > >>> Vihang for working on them as well. But we have not been able to
> > >> identify the root cause till now.
> > >>> These fixes can be tracked in the following tickets : [HIVE-27087]
> Fix
> > >>> TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF JIRA (
> > >>> apache.org)<
> >
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%2525&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KORvfBkmdpqIFOoWr6J4X%2BqAQO6jcykzjY3%2FU0pq0y4%3D&reserved=0
> > >>> 3A%2F%2Fissues.apache.org
> %2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7C
> > >>> Sankar.Hariappan%40microsoft.com
> %7C687a6a4dbd41454568e008db14d8cc23%7C
> > >>>
> 72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknow
> > >>>
> n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> > >>>
> JXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RbAqrwK6fQFDStufXYfpusNc81EzjtpiaHm
> > >>> qv5CFiAs%3D&reserved=0> and [HIVE-26940] Backport of HIVE-19882 : Fix
> > >>> QTestUtil session lifecycle - ASF JIRA
> > >>> (apache.org)<
> >
> https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttps%2F&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nmosJ%2FQ2JM1UBDuightSWLL9haNQFuc24Zkvo4RnUh4%3D&reserved=0
> > >>> %3A%2F%2Fissues.apache.org
> %2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7
> > >>> CSankar.Hariappan%40microsoft.com
> %7C687a6a4dbd41454568e008db14d8cc23%7
> > >>>
> C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnkno
> > >>>
> wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> > >>>
> CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PaDtEZD569Sv0ER9sC4l6q1ZxyoBeER3zn
> > >>> Bsc51PWI8%3D&reserved=0>
> > >>>
> > >>> Until we have a green branch-3, we cannot go ahead to push new
> > >>> features for the Hive-3.2.0 release. This is kind of a blocker for
> this
> > >> release.
> > >>> Already bringing the test fixes to the current state took more than 2
> > >>> months.
> > >>>
> > >>> I wanted to bring up a proposal to deprecate Hive on Spark from
> > >>> branch-3 altogether. This would ensure that branch-3 is aligned with
> > >>> the master as done in
> > >>>
> >
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu%2F&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=n51cF5fFuwSFFhX%2B0S828W3jYN3G3YwRwJWne1AMGtg%3D&reserved=0
> > >>> es.apache.org
> > >> %2Fjira%2Fbrowse%2FHIVE-26134&data=05%7C01%7CSankar.Hariappan%
> > >> 40microsoft.com
> >
> %7C687a6a4dbd41454568e008db14d8cc23%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j%2F%2BYTakSvfk1Mm%2Fq8kI84gKm6s%2Fg2iA1abIPzY7t%2Bsg%3D&reserved=0.
> > >> Just wanted to have a vote on this in parallel working on the test
> > fixes.
> > >> If we have the approval from the community, we can deprecate it
> > altogether.
> > >>>
> > >>> Please feel free to suggest any concerns or suggestions you have.
> > >>> Also, I welcome any possible fix suggestion for the test failures.
> > >>>
> > >>> Thanks,
> > >>> Aman.
> > >>>
> > >>
> >
>

Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3

[Alessandro Solimando <al...@gmail.com>]

+1 from me too

On Thu, 23 Feb 2023 at 06:09, Ayush Saxena <ay...@gmail.com> wrote:

> +1 on removing Hive on Spark from branch-3
>
> -Ayush
>
> > On 23-Feb-2023, at 6:40 AM, Wang, Yuming <yu...@ebay.com.invalid>
> wrote:
> >
> > +1.
> >
> > From: Naresh P R <na...@apache.org>
> > Date: Thursday, February 23, 2023 at 02:49
> > To: dev@hive.apache.org <de...@hive.apache.org>
> > Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from
> branch-3
> > External Email
> >
> > +1 to remove Hive on Spark in branch-3
> > ---
> > Regards,
> > Naresh P R
> >
> >> On Wed, Feb 22, 2023 at 5:37 AM Sankar Hariappan
> >> <Sa...@microsoft.com.invalid> wrote:
> >>
> >> +1, to remove Hive on Spark in branch-3.
> >>
> >> Thanks,
> >> Sankar
> >>
> >> -----Original Message-----
> >> From: Rajesh Balamohan <rb...@apache.org>
> >> Sent: Wednesday, February 22, 2023 6:58 PM
> >> To: dev@hive.apache.org
> >> Subject: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from
> branch-3
> >>
> >> +1 on removing Hive on Spark in branch-3.
> >>
> >> It was not done earlier since it was removing a feature in the branch.
> But
> >> if there is enough consensus, we should consider removing it.
> >>
> >> ~Rajesh.B
> >>
> >> On Wed, Feb 22, 2023 at 12:48 PM Aman Raj <rajaman@microsoft.com.invalid
> >
> >> wrote:
> >>
> >>> Hi team,
> >>>
> >>> We have been trying to fix Hive on Spark test failures for a long
> >>> time. As of now, branch-3 has less than 12 test failures (whose fix
> >>> have not been identified). 8 of them are related to Hive on Spark. I
> >>> had mailed about the failures in my previous mail threads. Thanks to
> >>> Vihang for working on them as well. But we have not been able to
> >> identify the root cause till now.
> >>> These fixes can be tracked in the following tickets : [HIVE-27087] Fix
> >>> TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF JIRA (
> >>> apache.org)<
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%2525&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KORvfBkmdpqIFOoWr6J4X%2BqAQO6jcykzjY3%2FU0pq0y4%3D&reserved=0
> >>> 3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7C
> >>> Sankar.Hariappan%40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7C
> >>> 72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknow
> >>> n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> >>> JXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RbAqrwK6fQFDStufXYfpusNc81EzjtpiaHm
> >>> qv5CFiAs%3D&reserved=0> and [HIVE-26940] Backport of HIVE-19882 : Fix
> >>> QTestUtil session lifecycle - ASF JIRA
> >>> (apache.org)<
> https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttps%2F&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nmosJ%2FQ2JM1UBDuightSWLL9haNQFuc24Zkvo4RnUh4%3D&reserved=0
> >>> %3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7
> >>> CSankar.Hariappan%40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7
> >>> C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnkno
> >>> wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> >>> CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PaDtEZD569Sv0ER9sC4l6q1ZxyoBeER3zn
> >>> Bsc51PWI8%3D&reserved=0>
> >>>
> >>> Until we have a green branch-3, we cannot go ahead to push new
> >>> features for the Hive-3.2.0 release. This is kind of a blocker for this
> >> release.
> >>> Already bringing the test fixes to the current state took more than 2
> >>> months.
> >>>
> >>> I wanted to bring up a proposal to deprecate Hive on Spark from
> >>> branch-3 altogether. This would ensure that branch-3 is aligned with
> >>> the master as done in
> >>>
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu%2F&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=n51cF5fFuwSFFhX%2B0S828W3jYN3G3YwRwJWne1AMGtg%3D&reserved=0
> >>> es.apache.org
> >> %2Fjira%2Fbrowse%2FHIVE-26134&data=05%7C01%7CSankar.Hariappan%
> >> 40microsoft.com
> %7C687a6a4dbd41454568e008db14d8cc23%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j%2F%2BYTakSvfk1Mm%2Fq8kI84gKm6s%2Fg2iA1abIPzY7t%2Bsg%3D&reserved=0.
> >> Just wanted to have a vote on this in parallel working on the test
> fixes.
> >> If we have the approval from the community, we can deprecate it
> altogether.
> >>>
> >>> Please feel free to suggest any concerns or suggestions you have.
> >>> Also, I welcome any possible fix suggestion for the test failures.
> >>>
> >>> Thanks,
> >>> Aman.
> >>>
> >>
>

Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3

[Ayush Saxena <ay...@gmail.com>]

+1 on removing Hive on Spark from branch-3

-Ayush

> On 23-Feb-2023, at 6:40 AM, Wang, Yuming <yu...@ebay.com.invalid> wrote:
> 
> +1.
> 
> From: Naresh P R <na...@apache.org>
> Date: Thursday, February 23, 2023 at 02:49
> To: dev@hive.apache.org <de...@hive.apache.org>
> Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3
> External Email
> 
> +1 to remove Hive on Spark in branch-3
> ---
> Regards,
> Naresh P R
> 
>> On Wed, Feb 22, 2023 at 5:37 AM Sankar Hariappan
>> <Sa...@microsoft.com.invalid> wrote:
>> 
>> +1, to remove Hive on Spark in branch-3.
>> 
>> Thanks,
>> Sankar
>> 
>> -----Original Message-----
>> From: Rajesh Balamohan <rb...@apache.org>
>> Sent: Wednesday, February 22, 2023 6:58 PM
>> To: dev@hive.apache.org
>> Subject: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3
>> 
>> +1 on removing Hive on Spark in branch-3.
>> 
>> It was not done earlier since it was removing a feature in the branch. But
>> if there is enough consensus, we should consider removing it.
>> 
>> ~Rajesh.B
>> 
>> On Wed, Feb 22, 2023 at 12:48 PM Aman Raj <ra...@microsoft.com.invalid>
>> wrote:
>> 
>>> Hi team,
>>> 
>>> We have been trying to fix Hive on Spark test failures for a long
>>> time. As of now, branch-3 has less than 12 test failures (whose fix
>>> have not been identified). 8 of them are related to Hive on Spark. I
>>> had mailed about the failures in my previous mail threads. Thanks to
>>> Vihang for working on them as well. But we have not been able to
>> identify the root cause till now.
>>> These fixes can be tracked in the following tickets : [HIVE-27087] Fix
>>> TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF JIRA (
>>> apache.org)<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%2525&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KORvfBkmdpqIFOoWr6J4X%2BqAQO6jcykzjY3%2FU0pq0y4%3D&reserved=0
>>> 3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7C
>>> Sankar.Hariappan%40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7C
>>> 72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknow
>>> n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
>>> JXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RbAqrwK6fQFDStufXYfpusNc81EzjtpiaHm
>>> qv5CFiAs%3D&reserved=0> and [HIVE-26940] Backport of HIVE-19882 : Fix
>>> QTestUtil session lifecycle - ASF JIRA
>>> (apache.org)<https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttps%2F&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nmosJ%2FQ2JM1UBDuightSWLL9haNQFuc24Zkvo4RnUh4%3D&reserved=0
>>> %3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7
>>> CSankar.Hariappan%40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7
>>> C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnkno
>>> wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
>>> CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PaDtEZD569Sv0ER9sC4l6q1ZxyoBeER3zn
>>> Bsc51PWI8%3D&reserved=0>
>>> 
>>> Until we have a green branch-3, we cannot go ahead to push new
>>> features for the Hive-3.2.0 release. This is kind of a blocker for this
>> release.
>>> Already bringing the test fixes to the current state took more than 2
>>> months.
>>> 
>>> I wanted to bring up a proposal to deprecate Hive on Spark from
>>> branch-3 altogether. This would ensure that branch-3 is aligned with
>>> the master as done in
>>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu%2F&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=n51cF5fFuwSFFhX%2B0S828W3jYN3G3YwRwJWne1AMGtg%3D&reserved=0
>>> es.apache.org
>> %2Fjira%2Fbrowse%2FHIVE-26134&data=05%7C01%7CSankar.Hariappan%
>> 40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j%2F%2BYTakSvfk1Mm%2Fq8kI84gKm6s%2Fg2iA1abIPzY7t%2Bsg%3D&reserved=0.
>> Just wanted to have a vote on this in parallel working on the test fixes.
>> If we have the approval from the community, we can deprecate it altogether.
>>> 
>>> Please feel free to suggest any concerns or suggestions you have.
>>> Also, I welcome any possible fix suggestion for the test failures.
>>> 
>>> Thanks,
>>> Aman.
>>> 
>> 

Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3

["Wang, Yuming" <yu...@ebay.com.INVALID>]

+1.

From: Naresh P R <na...@apache.org>
Date: Thursday, February 23, 2023 at 02:49
To: dev@hive.apache.org <de...@hive.apache.org>
Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3
External Email

+1 to remove Hive on Spark in branch-3
---
Regards,
Naresh P R

On Wed, Feb 22, 2023 at 5:37 AM Sankar Hariappan
<Sa...@microsoft.com.invalid> wrote:

> +1, to remove Hive on Spark in branch-3.
>
> Thanks,
> Sankar
>
> -----Original Message-----
> From: Rajesh Balamohan <rb...@apache.org>
> Sent: Wednesday, February 22, 2023 6:58 PM
> To: dev@hive.apache.org
> Subject: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3
>
> +1 on removing Hive on Spark in branch-3.
>
> It was not done earlier since it was removing a feature in the branch. But
> if there is enough consensus, we should consider removing it.
>
> ~Rajesh.B
>
> On Wed, Feb 22, 2023 at 12:48 PM Aman Raj <ra...@microsoft.com.invalid>
> wrote:
>
> > Hi team,
> >
> > We have been trying to fix Hive on Spark test failures for a long
> > time. As of now, branch-3 has less than 12 test failures (whose fix
> > have not been identified). 8 of them are related to Hive on Spark. I
> > had mailed about the failures in my previous mail threads. Thanks to
> > Vihang for working on them as well. But we have not been able to
> identify the root cause till now.
> > These fixes can be tracked in the following tickets : [HIVE-27087] Fix
> > TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF JIRA (
> > apache.org)<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%2525&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KORvfBkmdpqIFOoWr6J4X%2BqAQO6jcykzjY3%2FU0pq0y4%3D&reserved=0
> > 3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7C
> > Sankar.Hariappan%40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7C
> > 72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknow
> > n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> > JXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RbAqrwK6fQFDStufXYfpusNc81EzjtpiaHm
> > qv5CFiAs%3D&reserved=0> and [HIVE-26940] Backport of HIVE-19882 : Fix
> > QTestUtil session lifecycle - ASF JIRA
> > (apache.org)<https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttps%2F&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nmosJ%2FQ2JM1UBDuightSWLL9haNQFuc24Zkvo4RnUh4%3D&reserved=0
> > %3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7
> > CSankar.Hariappan%40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7
> > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnkno
> > wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> > CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PaDtEZD569Sv0ER9sC4l6q1ZxyoBeER3zn
> > Bsc51PWI8%3D&reserved=0>
> >
> > Until we have a green branch-3, we cannot go ahead to push new
> > features for the Hive-3.2.0 release. This is kind of a blocker for this
> release.
> > Already bringing the test fixes to the current state took more than 2
> > months.
> >
> > I wanted to bring up a proposal to deprecate Hive on Spark from
> > branch-3 altogether. This would ensure that branch-3 is aligned with
> > the master as done in
> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu%2F&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=n51cF5fFuwSFFhX%2B0S828W3jYN3G3YwRwJWne1AMGtg%3D&reserved=0
> > es.apache.org
> %2Fjira%2Fbrowse%2FHIVE-26134&data=05%7C01%7CSankar.Hariappan%
> 40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j%2F%2BYTakSvfk1Mm%2Fq8kI84gKm6s%2Fg2iA1abIPzY7t%2Bsg%3D&reserved=0.
> Just wanted to have a vote on this in parallel working on the test fixes.
> If we have the approval from the community, we can deprecate it altogether.
> >
> > Please feel free to suggest any concerns or suggestions you have.
> > Also, I welcome any possible fix suggestion for the test failures.
> >
> > Thanks,
> > Aman.
> >
>

Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3

[Naresh P R <na...@apache.org>]

+1 to remove Hive on Spark in branch-3
---
Regards,
Naresh P R

On Wed, Feb 22, 2023 at 5:37 AM Sankar Hariappan
<Sa...@microsoft.com.invalid> wrote:

> +1, to remove Hive on Spark in branch-3.
>
> Thanks,
> Sankar
>
> -----Original Message-----
> From: Rajesh Balamohan <rb...@apache.org>
> Sent: Wednesday, February 22, 2023 6:58 PM
> To: dev@hive.apache.org
> Subject: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3
>
> +1 on removing Hive on Spark in branch-3.
>
> It was not done earlier since it was removing a feature in the branch. But
> if there is enough consensus, we should consider removing it.
>
> ~Rajesh.B
>
> On Wed, Feb 22, 2023 at 12:48 PM Aman Raj <ra...@microsoft.com.invalid>
> wrote:
>
> > Hi team,
> >
> > We have been trying to fix Hive on Spark test failures for a long
> > time. As of now, branch-3 has less than 12 test failures (whose fix
> > have not been identified). 8 of them are related to Hive on Spark. I
> > had mailed about the failures in my previous mail threads. Thanks to
> > Vihang for working on them as well. But we have not been able to
> identify the root cause till now.
> > These fixes can be tracked in the following tickets : [HIVE-27087] Fix
> > TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF JIRA (
> > apache.org)<https://nam06.safelinks.protection.outlook.com/?url=https%
> > 3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7C
> > Sankar.Hariappan%40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7C
> > 72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknow
> > n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> > JXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RbAqrwK6fQFDStufXYfpusNc81EzjtpiaHm
> > qv5CFiAs%3D&reserved=0> and [HIVE-26940] Backport of HIVE-19882 : Fix
> > QTestUtil session lifecycle - ASF JIRA
> > (apache.org)<https://nam06.safelinks.protection.outlook.com/?url=https
> > %3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7
> > CSankar.Hariappan%40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7
> > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnkno
> > wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> > CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PaDtEZD569Sv0ER9sC4l6q1ZxyoBeER3zn
> > Bsc51PWI8%3D&reserved=0>
> >
> > Until we have a green branch-3, we cannot go ahead to push new
> > features for the Hive-3.2.0 release. This is kind of a blocker for this
> release.
> > Already bringing the test fixes to the current state took more than 2
> > months.
> >
> > I wanted to bring up a proposal to deprecate Hive on Spark from
> > branch-3 altogether. This would ensure that branch-3 is aligned with
> > the master as done in
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu
> > es.apache.org
> %2Fjira%2Fbrowse%2FHIVE-26134&data=05%7C01%7CSankar.Hariappan%
> 40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j%2F%2BYTakSvfk1Mm%2Fq8kI84gKm6s%2Fg2iA1abIPzY7t%2Bsg%3D&reserved=0.
> Just wanted to have a vote on this in parallel working on the test fixes.
> If we have the approval from the community, we can deprecate it altogether.
> >
> > Please feel free to suggest any concerns or suggestions you have.
> > Also, I welcome any possible fix suggestion for the test failures.
> >
> > Thanks,
> > Aman.
> >
>

RE: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3

[Sankar Hariappan <Sa...@microsoft.com.INVALID>]

+1, to remove Hive on Spark in branch-3.

Thanks,
Sankar

-----Original Message-----
From: Rajesh Balamohan <rb...@apache.org> 
Sent: Wednesday, February 22, 2023 6:58 PM
To: dev@hive.apache.org
Subject: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3

+1 on removing Hive on Spark in branch-3.

It was not done earlier since it was removing a feature in the branch. But if there is enough consensus, we should consider removing it.

~Rajesh.B

On Wed, Feb 22, 2023 at 12:48 PM Aman Raj <ra...@microsoft.com.invalid>
wrote:

> Hi team,
>
> We have been trying to fix Hive on Spark test failures for a long 
> time. As of now, branch-3 has less than 12 test failures (whose fix 
> have not been identified). 8 of them are related to Hive on Spark. I 
> had mailed about the failures in my previous mail threads. Thanks to 
> Vihang for working on them as well. But we have not been able to identify the root cause till now.
> These fixes can be tracked in the following tickets : [HIVE-27087] Fix 
> TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF JIRA ( 
> apache.org)<https://nam06.safelinks.protection.outlook.com/?url=https%
> 3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7C
> Sankar.Hariappan%40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7C
> 72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknow
> n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> JXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RbAqrwK6fQFDStufXYfpusNc81EzjtpiaHm
> qv5CFiAs%3D&reserved=0> and [HIVE-26940] Backport of HIVE-19882 : Fix 
> QTestUtil session lifecycle - ASF JIRA 
> (apache.org)<https://nam06.safelinks.protection.outlook.com/?url=https
> %3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7
> CSankar.Hariappan%40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7
> C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnkno
> wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PaDtEZD569Sv0ER9sC4l6q1ZxyoBeER3zn
> Bsc51PWI8%3D&reserved=0>
>
> Until we have a green branch-3, we cannot go ahead to push new 
> features for the Hive-3.2.0 release. This is kind of a blocker for this release.
> Already bringing the test fixes to the current state took more than 2 
> months.
>
> I wanted to bring up a proposal to deprecate Hive on Spark from 
> branch-3 altogether. This would ensure that branch-3 is aligned with 
> the master as done in 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu
> es.apache.org%2Fjira%2Fbrowse%2FHIVE-26134&data=05%7C01%7CSankar.Hariappan%40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j%2F%2BYTakSvfk1Mm%2Fq8kI84gKm6s%2Fg2iA1abIPzY7t%2Bsg%3D&reserved=0. Just wanted to have a vote on this in parallel working on the test fixes. If we have the approval from the community, we can deprecate it altogether.
>
> Please feel free to suggest any concerns or suggestions you have. 
> Also, I welcome any possible fix suggestion for the test failures.
>
> Thanks,
> Aman.
>

Re: Proposal to deprecate Hive on Spark from branch-3

[Rajesh Balamohan <rb...@apache.org>]

+1 on removing Hive on Spark in branch-3.

It was not done earlier since it was removing a feature in the branch. But
if there is enough consensus, we should consider removing it.

~Rajesh.B

On Wed, Feb 22, 2023 at 12:48 PM Aman Raj <ra...@microsoft.com.invalid>
wrote:

> Hi team,
>
> We have been trying to fix Hive on Spark test failures for a long time. As
> of now, branch-3 has less than 12 test failures (whose fix have not been
> identified). 8 of them are related to Hive on Spark. I had mailed about the
> failures in my previous mail threads. Thanks to Vihang for working on them
> as well. But we have not been able to identify the root cause till now.
> These fixes can be tracked in the following tickets : [HIVE-27087] Fix
> TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF JIRA (
> apache.org)<https://issues.apache.org/jira/browse/HIVE-27087> and
> [HIVE-26940] Backport of HIVE-19882 : Fix QTestUtil session lifecycle - ASF
> JIRA (apache.org)<https://issues.apache.org/jira/browse/HIVE-26940>
>
> Until we have a green branch-3, we cannot go ahead to push new features
> for the Hive-3.2.0 release. This is kind of a blocker for this release.
> Already bringing the test fixes to the current state took more than 2
> months.
>
> I wanted to bring up a proposal to deprecate Hive on Spark from branch-3
> altogether. This would ensure that branch-3 is aligned with the master as
> done in https://issues.apache.org/jira/browse/HIVE-26134. Just wanted to
> have a vote on this in parallel working on the test fixes. If we have the
> approval from the community, we can deprecate it altogether.
>
> Please feel free to suggest any concerns or suggestions you have. Also, I
> welcome any possible fix suggestion for the test failures.
>
> Thanks,
> Aman.
>