Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3

[Ayush Saxena <ay...@gmail.com>]

Regarding deprecation, I think the general meaning is "this will be removed
in the future releases", it nowhere means it is broken or has issues. I
don't think if we are deprecating it, we can just ignore the failures if
they are indeed bugs or issues with the prod code. We shouldn't release
broken code under the label: "It is deprecated"

3.2.0 indeed is a minor release, which as Stamatis mentioned now might
already have some breaking changes, that is my guess as well, the way
thirdparty libs are being upgraded, a bunch of them are well known for poor
backward compatibility. And these library compatibility usually is one of
the core reasons which pushes other projects from upgrading...

So, If we are taking it that way, we should avoid any such breaking changes
which aren't in scope of a minor release. Don't pull in any major features
or any change which have major user level impacts and beyond the scope of a
minor release.

BTW. A fix even if committed to the branch, if it isn't in line with the
scope of a "maintenance release" can not be considered "accepted". It can
be vetoed during the release vote and will push everything back, So, either
have a prior agreement or not push any breaking change via hive code or
coming via any thirdparty lib upgrades. It is just gonna create troubles
and might fetch vetoes during the release vote, just my 2 cents...

-Ayush

On Tue, 28 Feb 2023 at 04:08, vihang karajgaonkar <vi...@apache.org>
wrote:

> I think 3.2.0 is seen as a minor release, not a maintenance release. Eg.
> 3.2.1 would be a maintenance release which typically includes bug fixes,
> minor usability improvements and security fixes. If we block new features
> from going into minor releases that would be a step back in my opinion.
> I am pretty sure we have released small features in minor releases before
> (e.g 2.2.0 release notes
> <
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12335837&styleName=Text&projectId=12310843
> >).
> I would request not to club new features and breaking features together in
> the same decision.
>
> For breaking changes, I don't think we have published guidelines before so
> I am open to that. Especially for the HoS tests which are failing, some of
> the failures are related to dependency upgrades (done for security reasons)
> while some have no clear root-cause yet. For a feature which has been
> removed in master branch, I think it is okay to deprecate it and disable
> the tests given that we have spent considerable time on them already and
> none of the original HoS contributors are available to help.
>
> Thanks,
> Vihang
>
> On Mon, Feb 27, 2023 at 1:24 PM Stamatis Zampetakis <za...@gmail.com>
> wrote:
>
> > Some people raised a valid point that branch-3 is a maintenance branch.
> If
> > we really aim 3.2.0 to be a maintenance release then we should minimize
> > breaking changes and prohibit new features. In this case Spark cannot go
> > away and the only thing we can do is deprecate it. It also means that we
> > should fix the tests cause failures typically indicate breaking changes
> > which again are not tolerable for a maintenance release.
> >
> > On the other hand,
> > I got the impression that some people were interested for getting new
> > features in 3.2.0 (  some may be in already). Furthermore, some
> dependency
> > upgrades may also lead to breaking changes/different behavior so we
> should
> > definitely agree on what is acceptable and what is not for branch-3.
> >
> > Summing up the question boils down to the following. Do we allow breaking
> > changes and new features in branch-3 or not?
> >
> > Best,
> > Stamatis
> >
> > On Fri, Feb 24, 2023, 10:41 AM Aman Raj <ra...@microsoft.com.invalid>
> > wrote:
> >
> > > Hi Laszlo,
> > >
> > > I am perfectly fine with disabling the Hive on Spark tests. In fact, I
> > > prefer that. I agree with Vihang and you on this. I had proposed this
> > idea
> > > long back (of disabling the test cases) and then we had discussed on
> the
> > > community that either we fix the Hive on Spark test cases or remove
> Hive
> > on
> > > Spark. Therefore, I initiated this thread of removing Hive on Spark
> since
> > > we are not still able to resolve the test cases since the past couple
> of
> > > months.
> > >
> > > Thanks,
> > > Aman.
> > >
> > > ________________________________
> > > From: László Bodor <bo...@gmail.com>
> > > Sent: Friday, February 24, 2023 2:57 PM
> > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from
> > > branch-3
> > >
> > > +1 on Vihang's suggestion
> > > I remember that spark removal was a debated thing even on master, so
> > > completely removing it backwards from a "maintenance" branch-3 line is
> > not
> > > really acceptable (actually, I'm surprised it's not -1ed yet by hive on
> > > spark folks), but it depends on what *deprecation* really means: I mean
> > > disabling some spark tests to stabilize precommit is completely fine in
> > the
> > > absence of community aspiration to fix them properly
> > >
> > > regarding the motivation: "This would ensure that branch-3 is aligned
> > with
> > > the master as done in ..."  <-- I don't think we're targeting this, we
> > are
> > > about to make 3.x releases as simply as possible
> > >
> > > I'm hoping/assuming that most of the +1s so far are in line with
> Vihang's
> > > suggestion
> > >
> > > vihang karajgaonkar <vi...@apache.org> ezt írta (időpont: 2023.
> febr.
> > > 23., Cs, 16:37):
> > >
> > > > +1 to deprecate Hive on Spark.
> > > >
> > > > I feel directly removing it in a minor release is probably a bad
> idea.
> > > Most
> > > > users will upgrade to 3.2 first and go to 4.0 later. If we deprecate
> it
> > > in
> > > > 3.2 it transitions well into its removal as users upgrade to 4.0
> > > > eventually.
> > > >
> > > > If the goal to stabilize the branch-3, we can disable the failing
> tests
> > > on
> > > > Hive on Spark.
> > > >
> > > > Thanks,
> > > > Vihang
> > > >
> > > > On Thu, Feb 23, 2023 at 12:32 AM Alessandro Solimando <
> > > > alessandro.solimando@gmail.com> wrote:
> > > >
> > > > > +1 from me too
> > > > >
> > > > > On Thu, 23 Feb 2023 at 06:09, Ayush Saxena <ay...@gmail.com>
> > wrote:
> > > > >
> > > > > > +1 on removing Hive on Spark from branch-3
> > > > > >
> > > > > > -Ayush
> > > > > >
> > > > > > > On 23-Feb-2023, at 6:40 AM, Wang, Yuming
> > <yumwang@ebay.com.invalid
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > +1.
> > > > > > >
> > > > > > > From: Naresh P R <na...@apache.org>
> > > > > > > Date: Thursday, February 23, 2023 at 02:49
> > > > > > > To: dev@hive.apache.org <de...@hive.apache.org>
> > > > > > > Subject: Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark
> > > from
> > > > > > branch-3
> > > > > > > External Email
> > > > > > >
> > > > > > > +1 to remove Hive on Spark in branch-3
> > > > > > > ---
> > > > > > > Regards,
> > > > > > > Naresh P R
> > > > > > >
> > > > > > >> On Wed, Feb 22, 2023 at 5:37 AM Sankar Hariappan
> > > > > > >> <Sa...@microsoft.com.invalid> wrote:
> > > > > > >>
> > > > > > >> +1, to remove Hive on Spark in branch-3.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Sankar
> > > > > > >>
> > > > > > >> -----Original Message-----
> > > > > > >> From: Rajesh Balamohan <rb...@apache.org>
> > > > > > >> Sent: Wednesday, February 22, 2023 6:58 PM
> > > > > > >> To: dev@hive.apache.org
> > > > > > >> Subject: [EXTERNAL] Re: Proposal to deprecate Hive on Spark
> from
> > > > > > branch-3
> > > > > > >>
> > > > > > >> +1 on removing Hive on Spark in branch-3.
> > > > > > >>
> > > > > > >> It was not done earlier since it was removing a feature in the
> > > > branch.
> > > > > > But
> > > > > > >> if there is enough consensus, we should consider removing it.
> > > > > > >>
> > > > > > >> ~Rajesh.B
> > > > > > >>
> > > > > > >> On Wed, Feb 22, 2023 at 12:48 PM Aman Raj
> > > > > <rajaman@microsoft.com.invalid
> > > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Hi team,
> > > > > > >>>
> > > > > > >>> We have been trying to fix Hive on Spark test failures for a
> > long
> > > > > > >>> time. As of now, branch-3 has less than 12 test failures
> (whose
> > > fix
> > > > > > >>> have not been identified). 8 of them are related to Hive on
> > > Spark.
> > > > I
> > > > > > >>> had mailed about the failures in my previous mail threads.
> > Thanks
> > > > to
> > > > > > >>> Vihang for working on them as well. But we have not been able
> > to
> > > > > > >> identify the root cause till now.
> > > > > > >>> These fixes can be tracked in the following tickets :
> > > [HIVE-27087]
> > > > > Fix
> > > > > > >>> TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF
> > > JIRA (
> > > > > > >>> apache.org)<
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%2525&data=05%7C01%7Cyumwang%40ebay.com%7C2bd54cc0c84a4e44a59e08db150574e5%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638126885411646147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KORvfBkmdpqIFOoWr6J4X%2BqAQO6jcykzjY3%2FU0pq0y4%3D&reserved=0
> > > > > > >>> 3A%2F%2Fissues.apache.org
> > > > > %2Fjira%2Fbrowse%2FHIVE-27087&data=05%7C01%7C
> > > > > > >>> Sankar.Hariappan%40microsoft.com
> > > > > %7C687a6a4dbd41454568e008db14d8cc23%7C
> > > > > > >>>
> > > > >
> > 72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknow
> > > > > > >>>
> > > > >
> > n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> > > > > > >>>
> > > > >
> > JXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RbAqrwK6fQFDStufXYfpusNc81EzjtpiaHm
> > > > > > >>> qv5CFiAs%3D&reserved=0> and [HIVE-26940] Backport of
> > HIVE-19882 :
> > > > Fix
> > > > > > >>> QTestUtil session lifecycle - ASF JIRA
> > > > > > >>> (apache.org)<
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttps%2F&data=05%7C01%7Crajaman%40microsoft.com%7C7342dd78ed7f424b36a608db16495672%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638128276496747003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rokHWpxWQ0xYFDr%2BC2p%2BFv4gjLtv%2FCXeZGxdcCSFcVM%3D&reserved=0
> > > > > > >>> %3A%2F%2Fissues.apache.org
> > > > > %2Fjira%2Fbrowse%2FHIVE-26940&data=05%7C01%7
> > > > > > >>> CSankar.Hariappan%40microsoft.com
> > > > > %7C687a6a4dbd41454568e008db14d8cc23%7
> > > > > > >>>
> > > > >
> > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnkno
> > > > > > >>>
> > > > >
> > wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> > > > > > >>>
> > > > >
> > CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PaDtEZD569Sv0ER9sC4l6q1ZxyoBeER3zn
> > > > > > >>> Bsc51PWI8%3D&reserved=0>
> > > > > > >>>
> > > > > > >>> Until we have a green branch-3, we cannot go ahead to push
> new
> > > > > > >>> features for the Hive-3.2.0 release. This is kind of a
> blocker
> > > for
> > > > > this
> > > > > > >> release.
> > > > > > >>> Already bringing the test fixes to the current state took
> more
> > > > than 2
> > > > > > >>> months.
> > > > > > >>>
> > > > > > >>> I wanted to bring up a proposal to deprecate Hive on Spark
> from
> > > > > > >>> branch-3 altogether. This would ensure that branch-3 is
> aligned
> > > > with
> > > > > > >>> the master as done in
> > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu%2F&data=05%7C01%7Crajaman%40microsoft.com%7C7342dd78ed7f424b36a608db16495672%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638128276496747003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ajkc6Ev5ts7dQobFo1gwBDRvTt9ggRFTasOz%2B5k3Ocg%3D&reserved=0
> > > > > > >>> es.apache.org
> > > > > > >> %2Fjira%2Fbrowse%2FHIVE-26134&data=05%7C01%7CSankar.Hariappan%
> > > > > > >> 40microsoft.com
> > > > > >
> > > > >
> > > >
> > >
> >
> %7C687a6a4dbd41454568e008db14d8cc23%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j%2F%2BYTakSvfk1Mm%2Fq8kI84gKm6s%2Fg2iA1abIPzY7t%2Bsg%3D&reserved=0.
> > > > > > >> Just wanted to have a vote on this in parallel working on the
> > test
> > > > > > fixes.
> > > > > > >> If we have the approval from the community, we can deprecate
> it
> > > > > > altogether.
> > > > > > >>>
> > > > > > >>> Please feel free to suggest any concerns or suggestions you
> > have.
> > > > > > >>> Also, I welcome any possible fix suggestion for the test
> > > failures.
> > > > > > >>>
> > > > > > >>> Thanks,
> > > > > > >>> Aman.
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>