Re: [DISCUSS] Do we still support hadoop-1.2.x API (-Phadoop-1)?

[Xuefu Zhang <xz...@cloudera.com>]

To close the loop, I have created HIVE-10597 to fix all the compilation
errors on Hadoop 1. Once this is in, @Szehon, could you please include
compilation on Hadoop-1 in precommit test?

Whether or when to remove Hadoop 1 deserves a formal [DISCUSS] thread.

Thanks,
Xuefu

On Thu, Apr 30, 2015 at 12:53 PM, Sergey Shelukhin <se...@hortonworks.com>
wrote:

> That runs the risk of not testing the code being committed against the
> Hadoop version for which it¹s actually going to be used.
> I¹d be +1 on deprecating Hadoop 1 support in a dot version (1.3?). It will
> also provide impetus for people to upgrade.
>
> It will allow us to remove some shims tooŠ
>
>
> On 15/4/29, 12:13, "Xuefu Zhang" <xz...@cloudera.com> wrote:
>
> >I think whether and when to deprecate Hadoop-1 is better to be moved to an
> >official [DISCUSS] thread. What I disliked is the sentiment that I felt
> >when we decide to deprecate something just because it's unfortunately
> >broken for one reason or the other. Nevertheless, we should fix these bugs
> >at this moment. We fixed such issues in previous release as well.
> >
> >I concur that we should have caught these breakages before they made their
> >way into the code. Previously we discussed about running pre-commit tests
> >alternatively or even randomly with either Hadoop-1 or Hadoop-2 profile.
> >Now it seems critical to do so. Thus, I support Szehon's proposal to make
> >this happen. This is orthogonal to the deprecation discussion.
> >
> >Thanks,
> >Xuefu
> >
> >On Wed, Apr 29, 2015 at 11:30 AM, Szehon Ho <sz...@cloudera.com> wrote:
> >
> >> Are you
> >> > personally willing to do the work to make sure that Hive works with
> >> Hadoop
> >> > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443, 10444)
> >> > because
> >> > trunk currently does not work with Hadoop 1.x and no one noticed.
> >>
> >> We setup a Hadoop-1 build sometime back and made sure a lot of breakage
> >> were fixed, but had not hooked it up to HiveQA (precommit) due to
> >>concerns
> >> of the time it would take to run these tests.  It was a balance between
> >> getting faster HiveQA results for everyone and testing all the paths,
> >>even
> >> today there are many paths we don't test.
> >>
> >> So if these JIRA's are fixed, would folks be amenable to alternate
> >>testing
> >> hadoop-1 and hadoop-2 on HiveQA precommit runs and for devs to not
> >>commit
> >> further breakages of Hadoop 1.x compat, for another release?  Yes I do
> >> understand its a burden to maintain hadoop-1.
> >>
> >> if you use HDP, CDH or other hadoop distro then hadoop-1.x and hive-1.2
> >> > combination does not exist.
> >> >
> >>
> >> I think Apache is agnostic to specific vendors so should not discuss
> >>based
> >> on that, but vendors still have this combination, I know of users that
> >>use
> >> Hadoop-1 and latest Hive 1.1.
> >>
> >> Thanks,
> >> Szehon
> >>
> >> On Wed, Apr 29, 2015 at 11:07 AM, Alexander Pivovarov <
> >> apivovarov@gmail.com>
> >> wrote:
> >>
> >> > HDP and CDH distro are based on hadoop-2.x since 2013-2014
> >> >
> >> > if you use HDP, CDH or other hadoop distro then hadoop-1.x and
> >>hive-1.2
> >> > combination does not exist.
> >> > hdp-1.3 has hive-0.11
> >> > cdh-4.7 has hive-0.10
> >> >
> >> > The only way to install hive-1.2 on hadoop-1.x is to do it manually.
> >> > But if you manually install hadoop components then most probably you
> >>will
> >> > install hadoop-2.x
> >> >
> >> > So, we can say that hadoop-1.x and hive-1.2 combination should not
> >>exist
> >> in
> >> > real life scenario.
> >> >
> >> >
> >> >
> >> > On Wed, Apr 29, 2015 at 8:03 AM, Owen O'Malley <om...@apache.org>
> >> wrote:
> >> >
> >> > > On Tue, Apr 28, 2015 at 10:16 PM, Szehon Ho <sz...@cloudera.com>
> >> wrote:
> >> > >
> >> > > > Yes, question is whether users want to run latest Hive version on
> >> > Hadoop
> >> > > > 1.x clusters.
> >> > > >
> >> > >
> >> > > But you ignored the more important question that I asked. Are you
> >> > > personally willing to do the work to make sure that Hive works with
> >> > Hadoop
> >> > > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443,
> >>10444)
> >> > > because
> >> > > trunk currently does not work with Hadoop 1.x and no one noticed.
> >> > >
> >> > > Supporting Hadoop 1.x costs the Hive project a lot of work and
> >>provides
> >> > > very little in return.
> >> > >
> >> > > .. Owen
> >> > >
> >> >
> >>
>
>

Re: [DISCUSS] Do we still support hadoop-1.2.x API (-Phadoop-1)?

[Szehon Ho <sz...@cloudera.com>]

Yes I think its a good compromise to at least add compilation of -Phadoop-1
until we reach consensus on a timeline.

I have actually filed a JIRA for that half a year back, HIVE-7375, I will
take a look.  It was unfortunate this was deprioritzed in favor of running
tests, which was not successful due to the time/resource taken.

Thanks
Szehon

PS all precommit tests are actually broken right now due to the build
machine not being able to checkout from ASF github, I have filed HIVE-10583
to fix it, will send a follow up email on that in another topic once that's
resolved.



On Fri, May 1, 2015 at 5:37 PM, Xuefu Zhang <xz...@cloudera.com> wrote:

> To close the loop, I have created HIVE-10597 to fix all the compilation
> errors on Hadoop 1. Once this is in, @Szehon, could you please include
> compilation on Hadoop-1 in precommit test?
>
> Whether or when to remove Hadoop 1 deserves a formal [DISCUSS] thread.
>
> Thanks,
> Xuefu
>
> On Thu, Apr 30, 2015 at 12:53 PM, Sergey Shelukhin <sergey@hortonworks.com
> >
> wrote:
>
> > That runs the risk of not testing the code being committed against the
> > Hadoop version for which it¹s actually going to be used.
> > I¹d be +1 on deprecating Hadoop 1 support in a dot version (1.3?). It
> will
> > also provide impetus for people to upgrade.
> >
> > It will allow us to remove some shims tooŠ
> >
> >
> > On 15/4/29, 12:13, "Xuefu Zhang" <xz...@cloudera.com> wrote:
> >
> > >I think whether and when to deprecate Hadoop-1 is better to be moved to
> an
> > >official [DISCUSS] thread. What I disliked is the sentiment that I felt
> > >when we decide to deprecate something just because it's unfortunately
> > >broken for one reason or the other. Nevertheless, we should fix these
> bugs
> > >at this moment. We fixed such issues in previous release as well.
> > >
> > >I concur that we should have caught these breakages before they made
> their
> > >way into the code. Previously we discussed about running pre-commit
> tests
> > >alternatively or even randomly with either Hadoop-1 or Hadoop-2 profile.
> > >Now it seems critical to do so. Thus, I support Szehon's proposal to
> make
> > >this happen. This is orthogonal to the deprecation discussion.
> > >
> > >Thanks,
> > >Xuefu
> > >
> > >On Wed, Apr 29, 2015 at 11:30 AM, Szehon Ho <sz...@cloudera.com>
> wrote:
> > >
> > >> Are you
> > >> > personally willing to do the work to make sure that Hive works with
> > >> Hadoop
> > >> > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443,
> 10444)
> > >> > because
> > >> > trunk currently does not work with Hadoop 1.x and no one noticed.
> > >>
> > >> We setup a Hadoop-1 build sometime back and made sure a lot of
> breakage
> > >> were fixed, but had not hooked it up to HiveQA (precommit) due to
> > >>concerns
> > >> of the time it would take to run these tests.  It was a balance
> between
> > >> getting faster HiveQA results for everyone and testing all the paths,
> > >>even
> > >> today there are many paths we don't test.
> > >>
> > >> So if these JIRA's are fixed, would folks be amenable to alternate
> > >>testing
> > >> hadoop-1 and hadoop-2 on HiveQA precommit runs and for devs to not
> > >>commit
> > >> further breakages of Hadoop 1.x compat, for another release?  Yes I do
> > >> understand its a burden to maintain hadoop-1.
> > >>
> > >> if you use HDP, CDH or other hadoop distro then hadoop-1.x and
> hive-1.2
> > >> > combination does not exist.
> > >> >
> > >>
> > >> I think Apache is agnostic to specific vendors so should not discuss
> > >>based
> > >> on that, but vendors still have this combination, I know of users that
> > >>use
> > >> Hadoop-1 and latest Hive 1.1.
> > >>
> > >> Thanks,
> > >> Szehon
> > >>
> > >> On Wed, Apr 29, 2015 at 11:07 AM, Alexander Pivovarov <
> > >> apivovarov@gmail.com>
> > >> wrote:
> > >>
> > >> > HDP and CDH distro are based on hadoop-2.x since 2013-2014
> > >> >
> > >> > if you use HDP, CDH or other hadoop distro then hadoop-1.x and
> > >>hive-1.2
> > >> > combination does not exist.
> > >> > hdp-1.3 has hive-0.11
> > >> > cdh-4.7 has hive-0.10
> > >> >
> > >> > The only way to install hive-1.2 on hadoop-1.x is to do it manually.
> > >> > But if you manually install hadoop components then most probably you
> > >>will
> > >> > install hadoop-2.x
> > >> >
> > >> > So, we can say that hadoop-1.x and hive-1.2 combination should not
> > >>exist
> > >> in
> > >> > real life scenario.
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Apr 29, 2015 at 8:03 AM, Owen O'Malley <om...@apache.org>
> > >> wrote:
> > >> >
> > >> > > On Tue, Apr 28, 2015 at 10:16 PM, Szehon Ho <sz...@cloudera.com>
> > >> wrote:
> > >> > >
> > >> > > > Yes, question is whether users want to run latest Hive version
> on
> > >> > Hadoop
> > >> > > > 1.x clusters.
> > >> > > >
> > >> > >
> > >> > > But you ignored the more important question that I asked. Are you
> > >> > > personally willing to do the work to make sure that Hive works
> with
> > >> > Hadoop
> > >> > > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443,
> > >>10444)
> > >> > > because
> > >> > > trunk currently does not work with Hadoop 1.x and no one noticed.
> > >> > >
> > >> > > Supporting Hadoop 1.x costs the Hive project a lot of work and
> > >>provides
> > >> > > very little in return.
> > >> > >
> > >> > > .. Owen
> > >> > >
> > >> >
> > >>
> >
> >
>