You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Owen O'Malley <om...@apache.org> on 2015/04/29 17:03:42 UTC
Re: [DISCUSS] Do we still support hadoop-1.2.x API (-Phadoop-1)?
On Tue, Apr 28, 2015 at 10:16 PM, Szehon Ho <sz...@cloudera.com> wrote:
> Yes, question is whether users want to run latest Hive version on Hadoop
> 1.x clusters.
>
But you ignored the more important question that I asked. Are you
personally willing to do the work to make sure that Hive works with Hadoop
1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443, 10444) because
trunk currently does not work with Hadoop 1.x and no one noticed.
Supporting Hadoop 1.x costs the Hive project a lot of work and provides
very little in return.
.. Owen
Re: [DISCUSS] Do we still support hadoop-1.2.x API (-Phadoop-1)?
Posted by Szehon Ho <sz...@cloudera.com>.
Yes I think its a good compromise to at least add compilation of -Phadoop-1
until we reach consensus on a timeline.
I have actually filed a JIRA for that half a year back, HIVE-7375, I will
take a look. It was unfortunate this was deprioritzed in favor of running
tests, which was not successful due to the time/resource taken.
Thanks
Szehon
PS all precommit tests are actually broken right now due to the build
machine not being able to checkout from ASF github, I have filed HIVE-10583
to fix it, will send a follow up email on that in another topic once that's
resolved.
On Fri, May 1, 2015 at 5:37 PM, Xuefu Zhang <xz...@cloudera.com> wrote:
> To close the loop, I have created HIVE-10597 to fix all the compilation
> errors on Hadoop 1. Once this is in, @Szehon, could you please include
> compilation on Hadoop-1 in precommit test?
>
> Whether or when to remove Hadoop 1 deserves a formal [DISCUSS] thread.
>
> Thanks,
> Xuefu
>
> On Thu, Apr 30, 2015 at 12:53 PM, Sergey Shelukhin <sergey@hortonworks.com
> >
> wrote:
>
> > That runs the risk of not testing the code being committed against the
> > Hadoop version for which it¹s actually going to be used.
> > I¹d be +1 on deprecating Hadoop 1 support in a dot version (1.3?). It
> will
> > also provide impetus for people to upgrade.
> >
> > It will allow us to remove some shims tooŠ
> >
> >
> > On 15/4/29, 12:13, "Xuefu Zhang" <xz...@cloudera.com> wrote:
> >
> > >I think whether and when to deprecate Hadoop-1 is better to be moved to
> an
> > >official [DISCUSS] thread. What I disliked is the sentiment that I felt
> > >when we decide to deprecate something just because it's unfortunately
> > >broken for one reason or the other. Nevertheless, we should fix these
> bugs
> > >at this moment. We fixed such issues in previous release as well.
> > >
> > >I concur that we should have caught these breakages before they made
> their
> > >way into the code. Previously we discussed about running pre-commit
> tests
> > >alternatively or even randomly with either Hadoop-1 or Hadoop-2 profile.
> > >Now it seems critical to do so. Thus, I support Szehon's proposal to
> make
> > >this happen. This is orthogonal to the deprecation discussion.
> > >
> > >Thanks,
> > >Xuefu
> > >
> > >On Wed, Apr 29, 2015 at 11:30 AM, Szehon Ho <sz...@cloudera.com>
> wrote:
> > >
> > >> Are you
> > >> > personally willing to do the work to make sure that Hive works with
> > >> Hadoop
> > >> > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443,
> 10444)
> > >> > because
> > >> > trunk currently does not work with Hadoop 1.x and no one noticed.
> > >>
> > >> We setup a Hadoop-1 build sometime back and made sure a lot of
> breakage
> > >> were fixed, but had not hooked it up to HiveQA (precommit) due to
> > >>concerns
> > >> of the time it would take to run these tests. It was a balance
> between
> > >> getting faster HiveQA results for everyone and testing all the paths,
> > >>even
> > >> today there are many paths we don't test.
> > >>
> > >> So if these JIRA's are fixed, would folks be amenable to alternate
> > >>testing
> > >> hadoop-1 and hadoop-2 on HiveQA precommit runs and for devs to not
> > >>commit
> > >> further breakages of Hadoop 1.x compat, for another release? Yes I do
> > >> understand its a burden to maintain hadoop-1.
> > >>
> > >> if you use HDP, CDH or other hadoop distro then hadoop-1.x and
> hive-1.2
> > >> > combination does not exist.
> > >> >
> > >>
> > >> I think Apache is agnostic to specific vendors so should not discuss
> > >>based
> > >> on that, but vendors still have this combination, I know of users that
> > >>use
> > >> Hadoop-1 and latest Hive 1.1.
> > >>
> > >> Thanks,
> > >> Szehon
> > >>
> > >> On Wed, Apr 29, 2015 at 11:07 AM, Alexander Pivovarov <
> > >> apivovarov@gmail.com>
> > >> wrote:
> > >>
> > >> > HDP and CDH distro are based on hadoop-2.x since 2013-2014
> > >> >
> > >> > if you use HDP, CDH or other hadoop distro then hadoop-1.x and
> > >>hive-1.2
> > >> > combination does not exist.
> > >> > hdp-1.3 has hive-0.11
> > >> > cdh-4.7 has hive-0.10
> > >> >
> > >> > The only way to install hive-1.2 on hadoop-1.x is to do it manually.
> > >> > But if you manually install hadoop components then most probably you
> > >>will
> > >> > install hadoop-2.x
> > >> >
> > >> > So, we can say that hadoop-1.x and hive-1.2 combination should not
> > >>exist
> > >> in
> > >> > real life scenario.
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Apr 29, 2015 at 8:03 AM, Owen O'Malley <om...@apache.org>
> > >> wrote:
> > >> >
> > >> > > On Tue, Apr 28, 2015 at 10:16 PM, Szehon Ho <sz...@cloudera.com>
> > >> wrote:
> > >> > >
> > >> > > > Yes, question is whether users want to run latest Hive version
> on
> > >> > Hadoop
> > >> > > > 1.x clusters.
> > >> > > >
> > >> > >
> > >> > > But you ignored the more important question that I asked. Are you
> > >> > > personally willing to do the work to make sure that Hive works
> with
> > >> > Hadoop
> > >> > > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443,
> > >>10444)
> > >> > > because
> > >> > > trunk currently does not work with Hadoop 1.x and no one noticed.
> > >> > >
> > >> > > Supporting Hadoop 1.x costs the Hive project a lot of work and
> > >>provides
> > >> > > very little in return.
> > >> > >
> > >> > > .. Owen
> > >> > >
> > >> >
> > >>
> >
> >
>
Re: [DISCUSS] Do we still support hadoop-1.2.x API (-Phadoop-1)?
Posted by Xuefu Zhang <xz...@cloudera.com>.
To close the loop, I have created HIVE-10597 to fix all the compilation
errors on Hadoop 1. Once this is in, @Szehon, could you please include
compilation on Hadoop-1 in precommit test?
Whether or when to remove Hadoop 1 deserves a formal [DISCUSS] thread.
Thanks,
Xuefu
On Thu, Apr 30, 2015 at 12:53 PM, Sergey Shelukhin <se...@hortonworks.com>
wrote:
> That runs the risk of not testing the code being committed against the
> Hadoop version for which it¹s actually going to be used.
> I¹d be +1 on deprecating Hadoop 1 support in a dot version (1.3?). It will
> also provide impetus for people to upgrade.
>
> It will allow us to remove some shims tooŠ
>
>
> On 15/4/29, 12:13, "Xuefu Zhang" <xz...@cloudera.com> wrote:
>
> >I think whether and when to deprecate Hadoop-1 is better to be moved to an
> >official [DISCUSS] thread. What I disliked is the sentiment that I felt
> >when we decide to deprecate something just because it's unfortunately
> >broken for one reason or the other. Nevertheless, we should fix these bugs
> >at this moment. We fixed such issues in previous release as well.
> >
> >I concur that we should have caught these breakages before they made their
> >way into the code. Previously we discussed about running pre-commit tests
> >alternatively or even randomly with either Hadoop-1 or Hadoop-2 profile.
> >Now it seems critical to do so. Thus, I support Szehon's proposal to make
> >this happen. This is orthogonal to the deprecation discussion.
> >
> >Thanks,
> >Xuefu
> >
> >On Wed, Apr 29, 2015 at 11:30 AM, Szehon Ho <sz...@cloudera.com> wrote:
> >
> >> Are you
> >> > personally willing to do the work to make sure that Hive works with
> >> Hadoop
> >> > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443, 10444)
> >> > because
> >> > trunk currently does not work with Hadoop 1.x and no one noticed.
> >>
> >> We setup a Hadoop-1 build sometime back and made sure a lot of breakage
> >> were fixed, but had not hooked it up to HiveQA (precommit) due to
> >>concerns
> >> of the time it would take to run these tests. It was a balance between
> >> getting faster HiveQA results for everyone and testing all the paths,
> >>even
> >> today there are many paths we don't test.
> >>
> >> So if these JIRA's are fixed, would folks be amenable to alternate
> >>testing
> >> hadoop-1 and hadoop-2 on HiveQA precommit runs and for devs to not
> >>commit
> >> further breakages of Hadoop 1.x compat, for another release? Yes I do
> >> understand its a burden to maintain hadoop-1.
> >>
> >> if you use HDP, CDH or other hadoop distro then hadoop-1.x and hive-1.2
> >> > combination does not exist.
> >> >
> >>
> >> I think Apache is agnostic to specific vendors so should not discuss
> >>based
> >> on that, but vendors still have this combination, I know of users that
> >>use
> >> Hadoop-1 and latest Hive 1.1.
> >>
> >> Thanks,
> >> Szehon
> >>
> >> On Wed, Apr 29, 2015 at 11:07 AM, Alexander Pivovarov <
> >> apivovarov@gmail.com>
> >> wrote:
> >>
> >> > HDP and CDH distro are based on hadoop-2.x since 2013-2014
> >> >
> >> > if you use HDP, CDH or other hadoop distro then hadoop-1.x and
> >>hive-1.2
> >> > combination does not exist.
> >> > hdp-1.3 has hive-0.11
> >> > cdh-4.7 has hive-0.10
> >> >
> >> > The only way to install hive-1.2 on hadoop-1.x is to do it manually.
> >> > But if you manually install hadoop components then most probably you
> >>will
> >> > install hadoop-2.x
> >> >
> >> > So, we can say that hadoop-1.x and hive-1.2 combination should not
> >>exist
> >> in
> >> > real life scenario.
> >> >
> >> >
> >> >
> >> > On Wed, Apr 29, 2015 at 8:03 AM, Owen O'Malley <om...@apache.org>
> >> wrote:
> >> >
> >> > > On Tue, Apr 28, 2015 at 10:16 PM, Szehon Ho <sz...@cloudera.com>
> >> wrote:
> >> > >
> >> > > > Yes, question is whether users want to run latest Hive version on
> >> > Hadoop
> >> > > > 1.x clusters.
> >> > > >
> >> > >
> >> > > But you ignored the more important question that I asked. Are you
> >> > > personally willing to do the work to make sure that Hive works with
> >> > Hadoop
> >> > > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443,
> >>10444)
> >> > > because
> >> > > trunk currently does not work with Hadoop 1.x and no one noticed.
> >> > >
> >> > > Supporting Hadoop 1.x costs the Hive project a lot of work and
> >>provides
> >> > > very little in return.
> >> > >
> >> > > .. Owen
> >> > >
> >> >
> >>
>
>
Re: [DISCUSS] Do we still support hadoop-1.2.x API (-Phadoop-1)?
Posted by Sergey Shelukhin <se...@hortonworks.com>.
That runs the risk of not testing the code being committed against the
Hadoop version for which it¹s actually going to be used.
I¹d be +1 on deprecating Hadoop 1 support in a dot version (1.3?). It will
also provide impetus for people to upgrade.
It will allow us to remove some shims tooŠ
On 15/4/29, 12:13, "Xuefu Zhang" <xz...@cloudera.com> wrote:
>I think whether and when to deprecate Hadoop-1 is better to be moved to an
>official [DISCUSS] thread. What I disliked is the sentiment that I felt
>when we decide to deprecate something just because it's unfortunately
>broken for one reason or the other. Nevertheless, we should fix these bugs
>at this moment. We fixed such issues in previous release as well.
>
>I concur that we should have caught these breakages before they made their
>way into the code. Previously we discussed about running pre-commit tests
>alternatively or even randomly with either Hadoop-1 or Hadoop-2 profile.
>Now it seems critical to do so. Thus, I support Szehon's proposal to make
>this happen. This is orthogonal to the deprecation discussion.
>
>Thanks,
>Xuefu
>
>On Wed, Apr 29, 2015 at 11:30 AM, Szehon Ho <sz...@cloudera.com> wrote:
>
>> Are you
>> > personally willing to do the work to make sure that Hive works with
>> Hadoop
>> > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443, 10444)
>> > because
>> > trunk currently does not work with Hadoop 1.x and no one noticed.
>>
>> We setup a Hadoop-1 build sometime back and made sure a lot of breakage
>> were fixed, but had not hooked it up to HiveQA (precommit) due to
>>concerns
>> of the time it would take to run these tests. It was a balance between
>> getting faster HiveQA results for everyone and testing all the paths,
>>even
>> today there are many paths we don't test.
>>
>> So if these JIRA's are fixed, would folks be amenable to alternate
>>testing
>> hadoop-1 and hadoop-2 on HiveQA precommit runs and for devs to not
>>commit
>> further breakages of Hadoop 1.x compat, for another release? Yes I do
>> understand its a burden to maintain hadoop-1.
>>
>> if you use HDP, CDH or other hadoop distro then hadoop-1.x and hive-1.2
>> > combination does not exist.
>> >
>>
>> I think Apache is agnostic to specific vendors so should not discuss
>>based
>> on that, but vendors still have this combination, I know of users that
>>use
>> Hadoop-1 and latest Hive 1.1.
>>
>> Thanks,
>> Szehon
>>
>> On Wed, Apr 29, 2015 at 11:07 AM, Alexander Pivovarov <
>> apivovarov@gmail.com>
>> wrote:
>>
>> > HDP and CDH distro are based on hadoop-2.x since 2013-2014
>> >
>> > if you use HDP, CDH or other hadoop distro then hadoop-1.x and
>>hive-1.2
>> > combination does not exist.
>> > hdp-1.3 has hive-0.11
>> > cdh-4.7 has hive-0.10
>> >
>> > The only way to install hive-1.2 on hadoop-1.x is to do it manually.
>> > But if you manually install hadoop components then most probably you
>>will
>> > install hadoop-2.x
>> >
>> > So, we can say that hadoop-1.x and hive-1.2 combination should not
>>exist
>> in
>> > real life scenario.
>> >
>> >
>> >
>> > On Wed, Apr 29, 2015 at 8:03 AM, Owen O'Malley <om...@apache.org>
>> wrote:
>> >
>> > > On Tue, Apr 28, 2015 at 10:16 PM, Szehon Ho <sz...@cloudera.com>
>> wrote:
>> > >
>> > > > Yes, question is whether users want to run latest Hive version on
>> > Hadoop
>> > > > 1.x clusters.
>> > > >
>> > >
>> > > But you ignored the more important question that I asked. Are you
>> > > personally willing to do the work to make sure that Hive works with
>> > Hadoop
>> > > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443,
>>10444)
>> > > because
>> > > trunk currently does not work with Hadoop 1.x and no one noticed.
>> > >
>> > > Supporting Hadoop 1.x costs the Hive project a lot of work and
>>provides
>> > > very little in return.
>> > >
>> > > .. Owen
>> > >
>> >
>>
Re: [DISCUSS] Do we still support hadoop-1.2.x API (-Phadoop-1)?
Posted by Alexander Pivovarov <ap...@gmail.com>.
Szehon, can we agree that hive-1.2 is the last hive version which supports
hadoop-1.
Starting from hive-1.3 it requires hadoop-2
So, most probably "organizations" will start to install hive-1.2 in the
next 6 months.
Then they will use it for 1-2 years until they want to upgrade to the next
hive version.
So, for the next 2 years "organizations" as good. They can use hive-1.2
with hadoop-1.
But after 2 years (in 2017) they are kinda forced to uprate to hadoop-2 if
they want to use hive-1.3
Alex
On Wed, Apr 29, 2015 at 1:51 PM, Szehon Ho <sz...@cloudera.com> wrote:
> Alex, I'd not confuse these two things (hadoop-1 and HiveCLI), after the
> proposed change from hive script to wrap beeline, if we do the job right
> then user should not notice much difference. There we can eliminate dev
> work while not affecting user. Whereas migration of an organization from
> using Hadoop-1 cluster to Hadoop-2 cluster is a much-bigger change.
>
> Thanks,
> Szehon
>
>
>
> On Wed, Apr 29, 2015 at 1:27 PM, Thejas Nair <th...@gmail.com>
> wrote:
>
> > I agree, the number of users who use Hive 1.1 and newer versions
> > against Hadoop 1.x is likely to very small. (Most likely an orders of
> > magnitude smaller than hive cli users in same versions).
> > The big improvements in Hive 1.x line require Hadoop 2.x. Users who
> > are motivated to upgrade to newer 1.x lines are likely to be using
> > Hadoop 2.x .
> >
> >
> >
> > On Wed, Apr 29, 2015 at 12:41 PM, Alexander Pivovarov
> > <ap...@gmail.com> wrote:
> > > Xuefu,
> > >
> > > Remember when it was beeline discussion you mentioned "Hive community
> > don't
> > > need to maintain two code paths"
> > > Totally agree. Resources are limited.
> > >
> > > I believe the same approach is applicable here.
> > > Lets assume hadoop-1 + hive-1.2 is going to be used by 1% of the users.
> > >
> > > We'd better spend resources on implementing new functionality (useful
> for
> > > 99% of the users) instead of spending resources to support hadoop-1
> > profile
> > > (useful for 1% of the users).
> > >
> > > Alex
> > >
> > >
> > > On Wed, Apr 29, 2015 at 12:13 PM, Xuefu Zhang <xz...@cloudera.com>
> > wrote:
> > >
> > >> I think whether and when to deprecate Hadoop-1 is better to be moved
> to
> > an
> > >> official [DISCUSS] thread. What I disliked is the sentiment that I
> felt
> > >> when we decide to deprecate something just because it's unfortunately
> > >> broken for one reason or the other. Nevertheless, we should fix these
> > bugs
> > >> at this moment. We fixed such issues in previous release as well.
> > >>
> > >> I concur that we should have caught these breakages before they made
> > their
> > >> way into the code. Previously we discussed about running pre-commit
> > tests
> > >> alternatively or even randomly with either Hadoop-1 or Hadoop-2
> profile.
> > >> Now it seems critical to do so. Thus, I support Szehon's proposal to
> > make
> > >> this happen. This is orthogonal to the deprecation discussion.
> > >>
> > >> Thanks,
> > >> Xuefu
> > >>
> > >> On Wed, Apr 29, 2015 at 11:30 AM, Szehon Ho <sz...@cloudera.com>
> > wrote:
> > >>
> > >> > Are you
> > >> > > personally willing to do the work to make sure that Hive works
> with
> > >> > Hadoop
> > >> > > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443,
> > 10444)
> > >> > > because
> > >> > > trunk currently does not work with Hadoop 1.x and no one noticed.
> > >> >
> > >> > We setup a Hadoop-1 build sometime back and made sure a lot of
> > breakage
> > >> > were fixed, but had not hooked it up to HiveQA (precommit) due to
> > >> concerns
> > >> > of the time it would take to run these tests. It was a balance
> > between
> > >> > getting faster HiveQA results for everyone and testing all the
> paths,
> > >> even
> > >> > today there are many paths we don't test.
> > >> >
> > >> > So if these JIRA's are fixed, would folks be amenable to alternate
> > >> testing
> > >> > hadoop-1 and hadoop-2 on HiveQA precommit runs and for devs to not
> > commit
> > >> > further breakages of Hadoop 1.x compat, for another release? Yes I
> do
> > >> > understand its a burden to maintain hadoop-1.
> > >> >
> > >> > if you use HDP, CDH or other hadoop distro then hadoop-1.x and
> > hive-1.2
> > >> > > combination does not exist.
> > >> > >
> > >> >
> > >> > I think Apache is agnostic to specific vendors so should not discuss
> > >> based
> > >> > on that, but vendors still have this combination, I know of users
> that
> > >> use
> > >> > Hadoop-1 and latest Hive 1.1.
> > >> >
> > >> > Thanks,
> > >> > Szehon
> > >> >
> > >> > On Wed, Apr 29, 2015 at 11:07 AM, Alexander Pivovarov <
> > >> > apivovarov@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > HDP and CDH distro are based on hadoop-2.x since 2013-2014
> > >> > >
> > >> > > if you use HDP, CDH or other hadoop distro then hadoop-1.x and
> > hive-1.2
> > >> > > combination does not exist.
> > >> > > hdp-1.3 has hive-0.11
> > >> > > cdh-4.7 has hive-0.10
> > >> > >
> > >> > > The only way to install hive-1.2 on hadoop-1.x is to do it
> manually.
> > >> > > But if you manually install hadoop components then most probably
> you
> > >> will
> > >> > > install hadoop-2.x
> > >> > >
> > >> > > So, we can say that hadoop-1.x and hive-1.2 combination should not
> > >> exist
> > >> > in
> > >> > > real life scenario.
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Wed, Apr 29, 2015 at 8:03 AM, Owen O'Malley <
> omalley@apache.org>
> > >> > wrote:
> > >> > >
> > >> > > > On Tue, Apr 28, 2015 at 10:16 PM, Szehon Ho <
> szehon@cloudera.com>
> > >> > wrote:
> > >> > > >
> > >> > > > > Yes, question is whether users want to run latest Hive version
> > on
> > >> > > Hadoop
> > >> > > > > 1.x clusters.
> > >> > > > >
> > >> > > >
> > >> > > > But you ignored the more important question that I asked. Are
> you
> > >> > > > personally willing to do the work to make sure that Hive works
> > with
> > >> > > Hadoop
> > >> > > > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443,
> > >> 10444)
> > >> > > > because
> > >> > > > trunk currently does not work with Hadoop 1.x and no one
> noticed.
> > >> > > >
> > >> > > > Supporting Hadoop 1.x costs the Hive project a lot of work and
> > >> provides
> > >> > > > very little in return.
> > >> > > >
> > >> > > > .. Owen
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
Re: [DISCUSS] Do we still support hadoop-1.2.x API (-Phadoop-1)?
Posted by Szehon Ho <sz...@cloudera.com>.
Alex, I'd not confuse these two things (hadoop-1 and HiveCLI), after the
proposed change from hive script to wrap beeline, if we do the job right
then user should not notice much difference. There we can eliminate dev
work while not affecting user. Whereas migration of an organization from
using Hadoop-1 cluster to Hadoop-2 cluster is a much-bigger change.
Thanks,
Szehon
On Wed, Apr 29, 2015 at 1:27 PM, Thejas Nair <th...@gmail.com> wrote:
> I agree, the number of users who use Hive 1.1 and newer versions
> against Hadoop 1.x is likely to very small. (Most likely an orders of
> magnitude smaller than hive cli users in same versions).
> The big improvements in Hive 1.x line require Hadoop 2.x. Users who
> are motivated to upgrade to newer 1.x lines are likely to be using
> Hadoop 2.x .
>
>
>
> On Wed, Apr 29, 2015 at 12:41 PM, Alexander Pivovarov
> <ap...@gmail.com> wrote:
> > Xuefu,
> >
> > Remember when it was beeline discussion you mentioned "Hive community
> don't
> > need to maintain two code paths"
> > Totally agree. Resources are limited.
> >
> > I believe the same approach is applicable here.
> > Lets assume hadoop-1 + hive-1.2 is going to be used by 1% of the users.
> >
> > We'd better spend resources on implementing new functionality (useful for
> > 99% of the users) instead of spending resources to support hadoop-1
> profile
> > (useful for 1% of the users).
> >
> > Alex
> >
> >
> > On Wed, Apr 29, 2015 at 12:13 PM, Xuefu Zhang <xz...@cloudera.com>
> wrote:
> >
> >> I think whether and when to deprecate Hadoop-1 is better to be moved to
> an
> >> official [DISCUSS] thread. What I disliked is the sentiment that I felt
> >> when we decide to deprecate something just because it's unfortunately
> >> broken for one reason or the other. Nevertheless, we should fix these
> bugs
> >> at this moment. We fixed such issues in previous release as well.
> >>
> >> I concur that we should have caught these breakages before they made
> their
> >> way into the code. Previously we discussed about running pre-commit
> tests
> >> alternatively or even randomly with either Hadoop-1 or Hadoop-2 profile.
> >> Now it seems critical to do so. Thus, I support Szehon's proposal to
> make
> >> this happen. This is orthogonal to the deprecation discussion.
> >>
> >> Thanks,
> >> Xuefu
> >>
> >> On Wed, Apr 29, 2015 at 11:30 AM, Szehon Ho <sz...@cloudera.com>
> wrote:
> >>
> >> > Are you
> >> > > personally willing to do the work to make sure that Hive works with
> >> > Hadoop
> >> > > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443,
> 10444)
> >> > > because
> >> > > trunk currently does not work with Hadoop 1.x and no one noticed.
> >> >
> >> > We setup a Hadoop-1 build sometime back and made sure a lot of
> breakage
> >> > were fixed, but had not hooked it up to HiveQA (precommit) due to
> >> concerns
> >> > of the time it would take to run these tests. It was a balance
> between
> >> > getting faster HiveQA results for everyone and testing all the paths,
> >> even
> >> > today there are many paths we don't test.
> >> >
> >> > So if these JIRA's are fixed, would folks be amenable to alternate
> >> testing
> >> > hadoop-1 and hadoop-2 on HiveQA precommit runs and for devs to not
> commit
> >> > further breakages of Hadoop 1.x compat, for another release? Yes I do
> >> > understand its a burden to maintain hadoop-1.
> >> >
> >> > if you use HDP, CDH or other hadoop distro then hadoop-1.x and
> hive-1.2
> >> > > combination does not exist.
> >> > >
> >> >
> >> > I think Apache is agnostic to specific vendors so should not discuss
> >> based
> >> > on that, but vendors still have this combination, I know of users that
> >> use
> >> > Hadoop-1 and latest Hive 1.1.
> >> >
> >> > Thanks,
> >> > Szehon
> >> >
> >> > On Wed, Apr 29, 2015 at 11:07 AM, Alexander Pivovarov <
> >> > apivovarov@gmail.com>
> >> > wrote:
> >> >
> >> > > HDP and CDH distro are based on hadoop-2.x since 2013-2014
> >> > >
> >> > > if you use HDP, CDH or other hadoop distro then hadoop-1.x and
> hive-1.2
> >> > > combination does not exist.
> >> > > hdp-1.3 has hive-0.11
> >> > > cdh-4.7 has hive-0.10
> >> > >
> >> > > The only way to install hive-1.2 on hadoop-1.x is to do it manually.
> >> > > But if you manually install hadoop components then most probably you
> >> will
> >> > > install hadoop-2.x
> >> > >
> >> > > So, we can say that hadoop-1.x and hive-1.2 combination should not
> >> exist
> >> > in
> >> > > real life scenario.
> >> > >
> >> > >
> >> > >
> >> > > On Wed, Apr 29, 2015 at 8:03 AM, Owen O'Malley <om...@apache.org>
> >> > wrote:
> >> > >
> >> > > > On Tue, Apr 28, 2015 at 10:16 PM, Szehon Ho <sz...@cloudera.com>
> >> > wrote:
> >> > > >
> >> > > > > Yes, question is whether users want to run latest Hive version
> on
> >> > > Hadoop
> >> > > > > 1.x clusters.
> >> > > > >
> >> > > >
> >> > > > But you ignored the more important question that I asked. Are you
> >> > > > personally willing to do the work to make sure that Hive works
> with
> >> > > Hadoop
> >> > > > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443,
> >> 10444)
> >> > > > because
> >> > > > trunk currently does not work with Hadoop 1.x and no one noticed.
> >> > > >
> >> > > > Supporting Hadoop 1.x costs the Hive project a lot of work and
> >> provides
> >> > > > very little in return.
> >> > > >
> >> > > > .. Owen
> >> > > >
> >> > >
> >> >
> >>
>
Re: [DISCUSS] Do we still support hadoop-1.2.x API (-Phadoop-1)?
Posted by Thejas Nair <th...@gmail.com>.
I agree, the number of users who use Hive 1.1 and newer versions
against Hadoop 1.x is likely to very small. (Most likely an orders of
magnitude smaller than hive cli users in same versions).
The big improvements in Hive 1.x line require Hadoop 2.x. Users who
are motivated to upgrade to newer 1.x lines are likely to be using
Hadoop 2.x .
On Wed, Apr 29, 2015 at 12:41 PM, Alexander Pivovarov
<ap...@gmail.com> wrote:
> Xuefu,
>
> Remember when it was beeline discussion you mentioned "Hive community don't
> need to maintain two code paths"
> Totally agree. Resources are limited.
>
> I believe the same approach is applicable here.
> Lets assume hadoop-1 + hive-1.2 is going to be used by 1% of the users.
>
> We'd better spend resources on implementing new functionality (useful for
> 99% of the users) instead of spending resources to support hadoop-1 profile
> (useful for 1% of the users).
>
> Alex
>
>
> On Wed, Apr 29, 2015 at 12:13 PM, Xuefu Zhang <xz...@cloudera.com> wrote:
>
>> I think whether and when to deprecate Hadoop-1 is better to be moved to an
>> official [DISCUSS] thread. What I disliked is the sentiment that I felt
>> when we decide to deprecate something just because it's unfortunately
>> broken for one reason or the other. Nevertheless, we should fix these bugs
>> at this moment. We fixed such issues in previous release as well.
>>
>> I concur that we should have caught these breakages before they made their
>> way into the code. Previously we discussed about running pre-commit tests
>> alternatively or even randomly with either Hadoop-1 or Hadoop-2 profile.
>> Now it seems critical to do so. Thus, I support Szehon's proposal to make
>> this happen. This is orthogonal to the deprecation discussion.
>>
>> Thanks,
>> Xuefu
>>
>> On Wed, Apr 29, 2015 at 11:30 AM, Szehon Ho <sz...@cloudera.com> wrote:
>>
>> > Are you
>> > > personally willing to do the work to make sure that Hive works with
>> > Hadoop
>> > > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443, 10444)
>> > > because
>> > > trunk currently does not work with Hadoop 1.x and no one noticed.
>> >
>> > We setup a Hadoop-1 build sometime back and made sure a lot of breakage
>> > were fixed, but had not hooked it up to HiveQA (precommit) due to
>> concerns
>> > of the time it would take to run these tests. It was a balance between
>> > getting faster HiveQA results for everyone and testing all the paths,
>> even
>> > today there are many paths we don't test.
>> >
>> > So if these JIRA's are fixed, would folks be amenable to alternate
>> testing
>> > hadoop-1 and hadoop-2 on HiveQA precommit runs and for devs to not commit
>> > further breakages of Hadoop 1.x compat, for another release? Yes I do
>> > understand its a burden to maintain hadoop-1.
>> >
>> > if you use HDP, CDH or other hadoop distro then hadoop-1.x and hive-1.2
>> > > combination does not exist.
>> > >
>> >
>> > I think Apache is agnostic to specific vendors so should not discuss
>> based
>> > on that, but vendors still have this combination, I know of users that
>> use
>> > Hadoop-1 and latest Hive 1.1.
>> >
>> > Thanks,
>> > Szehon
>> >
>> > On Wed, Apr 29, 2015 at 11:07 AM, Alexander Pivovarov <
>> > apivovarov@gmail.com>
>> > wrote:
>> >
>> > > HDP and CDH distro are based on hadoop-2.x since 2013-2014
>> > >
>> > > if you use HDP, CDH or other hadoop distro then hadoop-1.x and hive-1.2
>> > > combination does not exist.
>> > > hdp-1.3 has hive-0.11
>> > > cdh-4.7 has hive-0.10
>> > >
>> > > The only way to install hive-1.2 on hadoop-1.x is to do it manually.
>> > > But if you manually install hadoop components then most probably you
>> will
>> > > install hadoop-2.x
>> > >
>> > > So, we can say that hadoop-1.x and hive-1.2 combination should not
>> exist
>> > in
>> > > real life scenario.
>> > >
>> > >
>> > >
>> > > On Wed, Apr 29, 2015 at 8:03 AM, Owen O'Malley <om...@apache.org>
>> > wrote:
>> > >
>> > > > On Tue, Apr 28, 2015 at 10:16 PM, Szehon Ho <sz...@cloudera.com>
>> > wrote:
>> > > >
>> > > > > Yes, question is whether users want to run latest Hive version on
>> > > Hadoop
>> > > > > 1.x clusters.
>> > > > >
>> > > >
>> > > > But you ignored the more important question that I asked. Are you
>> > > > personally willing to do the work to make sure that Hive works with
>> > > Hadoop
>> > > > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443,
>> 10444)
>> > > > because
>> > > > trunk currently does not work with Hadoop 1.x and no one noticed.
>> > > >
>> > > > Supporting Hadoop 1.x costs the Hive project a lot of work and
>> provides
>> > > > very little in return.
>> > > >
>> > > > .. Owen
>> > > >
>> > >
>> >
>>
Re: [DISCUSS] Do we still support hadoop-1.2.x API (-Phadoop-1)?
Posted by Alexander Pivovarov <ap...@gmail.com>.
Xuefu,
Remember when it was beeline discussion you mentioned "Hive community don't
need to maintain two code paths"
Totally agree. Resources are limited.
I believe the same approach is applicable here.
Lets assume hadoop-1 + hive-1.2 is going to be used by 1% of the users.
We'd better spend resources on implementing new functionality (useful for
99% of the users) instead of spending resources to support hadoop-1 profile
(useful for 1% of the users).
Alex
On Wed, Apr 29, 2015 at 12:13 PM, Xuefu Zhang <xz...@cloudera.com> wrote:
> I think whether and when to deprecate Hadoop-1 is better to be moved to an
> official [DISCUSS] thread. What I disliked is the sentiment that I felt
> when we decide to deprecate something just because it's unfortunately
> broken for one reason or the other. Nevertheless, we should fix these bugs
> at this moment. We fixed such issues in previous release as well.
>
> I concur that we should have caught these breakages before they made their
> way into the code. Previously we discussed about running pre-commit tests
> alternatively or even randomly with either Hadoop-1 or Hadoop-2 profile.
> Now it seems critical to do so. Thus, I support Szehon's proposal to make
> this happen. This is orthogonal to the deprecation discussion.
>
> Thanks,
> Xuefu
>
> On Wed, Apr 29, 2015 at 11:30 AM, Szehon Ho <sz...@cloudera.com> wrote:
>
> > Are you
> > > personally willing to do the work to make sure that Hive works with
> > Hadoop
> > > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443, 10444)
> > > because
> > > trunk currently does not work with Hadoop 1.x and no one noticed.
> >
> > We setup a Hadoop-1 build sometime back and made sure a lot of breakage
> > were fixed, but had not hooked it up to HiveQA (precommit) due to
> concerns
> > of the time it would take to run these tests. It was a balance between
> > getting faster HiveQA results for everyone and testing all the paths,
> even
> > today there are many paths we don't test.
> >
> > So if these JIRA's are fixed, would folks be amenable to alternate
> testing
> > hadoop-1 and hadoop-2 on HiveQA precommit runs and for devs to not commit
> > further breakages of Hadoop 1.x compat, for another release? Yes I do
> > understand its a burden to maintain hadoop-1.
> >
> > if you use HDP, CDH or other hadoop distro then hadoop-1.x and hive-1.2
> > > combination does not exist.
> > >
> >
> > I think Apache is agnostic to specific vendors so should not discuss
> based
> > on that, but vendors still have this combination, I know of users that
> use
> > Hadoop-1 and latest Hive 1.1.
> >
> > Thanks,
> > Szehon
> >
> > On Wed, Apr 29, 2015 at 11:07 AM, Alexander Pivovarov <
> > apivovarov@gmail.com>
> > wrote:
> >
> > > HDP and CDH distro are based on hadoop-2.x since 2013-2014
> > >
> > > if you use HDP, CDH or other hadoop distro then hadoop-1.x and hive-1.2
> > > combination does not exist.
> > > hdp-1.3 has hive-0.11
> > > cdh-4.7 has hive-0.10
> > >
> > > The only way to install hive-1.2 on hadoop-1.x is to do it manually.
> > > But if you manually install hadoop components then most probably you
> will
> > > install hadoop-2.x
> > >
> > > So, we can say that hadoop-1.x and hive-1.2 combination should not
> exist
> > in
> > > real life scenario.
> > >
> > >
> > >
> > > On Wed, Apr 29, 2015 at 8:03 AM, Owen O'Malley <om...@apache.org>
> > wrote:
> > >
> > > > On Tue, Apr 28, 2015 at 10:16 PM, Szehon Ho <sz...@cloudera.com>
> > wrote:
> > > >
> > > > > Yes, question is whether users want to run latest Hive version on
> > > Hadoop
> > > > > 1.x clusters.
> > > > >
> > > >
> > > > But you ignored the more important question that I asked. Are you
> > > > personally willing to do the work to make sure that Hive works with
> > > Hadoop
> > > > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443,
> 10444)
> > > > because
> > > > trunk currently does not work with Hadoop 1.x and no one noticed.
> > > >
> > > > Supporting Hadoop 1.x costs the Hive project a lot of work and
> provides
> > > > very little in return.
> > > >
> > > > .. Owen
> > > >
> > >
> >
>
Re: [DISCUSS] Do we still support hadoop-1.2.x API (-Phadoop-1)?
Posted by Xuefu Zhang <xz...@cloudera.com>.
I think whether and when to deprecate Hadoop-1 is better to be moved to an
official [DISCUSS] thread. What I disliked is the sentiment that I felt
when we decide to deprecate something just because it's unfortunately
broken for one reason or the other. Nevertheless, we should fix these bugs
at this moment. We fixed such issues in previous release as well.
I concur that we should have caught these breakages before they made their
way into the code. Previously we discussed about running pre-commit tests
alternatively or even randomly with either Hadoop-1 or Hadoop-2 profile.
Now it seems critical to do so. Thus, I support Szehon's proposal to make
this happen. This is orthogonal to the deprecation discussion.
Thanks,
Xuefu
On Wed, Apr 29, 2015 at 11:30 AM, Szehon Ho <sz...@cloudera.com> wrote:
> Are you
> > personally willing to do the work to make sure that Hive works with
> Hadoop
> > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443, 10444)
> > because
> > trunk currently does not work with Hadoop 1.x and no one noticed.
>
> We setup a Hadoop-1 build sometime back and made sure a lot of breakage
> were fixed, but had not hooked it up to HiveQA (precommit) due to concerns
> of the time it would take to run these tests. It was a balance between
> getting faster HiveQA results for everyone and testing all the paths, even
> today there are many paths we don't test.
>
> So if these JIRA's are fixed, would folks be amenable to alternate testing
> hadoop-1 and hadoop-2 on HiveQA precommit runs and for devs to not commit
> further breakages of Hadoop 1.x compat, for another release? Yes I do
> understand its a burden to maintain hadoop-1.
>
> if you use HDP, CDH or other hadoop distro then hadoop-1.x and hive-1.2
> > combination does not exist.
> >
>
> I think Apache is agnostic to specific vendors so should not discuss based
> on that, but vendors still have this combination, I know of users that use
> Hadoop-1 and latest Hive 1.1.
>
> Thanks,
> Szehon
>
> On Wed, Apr 29, 2015 at 11:07 AM, Alexander Pivovarov <
> apivovarov@gmail.com>
> wrote:
>
> > HDP and CDH distro are based on hadoop-2.x since 2013-2014
> >
> > if you use HDP, CDH or other hadoop distro then hadoop-1.x and hive-1.2
> > combination does not exist.
> > hdp-1.3 has hive-0.11
> > cdh-4.7 has hive-0.10
> >
> > The only way to install hive-1.2 on hadoop-1.x is to do it manually.
> > But if you manually install hadoop components then most probably you will
> > install hadoop-2.x
> >
> > So, we can say that hadoop-1.x and hive-1.2 combination should not exist
> in
> > real life scenario.
> >
> >
> >
> > On Wed, Apr 29, 2015 at 8:03 AM, Owen O'Malley <om...@apache.org>
> wrote:
> >
> > > On Tue, Apr 28, 2015 at 10:16 PM, Szehon Ho <sz...@cloudera.com>
> wrote:
> > >
> > > > Yes, question is whether users want to run latest Hive version on
> > Hadoop
> > > > 1.x clusters.
> > > >
> > >
> > > But you ignored the more important question that I asked. Are you
> > > personally willing to do the work to make sure that Hive works with
> > Hadoop
> > > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443, 10444)
> > > because
> > > trunk currently does not work with Hadoop 1.x and no one noticed.
> > >
> > > Supporting Hadoop 1.x costs the Hive project a lot of work and provides
> > > very little in return.
> > >
> > > .. Owen
> > >
> >
>
Re: [DISCUSS] Do we still support hadoop-1.2.x API (-Phadoop-1)?
Posted by Szehon Ho <sz...@cloudera.com>.
Are you
> personally willing to do the work to make sure that Hive works with Hadoop
> 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443, 10444)
> because
> trunk currently does not work with Hadoop 1.x and no one noticed.
We setup a Hadoop-1 build sometime back and made sure a lot of breakage
were fixed, but had not hooked it up to HiveQA (precommit) due to concerns
of the time it would take to run these tests. It was a balance between
getting faster HiveQA results for everyone and testing all the paths, even
today there are many paths we don't test.
So if these JIRA's are fixed, would folks be amenable to alternate testing
hadoop-1 and hadoop-2 on HiveQA precommit runs and for devs to not commit
further breakages of Hadoop 1.x compat, for another release? Yes I do
understand its a burden to maintain hadoop-1.
if you use HDP, CDH or other hadoop distro then hadoop-1.x and hive-1.2
> combination does not exist.
>
I think Apache is agnostic to specific vendors so should not discuss based
on that, but vendors still have this combination, I know of users that use
Hadoop-1 and latest Hive 1.1.
Thanks,
Szehon
On Wed, Apr 29, 2015 at 11:07 AM, Alexander Pivovarov <ap...@gmail.com>
wrote:
> HDP and CDH distro are based on hadoop-2.x since 2013-2014
>
> if you use HDP, CDH or other hadoop distro then hadoop-1.x and hive-1.2
> combination does not exist.
> hdp-1.3 has hive-0.11
> cdh-4.7 has hive-0.10
>
> The only way to install hive-1.2 on hadoop-1.x is to do it manually.
> But if you manually install hadoop components then most probably you will
> install hadoop-2.x
>
> So, we can say that hadoop-1.x and hive-1.2 combination should not exist in
> real life scenario.
>
>
>
> On Wed, Apr 29, 2015 at 8:03 AM, Owen O'Malley <om...@apache.org> wrote:
>
> > On Tue, Apr 28, 2015 at 10:16 PM, Szehon Ho <sz...@cloudera.com> wrote:
> >
> > > Yes, question is whether users want to run latest Hive version on
> Hadoop
> > > 1.x clusters.
> > >
> >
> > But you ignored the more important question that I asked. Are you
> > personally willing to do the work to make sure that Hive works with
> Hadoop
> > 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443, 10444)
> > because
> > trunk currently does not work with Hadoop 1.x and no one noticed.
> >
> > Supporting Hadoop 1.x costs the Hive project a lot of work and provides
> > very little in return.
> >
> > .. Owen
> >
>
Re: [DISCUSS] Do we still support hadoop-1.2.x API (-Phadoop-1)?
Posted by Alexander Pivovarov <ap...@gmail.com>.
HDP and CDH distro are based on hadoop-2.x since 2013-2014
if you use HDP, CDH or other hadoop distro then hadoop-1.x and hive-1.2
combination does not exist.
hdp-1.3 has hive-0.11
cdh-4.7 has hive-0.10
The only way to install hive-1.2 on hadoop-1.x is to do it manually.
But if you manually install hadoop components then most probably you will
install hadoop-2.x
So, we can say that hadoop-1.x and hive-1.2 combination should not exist in
real life scenario.
On Wed, Apr 29, 2015 at 8:03 AM, Owen O'Malley <om...@apache.org> wrote:
> On Tue, Apr 28, 2015 at 10:16 PM, Szehon Ho <sz...@cloudera.com> wrote:
>
> > Yes, question is whether users want to run latest Hive version on Hadoop
> > 1.x clusters.
> >
>
> But you ignored the more important question that I asked. Are you
> personally willing to do the work to make sure that Hive works with Hadoop
> 1.x? There are five new jiras (HIVE-10430, 10431, 10442, 10443, 10444)
> because
> trunk currently does not work with Hadoop 1.x and no one noticed.
>
> Supporting Hadoop 1.x costs the Hive project a lot of work and provides
> very little in return.
>
> .. Owen
>