You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Alan Gates <ga...@hortonworks.com> on 2013/03/09 21:54:03 UTC
Re: Merging HCatalog into Hive
Alright, I've gotten some feedback from Brock around the JIRA stuff and Carl in a live conversation expressed his desire to move hcat into the Hive namespace sooner rather than later. So the proposal is that we'd move the code to org.apache.hive.hcatalog, though we would create shell classes and interfaces in org.apache.hcatalog for all public classes and interfaces so that it will be backward compatible. I'm fine with doing this now.
So, let's get started. Carl, could you create an hcatalog directory under trunk/hive and grant the listed hcat committers karma on it? Then I'll get started on moving the actual code.
Alan.
On Feb 24, 2013, at 12:22 PM, Brock Noland wrote:
> Looks good from my perspective and I glad to see this moving forward.
>
> Regarding #4 (JIRA)
>
> "I don't know if there's a way to upload existing JIRAs into Hive's JIRA,
> but I think it would be better to leave them where they are."
>
> JIRA has a bulk move feature, but I am curious as why we would leave them
> under the old project? There might be good reason to orphan them, but my
> first thought is that it would be nice to have them under the HIVE project
> simply for search purposes.
>
> Brock
>
>
>
>
> On Fri, Feb 22, 2013 at 7:12 PM, Alan Gates <ga...@hortonworks.com> wrote:
>
>> Alright, our vote has passed, it's time to get on with merging HCatalog
>> into Hive. Here's the things I can think of we need to deal with. Please
>> add additional issues I've missed:
>>
>> 1) Moving the code
>> 2) Dealing with domain names in the code
>> 3) The mailing lists
>> 4) The JIRA
>> 5) The website
>> 6) Committer rights
>> 7) Make a proposal for how HCat is released going forward
>> 8) Publish an FAQ
>>
>> Proposals for how we handle these:
>> Below I propose an approach for how to handle each of these. Feedback
>> welcome.
>>
>> 1) Moving the code
>> I propose that HCat move into a subdirectory of Hive. This fits nicely
>> into Hive's structure since it already has metastore, ql, etc. We'd just
>> add 'hcatalog' as a new directory. This directory would contain hcatalog
>> as it is today. It does not follow Hive's standard build model so we'd
>> need to do some work to make it so that building Hive also builds HCat, but
>> this should be minimal.
>>
>> 2) Dealing with domain names
>> HCat code currently is under org.apache.hcatalog. Do we want to change
>> it? In time we probably should change it to match the rest of Hive
>> (org.apache.hadoop.hive.hcatalog). We need to do this in a backward
>> compatible way. I propose we leave it as is for now and if we decide to in
>> the future we can move the actual code to org.apache.hadoop.hive.hcatalog
>> and create shell classes under org.apache.hcatalog.
>>
>> 3) The mailing lists
>> Given that our goal is to merge the projects and not create a subproject
>> we should merge the mailing lists rather than keep hcat specific lists. We
>> can ask infra to remove hcatalog-*@incubator.apache.org and forward any
>> new mail to the appropriate Hive lists. We need to find out if they can
>> auto-subscribe people from the hcat lists to the hive lists. Given that
>> traffic on the Hive lists is an order of magnitude higher we should warn
>> people before we auto-subscribe them and allow them a chance to get off.
>>
>> 4) JIRA
>> We can create an hcatalog component in Hive's JIRA. All new HCat issues
>> could be filed there. I don't know if there's a way to upload existing
>> JIRAs into Hive's JIRA, but I think it would be better to leave them where
>> they are. We should see if infra can turn off the ability to create new
>> JIRAs in hcatalog.
>>
>> 5) Website
>> We will need to integrate HCatalog's website with Hive's. This should be
>> easy except for the documentation. HCat uses forrest for docs, Hive uses
>> wiki. We will need to put links under 'Documentation' for older versions
>> of HCat docs so users can find them. As far as how docs are handled for
>> the next version of HCatalog, I think that depends on the answer to
>> question 7 (next release of HCat), but I propose that HCat needs to conform
>> to the way Hive does docs on wiki. Though I would strongly encourage the
>> HCat docs to be version specific (that is, have a set of wiki pages for
>> each version). incubator.apache.org/hcatalog should be changed to
>> forward to hive.apache.org.
>>
>> 6) Committer rights
>> Carl will need to set up committer rights for all the new HCat committers.
>> Based on our discussion of making active HCat committers Hive submodule
>> committers this would add the following set: Alan, Sushanth, Francis,
>> Daniel, Vandana, Travis, and Mithun. Ashutosh and Paul are already Hive
>> committers, and neither Devaraj nor Mac have been active in HCat in over a
>> year.
>>
>> 7) Future releases
>> We need to discuss how future releases will happen, as I think this will
>> help developers and users know how to respond to the merge. I propose that
>> HCat will simply become part of future Hive releases. Thus Hive 0.11 (or
>> whatever the next major release is) will include HCatalog. If there are
>> issues found we may need to make HCatalog 0.5.x releases from Hive, which
>> should be fine. But I propose there would not be an HCat 0.6. To be clear
>> I am not proposing that HCat functionality would be subsumed into Hive
>> jars. Just that the existing hcat jars would become part of Hive's release.
>>
>> 8) Communicate all of this
>> We should put up an FAQ page that has this information, as well as tracks
>> our progress while we work on getting these things done.
>>
>> Alan.
>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Merging HCatalog into Hive
Posted by Alan Gates <ga...@hortonworks.com>.
Cool, it works now. Thanks for the fast response.
Alan.
On Mar 26, 2013, at 2:58 PM, Carl Steinbach wrote:
> Hi Alan,
>
> I agree that it will probably be too painful to enforce the rules with SVN, so I went ahead and gave all of the HCatalog committers RW access to /hive. Please follow the rules. If I receive any complaints about this I'll revert back to the old scheme.
>
> Thanks.
>
> Carl
>
> On Tue, Mar 26, 2013 at 2:34 PM, Alan Gates <ga...@hortonworks.com> wrote:
> There's an issue with the permissions here. In the authorization file you granted permission to hcatalog committers on a directory /hive/hcatalog. But in Hive you created /hive/trunk/hcatalog, which none of the hcatalog committers can access. In the authorization file you'll need to change hive-hcatalog to have authorization /hive/trunk/hcatalog.
>
> There is also a scalability issue. Every time Hive branches you'll have to add a line for that branch as well. Also, this will prohibit any dev branches for hcatalog users, or access to any dev branches done in Hive. I suspect you'll find it much easier to give the hive-hcatalog group access to /hive and then use community mores to enforce that no hcat committers commit outside the hcat directory.
>
> Alan.
>
> On Mar 15, 2013, at 5:26 PM, Carl Steinbach wrote:
>
> > Hi Alan,
> >
> > I committed HIVE-4145, created an HCatalog component on JIRA, and
> > updated the asf-authorization-template to give the HCatalog committers
> > karma on the hcatalog subdirectory. At this point I think everything should
> > be ready to go. Let me know if you run into any problems.
> >
> > Thanks.
> >
> > Carl
> >
> > On Wed, Mar 13, 2013 at 11:56 AM, Alan Gates <ga...@hortonworks.com> wrote:
> > Proposed changes look good to me. And you don't need an infra ticket to grant karma. Since you're Hive VP you can do it. See http://www.apache.org/dev/pmc.html#SVNaccess
> >
> > Alan.
> >
> > On Mar 10, 2013, at 9:29 PM, Carl Steinbach wrote:
> >
> > > Hi Alan,
> > >
> > > I submitted a patch that creates the hcatalog directory and makes some other necessary
> > > changes here:
> > >
> > > https://issues.apache.org/jira/browse/HIVE-4145
> > >
> > > Once this is committed I will contact ASFINFRA and ask them to grant the HCatalog
> > > committers karma.
> > >
> > > Thanks.
> > >
> > > Carl
> > >
> > > On Sat, Mar 9, 2013 at 12:54 PM, Alan Gates <ga...@hortonworks.com> wrote:
> > > Alright, I've gotten some feedback from Brock around the JIRA stuff and Carl in a live conversation expressed his desire to move hcat into the Hive namespace sooner rather than later. So the proposal is that we'd move the code to org.apache.hive.hcatalog, though we would create shell classes and interfaces in org.apache.hcatalog for all public classes and interfaces so that it will be backward compatible. I'm fine with doing this now.
> > >
> > > So, let's get started. Carl, could you create an hcatalog directory under trunk/hive and grant the listed hcat committers karma on it? Then I'll get started on moving the actual code.
> > >
> > > Alan.
> > >
> > > On Feb 24, 2013, at 12:22 PM, Brock Noland wrote:
> > >
> > > > Looks good from my perspective and I glad to see this moving forward.
> > > >
> > > > Regarding #4 (JIRA)
> > > >
> > > > "I don't know if there's a way to upload existing JIRAs into Hive's JIRA,
> > > > but I think it would be better to leave them where they are."
> > > >
> > > > JIRA has a bulk move feature, but I am curious as why we would leave them
> > > > under the old project? There might be good reason to orphan them, but my
> > > > first thought is that it would be nice to have them under the HIVE project
> > > > simply for search purposes.
> > > >
> > > > Brock
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Feb 22, 2013 at 7:12 PM, Alan Gates <ga...@hortonworks.com> wrote:
> > > >
> > > >> Alright, our vote has passed, it's time to get on with merging HCatalog
> > > >> into Hive. Here's the things I can think of we need to deal with. Please
> > > >> add additional issues I've missed:
> > > >>
> > > >> 1) Moving the code
> > > >> 2) Dealing with domain names in the code
> > > >> 3) The mailing lists
> > > >> 4) The JIRA
> > > >> 5) The website
> > > >> 6) Committer rights
> > > >> 7) Make a proposal for how HCat is released going forward
> > > >> 8) Publish an FAQ
> > > >>
> > > >> Proposals for how we handle these:
> > > >> Below I propose an approach for how to handle each of these. Feedback
> > > >> welcome.
> > > >>
> > > >> 1) Moving the code
> > > >> I propose that HCat move into a subdirectory of Hive. This fits nicely
> > > >> into Hive's structure since it already has metastore, ql, etc. We'd just
> > > >> add 'hcatalog' as a new directory. This directory would contain hcatalog
> > > >> as it is today. It does not follow Hive's standard build model so we'd
> > > >> need to do some work to make it so that building Hive also builds HCat, but
> > > >> this should be minimal.
> > > >>
> > > >> 2) Dealing with domain names
> > > >> HCat code currently is under org.apache.hcatalog. Do we want to change
> > > >> it? In time we probably should change it to match the rest of Hive
> > > >> (org.apache.hadoop.hive.hcatalog). We need to do this in a backward
> > > >> compatible way. I propose we leave it as is for now and if we decide to in
> > > >> the future we can move the actual code to org.apache.hadoop.hive.hcatalog
> > > >> and create shell classes under org.apache.hcatalog.
> > > >>
> > > >> 3) The mailing lists
> > > >> Given that our goal is to merge the projects and not create a subproject
> > > >> we should merge the mailing lists rather than keep hcat specific lists. We
> > > >> can ask infra to remove hcatalog-*@incubator.apache.org and forward any
> > > >> new mail to the appropriate Hive lists. We need to find out if they can
> > > >> auto-subscribe people from the hcat lists to the hive lists. Given that
> > > >> traffic on the Hive lists is an order of magnitude higher we should warn
> > > >> people before we auto-subscribe them and allow them a chance to get off.
> > > >>
> > > >> 4) JIRA
> > > >> We can create an hcatalog component in Hive's JIRA. All new HCat issues
> > > >> could be filed there. I don't know if there's a way to upload existing
> > > >> JIRAs into Hive's JIRA, but I think it would be better to leave them where
> > > >> they are. We should see if infra can turn off the ability to create new
> > > >> JIRAs in hcatalog.
> > > >>
> > > >> 5) Website
> > > >> We will need to integrate HCatalog's website with Hive's. This should be
> > > >> easy except for the documentation. HCat uses forrest for docs, Hive uses
> > > >> wiki. We will need to put links under 'Documentation' for older versions
> > > >> of HCat docs so users can find them. As far as how docs are handled for
> > > >> the next version of HCatalog, I think that depends on the answer to
> > > >> question 7 (next release of HCat), but I propose that HCat needs to conform
> > > >> to the way Hive does docs on wiki. Though I would strongly encourage the
> > > >> HCat docs to be version specific (that is, have a set of wiki pages for
> > > >> each version). incubator.apache.org/hcatalog should be changed to
> > > >> forward to hive.apache.org.
> > > >>
> > > >> 6) Committer rights
> > > >> Carl will need to set up committer rights for all the new HCat committers.
> > > >> Based on our discussion of making active HCat committers Hive submodule
> > > >> committers this would add the following set: Alan, Sushanth, Francis,
> > > >> Daniel, Vandana, Travis, and Mithun. Ashutosh and Paul are already Hive
> > > >> committers, and neither Devaraj nor Mac have been active in HCat in over a
> > > >> year.
> > > >>
> > > >> 7) Future releases
> > > >> We need to discuss how future releases will happen, as I think this will
> > > >> help developers and users know how to respond to the merge. I propose that
> > > >> HCat will simply become part of future Hive releases. Thus Hive 0.11 (or
> > > >> whatever the next major release is) will include HCatalog. If there are
> > > >> issues found we may need to make HCatalog 0.5.x releases from Hive, which
> > > >> should be fine. But I propose there would not be an HCat 0.6. To be clear
> > > >> I am not proposing that HCat functionality would be subsumed into Hive
> > > >> jars. Just that the existing hcat jars would become part of Hive's release.
> > > >>
> > > >> 8) Communicate all of this
> > > >> We should put up an FAQ page that has this information, as well as tracks
> > > >> our progress while we work on getting these things done.
> > > >>
> > > >> Alan.
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
> > >
> > >
> >
> >
>
>
Re: Merging HCatalog into Hive
Posted by Carl Steinbach <cw...@apache.org>.
Hi Alan,
I agree that it will probably be too painful to enforce the rules with SVN,
so I went ahead and gave all of the HCatalog committers RW access to /hive.
Please follow the rules. If I receive any complaints about this I'll revert
back to the old scheme.
Thanks.
Carl
On Tue, Mar 26, 2013 at 2:34 PM, Alan Gates <ga...@hortonworks.com> wrote:
> There's an issue with the permissions here. In the authorization file you
> granted permission to hcatalog committers on a directory /hive/hcatalog.
> But in Hive you created /hive/trunk/hcatalog, which none of the hcatalog
> committers can access. In the authorization file you'll need to change
> hive-hcatalog to have authorization /hive/trunk/hcatalog.
>
> There is also a scalability issue. Every time Hive branches you'll have
> to add a line for that branch as well. Also, this will prohibit any dev
> branches for hcatalog users, or access to any dev branches done in Hive. I
> suspect you'll find it much easier to give the hive-hcatalog group access
> to /hive and then use community mores to enforce that no hcat committers
> commit outside the hcat directory.
>
> Alan.
>
> On Mar 15, 2013, at 5:26 PM, Carl Steinbach wrote:
>
> > Hi Alan,
> >
> > I committed HIVE-4145, created an HCatalog component on JIRA, and
> > updated the asf-authorization-template to give the HCatalog committers
> > karma on the hcatalog subdirectory. At this point I think everything
> should
> > be ready to go. Let me know if you run into any problems.
> >
> > Thanks.
> >
> > Carl
> >
> > On Wed, Mar 13, 2013 at 11:56 AM, Alan Gates <ga...@hortonworks.com>
> wrote:
> > Proposed changes look good to me. And you don't need an infra ticket to
> grant karma. Since you're Hive VP you can do it. See
> http://www.apache.org/dev/pmc.html#SVNaccess
> >
> > Alan.
> >
> > On Mar 10, 2013, at 9:29 PM, Carl Steinbach wrote:
> >
> > > Hi Alan,
> > >
> > > I submitted a patch that creates the hcatalog directory and makes some
> other necessary
> > > changes here:
> > >
> > > https://issues.apache.org/jira/browse/HIVE-4145
> > >
> > > Once this is committed I will contact ASFINFRA and ask them to grant
> the HCatalog
> > > committers karma.
> > >
> > > Thanks.
> > >
> > > Carl
> > >
> > > On Sat, Mar 9, 2013 at 12:54 PM, Alan Gates <ga...@hortonworks.com>
> wrote:
> > > Alright, I've gotten some feedback from Brock around the JIRA stuff
> and Carl in a live conversation expressed his desire to move hcat into the
> Hive namespace sooner rather than later. So the proposal is that we'd move
> the code to org.apache.hive.hcatalog, though we would create shell classes
> and interfaces in org.apache.hcatalog for all public classes and interfaces
> so that it will be backward compatible. I'm fine with doing this now.
> > >
> > > So, let's get started. Carl, could you create an hcatalog directory
> under trunk/hive and grant the listed hcat committers karma on it? Then
> I'll get started on moving the actual code.
> > >
> > > Alan.
> > >
> > > On Feb 24, 2013, at 12:22 PM, Brock Noland wrote:
> > >
> > > > Looks good from my perspective and I glad to see this moving forward.
> > > >
> > > > Regarding #4 (JIRA)
> > > >
> > > > "I don't know if there's a way to upload existing JIRAs into Hive's
> JIRA,
> > > > but I think it would be better to leave them where they are."
> > > >
> > > > JIRA has a bulk move feature, but I am curious as why we would leave
> them
> > > > under the old project? There might be good reason to orphan them,
> but my
> > > > first thought is that it would be nice to have them under the HIVE
> project
> > > > simply for search purposes.
> > > >
> > > > Brock
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Feb 22, 2013 at 7:12 PM, Alan Gates <ga...@hortonworks.com>
> wrote:
> > > >
> > > >> Alright, our vote has passed, it's time to get on with merging
> HCatalog
> > > >> into Hive. Here's the things I can think of we need to deal with.
> Please
> > > >> add additional issues I've missed:
> > > >>
> > > >> 1) Moving the code
> > > >> 2) Dealing with domain names in the code
> > > >> 3) The mailing lists
> > > >> 4) The JIRA
> > > >> 5) The website
> > > >> 6) Committer rights
> > > >> 7) Make a proposal for how HCat is released going forward
> > > >> 8) Publish an FAQ
> > > >>
> > > >> Proposals for how we handle these:
> > > >> Below I propose an approach for how to handle each of these.
> Feedback
> > > >> welcome.
> > > >>
> > > >> 1) Moving the code
> > > >> I propose that HCat move into a subdirectory of Hive. This fits
> nicely
> > > >> into Hive's structure since it already has metastore, ql, etc.
> We'd just
> > > >> add 'hcatalog' as a new directory. This directory would contain
> hcatalog
> > > >> as it is today. It does not follow Hive's standard build model so
> we'd
> > > >> need to do some work to make it so that building Hive also builds
> HCat, but
> > > >> this should be minimal.
> > > >>
> > > >> 2) Dealing with domain names
> > > >> HCat code currently is under org.apache.hcatalog. Do we want to
> change
> > > >> it? In time we probably should change it to match the rest of Hive
> > > >> (org.apache.hadoop.hive.hcatalog). We need to do this in a backward
> > > >> compatible way. I propose we leave it as is for now and if we
> decide to in
> > > >> the future we can move the actual code to
> org.apache.hadoop.hive.hcatalog
> > > >> and create shell classes under org.apache.hcatalog.
> > > >>
> > > >> 3) The mailing lists
> > > >> Given that our goal is to merge the projects and not create a
> subproject
> > > >> we should merge the mailing lists rather than keep hcat specific
> lists. We
> > > >> can ask infra to remove hcatalog-*@incubator.apache.org and
> forward any
> > > >> new mail to the appropriate Hive lists. We need to find out if
> they can
> > > >> auto-subscribe people from the hcat lists to the hive lists. Given
> that
> > > >> traffic on the Hive lists is an order of magnitude higher we should
> warn
> > > >> people before we auto-subscribe them and allow them a chance to get
> off.
> > > >>
> > > >> 4) JIRA
> > > >> We can create an hcatalog component in Hive's JIRA. All new HCat
> issues
> > > >> could be filed there. I don't know if there's a way to upload
> existing
> > > >> JIRAs into Hive's JIRA, but I think it would be better to leave
> them where
> > > >> they are. We should see if infra can turn off the ability to
> create new
> > > >> JIRAs in hcatalog.
> > > >>
> > > >> 5) Website
> > > >> We will need to integrate HCatalog's website with Hive's. This
> should be
> > > >> easy except for the documentation. HCat uses forrest for docs,
> Hive uses
> > > >> wiki. We will need to put links under 'Documentation' for older
> versions
> > > >> of HCat docs so users can find them. As far as how docs are
> handled for
> > > >> the next version of HCatalog, I think that depends on the answer to
> > > >> question 7 (next release of HCat), but I propose that HCat needs to
> conform
> > > >> to the way Hive does docs on wiki. Though I would strongly
> encourage the
> > > >> HCat docs to be version specific (that is, have a set of wiki pages
> for
> > > >> each version). incubator.apache.org/hcatalog should be changed to
> > > >> forward to hive.apache.org.
> > > >>
> > > >> 6) Committer rights
> > > >> Carl will need to set up committer rights for all the new HCat
> committers.
> > > >> Based on our discussion of making active HCat committers Hive
> submodule
> > > >> committers this would add the following set: Alan, Sushanth,
> Francis,
> > > >> Daniel, Vandana, Travis, and Mithun. Ashutosh and Paul are already
> Hive
> > > >> committers, and neither Devaraj nor Mac have been active in HCat in
> over a
> > > >> year.
> > > >>
> > > >> 7) Future releases
> > > >> We need to discuss how future releases will happen, as I think this
> will
> > > >> help developers and users know how to respond to the merge. I
> propose that
> > > >> HCat will simply become part of future Hive releases. Thus Hive
> 0.11 (or
> > > >> whatever the next major release is) will include HCatalog. If
> there are
> > > >> issues found we may need to make HCatalog 0.5.x releases from Hive,
> which
> > > >> should be fine. But I propose there would not be an HCat 0.6. To
> be clear
> > > >> I am not proposing that HCat functionality would be subsumed into
> Hive
> > > >> jars. Just that the existing hcat jars would become part of Hive's
> release.
> > > >>
> > > >> 8) Communicate all of this
> > > >> We should put up an FAQ page that has this information, as well as
> tracks
> > > >> our progress while we work on getting these things done.
> > > >>
> > > >> Alan.
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
> > >
> > >
> >
> >
>
>
Re: Merging HCatalog into Hive
Posted by Alan Gates <ga...@hortonworks.com>.
There's an issue with the permissions here. In the authorization file you granted permission to hcatalog committers on a directory /hive/hcatalog. But in Hive you created /hive/trunk/hcatalog, which none of the hcatalog committers can access. In the authorization file you'll need to change hive-hcatalog to have authorization /hive/trunk/hcatalog.
There is also a scalability issue. Every time Hive branches you'll have to add a line for that branch as well. Also, this will prohibit any dev branches for hcatalog users, or access to any dev branches done in Hive. I suspect you'll find it much easier to give the hive-hcatalog group access to /hive and then use community mores to enforce that no hcat committers commit outside the hcat directory.
Alan.
On Mar 15, 2013, at 5:26 PM, Carl Steinbach wrote:
> Hi Alan,
>
> I committed HIVE-4145, created an HCatalog component on JIRA, and
> updated the asf-authorization-template to give the HCatalog committers
> karma on the hcatalog subdirectory. At this point I think everything should
> be ready to go. Let me know if you run into any problems.
>
> Thanks.
>
> Carl
>
> On Wed, Mar 13, 2013 at 11:56 AM, Alan Gates <ga...@hortonworks.com> wrote:
> Proposed changes look good to me. And you don't need an infra ticket to grant karma. Since you're Hive VP you can do it. See http://www.apache.org/dev/pmc.html#SVNaccess
>
> Alan.
>
> On Mar 10, 2013, at 9:29 PM, Carl Steinbach wrote:
>
> > Hi Alan,
> >
> > I submitted a patch that creates the hcatalog directory and makes some other necessary
> > changes here:
> >
> > https://issues.apache.org/jira/browse/HIVE-4145
> >
> > Once this is committed I will contact ASFINFRA and ask them to grant the HCatalog
> > committers karma.
> >
> > Thanks.
> >
> > Carl
> >
> > On Sat, Mar 9, 2013 at 12:54 PM, Alan Gates <ga...@hortonworks.com> wrote:
> > Alright, I've gotten some feedback from Brock around the JIRA stuff and Carl in a live conversation expressed his desire to move hcat into the Hive namespace sooner rather than later. So the proposal is that we'd move the code to org.apache.hive.hcatalog, though we would create shell classes and interfaces in org.apache.hcatalog for all public classes and interfaces so that it will be backward compatible. I'm fine with doing this now.
> >
> > So, let's get started. Carl, could you create an hcatalog directory under trunk/hive and grant the listed hcat committers karma on it? Then I'll get started on moving the actual code.
> >
> > Alan.
> >
> > On Feb 24, 2013, at 12:22 PM, Brock Noland wrote:
> >
> > > Looks good from my perspective and I glad to see this moving forward.
> > >
> > > Regarding #4 (JIRA)
> > >
> > > "I don't know if there's a way to upload existing JIRAs into Hive's JIRA,
> > > but I think it would be better to leave them where they are."
> > >
> > > JIRA has a bulk move feature, but I am curious as why we would leave them
> > > under the old project? There might be good reason to orphan them, but my
> > > first thought is that it would be nice to have them under the HIVE project
> > > simply for search purposes.
> > >
> > > Brock
> > >
> > >
> > >
> > >
> > > On Fri, Feb 22, 2013 at 7:12 PM, Alan Gates <ga...@hortonworks.com> wrote:
> > >
> > >> Alright, our vote has passed, it's time to get on with merging HCatalog
> > >> into Hive. Here's the things I can think of we need to deal with. Please
> > >> add additional issues I've missed:
> > >>
> > >> 1) Moving the code
> > >> 2) Dealing with domain names in the code
> > >> 3) The mailing lists
> > >> 4) The JIRA
> > >> 5) The website
> > >> 6) Committer rights
> > >> 7) Make a proposal for how HCat is released going forward
> > >> 8) Publish an FAQ
> > >>
> > >> Proposals for how we handle these:
> > >> Below I propose an approach for how to handle each of these. Feedback
> > >> welcome.
> > >>
> > >> 1) Moving the code
> > >> I propose that HCat move into a subdirectory of Hive. This fits nicely
> > >> into Hive's structure since it already has metastore, ql, etc. We'd just
> > >> add 'hcatalog' as a new directory. This directory would contain hcatalog
> > >> as it is today. It does not follow Hive's standard build model so we'd
> > >> need to do some work to make it so that building Hive also builds HCat, but
> > >> this should be minimal.
> > >>
> > >> 2) Dealing with domain names
> > >> HCat code currently is under org.apache.hcatalog. Do we want to change
> > >> it? In time we probably should change it to match the rest of Hive
> > >> (org.apache.hadoop.hive.hcatalog). We need to do this in a backward
> > >> compatible way. I propose we leave it as is for now and if we decide to in
> > >> the future we can move the actual code to org.apache.hadoop.hive.hcatalog
> > >> and create shell classes under org.apache.hcatalog.
> > >>
> > >> 3) The mailing lists
> > >> Given that our goal is to merge the projects and not create a subproject
> > >> we should merge the mailing lists rather than keep hcat specific lists. We
> > >> can ask infra to remove hcatalog-*@incubator.apache.org and forward any
> > >> new mail to the appropriate Hive lists. We need to find out if they can
> > >> auto-subscribe people from the hcat lists to the hive lists. Given that
> > >> traffic on the Hive lists is an order of magnitude higher we should warn
> > >> people before we auto-subscribe them and allow them a chance to get off.
> > >>
> > >> 4) JIRA
> > >> We can create an hcatalog component in Hive's JIRA. All new HCat issues
> > >> could be filed there. I don't know if there's a way to upload existing
> > >> JIRAs into Hive's JIRA, but I think it would be better to leave them where
> > >> they are. We should see if infra can turn off the ability to create new
> > >> JIRAs in hcatalog.
> > >>
> > >> 5) Website
> > >> We will need to integrate HCatalog's website with Hive's. This should be
> > >> easy except for the documentation. HCat uses forrest for docs, Hive uses
> > >> wiki. We will need to put links under 'Documentation' for older versions
> > >> of HCat docs so users can find them. As far as how docs are handled for
> > >> the next version of HCatalog, I think that depends on the answer to
> > >> question 7 (next release of HCat), but I propose that HCat needs to conform
> > >> to the way Hive does docs on wiki. Though I would strongly encourage the
> > >> HCat docs to be version specific (that is, have a set of wiki pages for
> > >> each version). incubator.apache.org/hcatalog should be changed to
> > >> forward to hive.apache.org.
> > >>
> > >> 6) Committer rights
> > >> Carl will need to set up committer rights for all the new HCat committers.
> > >> Based on our discussion of making active HCat committers Hive submodule
> > >> committers this would add the following set: Alan, Sushanth, Francis,
> > >> Daniel, Vandana, Travis, and Mithun. Ashutosh and Paul are already Hive
> > >> committers, and neither Devaraj nor Mac have been active in HCat in over a
> > >> year.
> > >>
> > >> 7) Future releases
> > >> We need to discuss how future releases will happen, as I think this will
> > >> help developers and users know how to respond to the merge. I propose that
> > >> HCat will simply become part of future Hive releases. Thus Hive 0.11 (or
> > >> whatever the next major release is) will include HCatalog. If there are
> > >> issues found we may need to make HCatalog 0.5.x releases from Hive, which
> > >> should be fine. But I propose there would not be an HCat 0.6. To be clear
> > >> I am not proposing that HCat functionality would be subsumed into Hive
> > >> jars. Just that the existing hcat jars would become part of Hive's release.
> > >>
> > >> 8) Communicate all of this
> > >> We should put up an FAQ page that has this information, as well as tracks
> > >> our progress while we work on getting these things done.
> > >>
> > >> Alan.
> > >
> > >
> > >
> > >
> > > --
> > > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
> >
> >
>
>
Re: Merging HCatalog into Hive
Posted by Alan Gates <ga...@hortonworks.com>.
Excellent, thank you Carl. I'll start on the process to move the code then.
Alan.
On Mar 15, 2013, at 5:26 PM, Carl Steinbach wrote:
> Hi Alan,
>
> I committed HIVE-4145, created an HCatalog component on JIRA, and
> updated the asf-authorization-template to give the HCatalog committers
> karma on the hcatalog subdirectory. At this point I think everything should
> be ready to go. Let me know if you run into any problems.
>
> Thanks.
>
> Carl
>
> On Wed, Mar 13, 2013 at 11:56 AM, Alan Gates <ga...@hortonworks.com> wrote:
> Proposed changes look good to me. And you don't need an infra ticket to grant karma. Since you're Hive VP you can do it. See http://www.apache.org/dev/pmc.html#SVNaccess
>
> Alan.
>
> On Mar 10, 2013, at 9:29 PM, Carl Steinbach wrote:
>
> > Hi Alan,
> >
> > I submitted a patch that creates the hcatalog directory and makes some other necessary
> > changes here:
> >
> > https://issues.apache.org/jira/browse/HIVE-4145
> >
> > Once this is committed I will contact ASFINFRA and ask them to grant the HCatalog
> > committers karma.
> >
> > Thanks.
> >
> > Carl
> >
> > On Sat, Mar 9, 2013 at 12:54 PM, Alan Gates <ga...@hortonworks.com> wrote:
> > Alright, I've gotten some feedback from Brock around the JIRA stuff and Carl in a live conversation expressed his desire to move hcat into the Hive namespace sooner rather than later. So the proposal is that we'd move the code to org.apache.hive.hcatalog, though we would create shell classes and interfaces in org.apache.hcatalog for all public classes and interfaces so that it will be backward compatible. I'm fine with doing this now.
> >
> > So, let's get started. Carl, could you create an hcatalog directory under trunk/hive and grant the listed hcat committers karma on it? Then I'll get started on moving the actual code.
> >
> > Alan.
> >
> > On Feb 24, 2013, at 12:22 PM, Brock Noland wrote:
> >
> > > Looks good from my perspective and I glad to see this moving forward.
> > >
> > > Regarding #4 (JIRA)
> > >
> > > "I don't know if there's a way to upload existing JIRAs into Hive's JIRA,
> > > but I think it would be better to leave them where they are."
> > >
> > > JIRA has a bulk move feature, but I am curious as why we would leave them
> > > under the old project? There might be good reason to orphan them, but my
> > > first thought is that it would be nice to have them under the HIVE project
> > > simply for search purposes.
> > >
> > > Brock
> > >
> > >
> > >
> > >
> > > On Fri, Feb 22, 2013 at 7:12 PM, Alan Gates <ga...@hortonworks.com> wrote:
> > >
> > >> Alright, our vote has passed, it's time to get on with merging HCatalog
> > >> into Hive. Here's the things I can think of we need to deal with. Please
> > >> add additional issues I've missed:
> > >>
> > >> 1) Moving the code
> > >> 2) Dealing with domain names in the code
> > >> 3) The mailing lists
> > >> 4) The JIRA
> > >> 5) The website
> > >> 6) Committer rights
> > >> 7) Make a proposal for how HCat is released going forward
> > >> 8) Publish an FAQ
> > >>
> > >> Proposals for how we handle these:
> > >> Below I propose an approach for how to handle each of these. Feedback
> > >> welcome.
> > >>
> > >> 1) Moving the code
> > >> I propose that HCat move into a subdirectory of Hive. This fits nicely
> > >> into Hive's structure since it already has metastore, ql, etc. We'd just
> > >> add 'hcatalog' as a new directory. This directory would contain hcatalog
> > >> as it is today. It does not follow Hive's standard build model so we'd
> > >> need to do some work to make it so that building Hive also builds HCat, but
> > >> this should be minimal.
> > >>
> > >> 2) Dealing with domain names
> > >> HCat code currently is under org.apache.hcatalog. Do we want to change
> > >> it? In time we probably should change it to match the rest of Hive
> > >> (org.apache.hadoop.hive.hcatalog). We need to do this in a backward
> > >> compatible way. I propose we leave it as is for now and if we decide to in
> > >> the future we can move the actual code to org.apache.hadoop.hive.hcatalog
> > >> and create shell classes under org.apache.hcatalog.
> > >>
> > >> 3) The mailing lists
> > >> Given that our goal is to merge the projects and not create a subproject
> > >> we should merge the mailing lists rather than keep hcat specific lists. We
> > >> can ask infra to remove hcatalog-*@incubator.apache.org and forward any
> > >> new mail to the appropriate Hive lists. We need to find out if they can
> > >> auto-subscribe people from the hcat lists to the hive lists. Given that
> > >> traffic on the Hive lists is an order of magnitude higher we should warn
> > >> people before we auto-subscribe them and allow them a chance to get off.
> > >>
> > >> 4) JIRA
> > >> We can create an hcatalog component in Hive's JIRA. All new HCat issues
> > >> could be filed there. I don't know if there's a way to upload existing
> > >> JIRAs into Hive's JIRA, but I think it would be better to leave them where
> > >> they are. We should see if infra can turn off the ability to create new
> > >> JIRAs in hcatalog.
> > >>
> > >> 5) Website
> > >> We will need to integrate HCatalog's website with Hive's. This should be
> > >> easy except for the documentation. HCat uses forrest for docs, Hive uses
> > >> wiki. We will need to put links under 'Documentation' for older versions
> > >> of HCat docs so users can find them. As far as how docs are handled for
> > >> the next version of HCatalog, I think that depends on the answer to
> > >> question 7 (next release of HCat), but I propose that HCat needs to conform
> > >> to the way Hive does docs on wiki. Though I would strongly encourage the
> > >> HCat docs to be version specific (that is, have a set of wiki pages for
> > >> each version). incubator.apache.org/hcatalog should be changed to
> > >> forward to hive.apache.org.
> > >>
> > >> 6) Committer rights
> > >> Carl will need to set up committer rights for all the new HCat committers.
> > >> Based on our discussion of making active HCat committers Hive submodule
> > >> committers this would add the following set: Alan, Sushanth, Francis,
> > >> Daniel, Vandana, Travis, and Mithun. Ashutosh and Paul are already Hive
> > >> committers, and neither Devaraj nor Mac have been active in HCat in over a
> > >> year.
> > >>
> > >> 7) Future releases
> > >> We need to discuss how future releases will happen, as I think this will
> > >> help developers and users know how to respond to the merge. I propose that
> > >> HCat will simply become part of future Hive releases. Thus Hive 0.11 (or
> > >> whatever the next major release is) will include HCatalog. If there are
> > >> issues found we may need to make HCatalog 0.5.x releases from Hive, which
> > >> should be fine. But I propose there would not be an HCat 0.6. To be clear
> > >> I am not proposing that HCat functionality would be subsumed into Hive
> > >> jars. Just that the existing hcat jars would become part of Hive's release.
> > >>
> > >> 8) Communicate all of this
> > >> We should put up an FAQ page that has this information, as well as tracks
> > >> our progress while we work on getting these things done.
> > >>
> > >> Alan.
> > >
> > >
> > >
> > >
> > > --
> > > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
> >
> >
>
>
Re: Merging HCatalog into Hive
Posted by Carl Steinbach <cw...@gmail.com>.
Hi Alan,
I committed HIVE-4145, created an HCatalog component on JIRA, and
updated the asf-authorization-template to give the HCatalog committers
karma on the hcatalog subdirectory. At this point I think everything should
be ready to go. Let me know if you run into any problems.
Thanks.
Carl
On Wed, Mar 13, 2013 at 11:56 AM, Alan Gates <ga...@hortonworks.com> wrote:
> Proposed changes look good to me. And you don't need an infra ticket to
> grant karma. Since you're Hive VP you can do it. See
> http://www.apache.org/dev/pmc.html#SVNaccess
>
> Alan.
>
> On Mar 10, 2013, at 9:29 PM, Carl Steinbach wrote:
>
> > Hi Alan,
> >
> > I submitted a patch that creates the hcatalog directory and makes some
> other necessary
> > changes here:
> >
> > https://issues.apache.org/jira/browse/HIVE-4145
> >
> > Once this is committed I will contact ASFINFRA and ask them to grant the
> HCatalog
> > committers karma.
> >
> > Thanks.
> >
> > Carl
> >
> > On Sat, Mar 9, 2013 at 12:54 PM, Alan Gates <ga...@hortonworks.com>
> wrote:
> > Alright, I've gotten some feedback from Brock around the JIRA stuff and
> Carl in a live conversation expressed his desire to move hcat into the Hive
> namespace sooner rather than later. So the proposal is that we'd move the
> code to org.apache.hive.hcatalog, though we would create shell classes and
> interfaces in org.apache.hcatalog for all public classes and interfaces so
> that it will be backward compatible. I'm fine with doing this now.
> >
> > So, let's get started. Carl, could you create an hcatalog directory
> under trunk/hive and grant the listed hcat committers karma on it? Then
> I'll get started on moving the actual code.
> >
> > Alan.
> >
> > On Feb 24, 2013, at 12:22 PM, Brock Noland wrote:
> >
> > > Looks good from my perspective and I glad to see this moving forward.
> > >
> > > Regarding #4 (JIRA)
> > >
> > > "I don't know if there's a way to upload existing JIRAs into Hive's
> JIRA,
> > > but I think it would be better to leave them where they are."
> > >
> > > JIRA has a bulk move feature, but I am curious as why we would leave
> them
> > > under the old project? There might be good reason to orphan them, but
> my
> > > first thought is that it would be nice to have them under the HIVE
> project
> > > simply for search purposes.
> > >
> > > Brock
> > >
> > >
> > >
> > >
> > > On Fri, Feb 22, 2013 at 7:12 PM, Alan Gates <ga...@hortonworks.com>
> wrote:
> > >
> > >> Alright, our vote has passed, it's time to get on with merging
> HCatalog
> > >> into Hive. Here's the things I can think of we need to deal with.
> Please
> > >> add additional issues I've missed:
> > >>
> > >> 1) Moving the code
> > >> 2) Dealing with domain names in the code
> > >> 3) The mailing lists
> > >> 4) The JIRA
> > >> 5) The website
> > >> 6) Committer rights
> > >> 7) Make a proposal for how HCat is released going forward
> > >> 8) Publish an FAQ
> > >>
> > >> Proposals for how we handle these:
> > >> Below I propose an approach for how to handle each of these. Feedback
> > >> welcome.
> > >>
> > >> 1) Moving the code
> > >> I propose that HCat move into a subdirectory of Hive. This fits
> nicely
> > >> into Hive's structure since it already has metastore, ql, etc. We'd
> just
> > >> add 'hcatalog' as a new directory. This directory would contain
> hcatalog
> > >> as it is today. It does not follow Hive's standard build model so
> we'd
> > >> need to do some work to make it so that building Hive also builds
> HCat, but
> > >> this should be minimal.
> > >>
> > >> 2) Dealing with domain names
> > >> HCat code currently is under org.apache.hcatalog. Do we want to
> change
> > >> it? In time we probably should change it to match the rest of Hive
> > >> (org.apache.hadoop.hive.hcatalog). We need to do this in a backward
> > >> compatible way. I propose we leave it as is for now and if we decide
> to in
> > >> the future we can move the actual code to
> org.apache.hadoop.hive.hcatalog
> > >> and create shell classes under org.apache.hcatalog.
> > >>
> > >> 3) The mailing lists
> > >> Given that our goal is to merge the projects and not create a
> subproject
> > >> we should merge the mailing lists rather than keep hcat specific
> lists. We
> > >> can ask infra to remove hcatalog-*@incubator.apache.org and forward
> any
> > >> new mail to the appropriate Hive lists. We need to find out if they
> can
> > >> auto-subscribe people from the hcat lists to the hive lists. Given
> that
> > >> traffic on the Hive lists is an order of magnitude higher we should
> warn
> > >> people before we auto-subscribe them and allow them a chance to get
> off.
> > >>
> > >> 4) JIRA
> > >> We can create an hcatalog component in Hive's JIRA. All new HCat
> issues
> > >> could be filed there. I don't know if there's a way to upload
> existing
> > >> JIRAs into Hive's JIRA, but I think it would be better to leave them
> where
> > >> they are. We should see if infra can turn off the ability to create
> new
> > >> JIRAs in hcatalog.
> > >>
> > >> 5) Website
> > >> We will need to integrate HCatalog's website with Hive's. This
> should be
> > >> easy except for the documentation. HCat uses forrest for docs, Hive
> uses
> > >> wiki. We will need to put links under 'Documentation' for older
> versions
> > >> of HCat docs so users can find them. As far as how docs are handled
> for
> > >> the next version of HCatalog, I think that depends on the answer to
> > >> question 7 (next release of HCat), but I propose that HCat needs to
> conform
> > >> to the way Hive does docs on wiki. Though I would strongly encourage
> the
> > >> HCat docs to be version specific (that is, have a set of wiki pages
> for
> > >> each version). incubator.apache.org/hcatalog should be changed to
> > >> forward to hive.apache.org.
> > >>
> > >> 6) Committer rights
> > >> Carl will need to set up committer rights for all the new HCat
> committers.
> > >> Based on our discussion of making active HCat committers Hive
> submodule
> > >> committers this would add the following set: Alan, Sushanth, Francis,
> > >> Daniel, Vandana, Travis, and Mithun. Ashutosh and Paul are already
> Hive
> > >> committers, and neither Devaraj nor Mac have been active in HCat in
> over a
> > >> year.
> > >>
> > >> 7) Future releases
> > >> We need to discuss how future releases will happen, as I think this
> will
> > >> help developers and users know how to respond to the merge. I
> propose that
> > >> HCat will simply become part of future Hive releases. Thus Hive 0.11
> (or
> > >> whatever the next major release is) will include HCatalog. If there
> are
> > >> issues found we may need to make HCatalog 0.5.x releases from Hive,
> which
> > >> should be fine. But I propose there would not be an HCat 0.6. To be
> clear
> > >> I am not proposing that HCat functionality would be subsumed into Hive
> > >> jars. Just that the existing hcat jars would become part of Hive's
> release.
> > >>
> > >> 8) Communicate all of this
> > >> We should put up an FAQ page that has this information, as well as
> tracks
> > >> our progress while we work on getting these things done.
> > >>
> > >> Alan.
> > >
> > >
> > >
> > >
> > > --
> > > Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
> >
> >
>
>
Re: Merging HCatalog into Hive
Posted by Alan Gates <ga...@hortonworks.com>.
Proposed changes look good to me. And you don't need an infra ticket to grant karma. Since you're Hive VP you can do it. See http://www.apache.org/dev/pmc.html#SVNaccess
Alan.
On Mar 10, 2013, at 9:29 PM, Carl Steinbach wrote:
> Hi Alan,
>
> I submitted a patch that creates the hcatalog directory and makes some other necessary
> changes here:
>
> https://issues.apache.org/jira/browse/HIVE-4145
>
> Once this is committed I will contact ASFINFRA and ask them to grant the HCatalog
> committers karma.
>
> Thanks.
>
> Carl
>
> On Sat, Mar 9, 2013 at 12:54 PM, Alan Gates <ga...@hortonworks.com> wrote:
> Alright, I've gotten some feedback from Brock around the JIRA stuff and Carl in a live conversation expressed his desire to move hcat into the Hive namespace sooner rather than later. So the proposal is that we'd move the code to org.apache.hive.hcatalog, though we would create shell classes and interfaces in org.apache.hcatalog for all public classes and interfaces so that it will be backward compatible. I'm fine with doing this now.
>
> So, let's get started. Carl, could you create an hcatalog directory under trunk/hive and grant the listed hcat committers karma on it? Then I'll get started on moving the actual code.
>
> Alan.
>
> On Feb 24, 2013, at 12:22 PM, Brock Noland wrote:
>
> > Looks good from my perspective and I glad to see this moving forward.
> >
> > Regarding #4 (JIRA)
> >
> > "I don't know if there's a way to upload existing JIRAs into Hive's JIRA,
> > but I think it would be better to leave them where they are."
> >
> > JIRA has a bulk move feature, but I am curious as why we would leave them
> > under the old project? There might be good reason to orphan them, but my
> > first thought is that it would be nice to have them under the HIVE project
> > simply for search purposes.
> >
> > Brock
> >
> >
> >
> >
> > On Fri, Feb 22, 2013 at 7:12 PM, Alan Gates <ga...@hortonworks.com> wrote:
> >
> >> Alright, our vote has passed, it's time to get on with merging HCatalog
> >> into Hive. Here's the things I can think of we need to deal with. Please
> >> add additional issues I've missed:
> >>
> >> 1) Moving the code
> >> 2) Dealing with domain names in the code
> >> 3) The mailing lists
> >> 4) The JIRA
> >> 5) The website
> >> 6) Committer rights
> >> 7) Make a proposal for how HCat is released going forward
> >> 8) Publish an FAQ
> >>
> >> Proposals for how we handle these:
> >> Below I propose an approach for how to handle each of these. Feedback
> >> welcome.
> >>
> >> 1) Moving the code
> >> I propose that HCat move into a subdirectory of Hive. This fits nicely
> >> into Hive's structure since it already has metastore, ql, etc. We'd just
> >> add 'hcatalog' as a new directory. This directory would contain hcatalog
> >> as it is today. It does not follow Hive's standard build model so we'd
> >> need to do some work to make it so that building Hive also builds HCat, but
> >> this should be minimal.
> >>
> >> 2) Dealing with domain names
> >> HCat code currently is under org.apache.hcatalog. Do we want to change
> >> it? In time we probably should change it to match the rest of Hive
> >> (org.apache.hadoop.hive.hcatalog). We need to do this in a backward
> >> compatible way. I propose we leave it as is for now and if we decide to in
> >> the future we can move the actual code to org.apache.hadoop.hive.hcatalog
> >> and create shell classes under org.apache.hcatalog.
> >>
> >> 3) The mailing lists
> >> Given that our goal is to merge the projects and not create a subproject
> >> we should merge the mailing lists rather than keep hcat specific lists. We
> >> can ask infra to remove hcatalog-*@incubator.apache.org and forward any
> >> new mail to the appropriate Hive lists. We need to find out if they can
> >> auto-subscribe people from the hcat lists to the hive lists. Given that
> >> traffic on the Hive lists is an order of magnitude higher we should warn
> >> people before we auto-subscribe them and allow them a chance to get off.
> >>
> >> 4) JIRA
> >> We can create an hcatalog component in Hive's JIRA. All new HCat issues
> >> could be filed there. I don't know if there's a way to upload existing
> >> JIRAs into Hive's JIRA, but I think it would be better to leave them where
> >> they are. We should see if infra can turn off the ability to create new
> >> JIRAs in hcatalog.
> >>
> >> 5) Website
> >> We will need to integrate HCatalog's website with Hive's. This should be
> >> easy except for the documentation. HCat uses forrest for docs, Hive uses
> >> wiki. We will need to put links under 'Documentation' for older versions
> >> of HCat docs so users can find them. As far as how docs are handled for
> >> the next version of HCatalog, I think that depends on the answer to
> >> question 7 (next release of HCat), but I propose that HCat needs to conform
> >> to the way Hive does docs on wiki. Though I would strongly encourage the
> >> HCat docs to be version specific (that is, have a set of wiki pages for
> >> each version). incubator.apache.org/hcatalog should be changed to
> >> forward to hive.apache.org.
> >>
> >> 6) Committer rights
> >> Carl will need to set up committer rights for all the new HCat committers.
> >> Based on our discussion of making active HCat committers Hive submodule
> >> committers this would add the following set: Alan, Sushanth, Francis,
> >> Daniel, Vandana, Travis, and Mithun. Ashutosh and Paul are already Hive
> >> committers, and neither Devaraj nor Mac have been active in HCat in over a
> >> year.
> >>
> >> 7) Future releases
> >> We need to discuss how future releases will happen, as I think this will
> >> help developers and users know how to respond to the merge. I propose that
> >> HCat will simply become part of future Hive releases. Thus Hive 0.11 (or
> >> whatever the next major release is) will include HCatalog. If there are
> >> issues found we may need to make HCatalog 0.5.x releases from Hive, which
> >> should be fine. But I propose there would not be an HCat 0.6. To be clear
> >> I am not proposing that HCat functionality would be subsumed into Hive
> >> jars. Just that the existing hcat jars would become part of Hive's release.
> >>
> >> 8) Communicate all of this
> >> We should put up an FAQ page that has this information, as well as tracks
> >> our progress while we work on getting these things done.
> >>
> >> Alan.
> >
> >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
>
>
Re: Merging HCatalog into Hive
Posted by Carl Steinbach <cw...@apache.org>.
Hi Alan,
I submitted a patch that creates the hcatalog directory and makes some
other necessary
changes here:
https://issues.apache.org/jira/browse/HIVE-4145
Once this is committed I will contact ASFINFRA and ask them to grant the
HCatalog
committers karma.
Thanks.
Carl
On Sat, Mar 9, 2013 at 12:54 PM, Alan Gates <ga...@hortonworks.com> wrote:
> Alright, I've gotten some feedback from Brock around the JIRA stuff and
> Carl in a live conversation expressed his desire to move hcat into the Hive
> namespace sooner rather than later. So the proposal is that we'd move the
> code to org.apache.hive.hcatalog, though we would create shell classes and
> interfaces in org.apache.hcatalog for all public classes and interfaces so
> that it will be backward compatible. I'm fine with doing this now.
>
> So, let's get started. Carl, could you create an hcatalog directory under
> trunk/hive and grant the listed hcat committers karma on it? Then I'll get
> started on moving the actual code.
>
> Alan.
>
> On Feb 24, 2013, at 12:22 PM, Brock Noland wrote:
>
> > Looks good from my perspective and I glad to see this moving forward.
> >
> > Regarding #4 (JIRA)
> >
> > "I don't know if there's a way to upload existing JIRAs into Hive's JIRA,
> > but I think it would be better to leave them where they are."
> >
> > JIRA has a bulk move feature, but I am curious as why we would leave them
> > under the old project? There might be good reason to orphan them, but my
> > first thought is that it would be nice to have them under the HIVE
> project
> > simply for search purposes.
> >
> > Brock
> >
> >
> >
> >
> > On Fri, Feb 22, 2013 at 7:12 PM, Alan Gates <ga...@hortonworks.com>
> wrote:
> >
> >> Alright, our vote has passed, it's time to get on with merging HCatalog
> >> into Hive. Here's the things I can think of we need to deal with.
> Please
> >> add additional issues I've missed:
> >>
> >> 1) Moving the code
> >> 2) Dealing with domain names in the code
> >> 3) The mailing lists
> >> 4) The JIRA
> >> 5) The website
> >> 6) Committer rights
> >> 7) Make a proposal for how HCat is released going forward
> >> 8) Publish an FAQ
> >>
> >> Proposals for how we handle these:
> >> Below I propose an approach for how to handle each of these. Feedback
> >> welcome.
> >>
> >> 1) Moving the code
> >> I propose that HCat move into a subdirectory of Hive. This fits nicely
> >> into Hive's structure since it already has metastore, ql, etc. We'd
> just
> >> add 'hcatalog' as a new directory. This directory would contain
> hcatalog
> >> as it is today. It does not follow Hive's standard build model so we'd
> >> need to do some work to make it so that building Hive also builds HCat,
> but
> >> this should be minimal.
> >>
> >> 2) Dealing with domain names
> >> HCat code currently is under org.apache.hcatalog. Do we want to change
> >> it? In time we probably should change it to match the rest of Hive
> >> (org.apache.hadoop.hive.hcatalog). We need to do this in a backward
> >> compatible way. I propose we leave it as is for now and if we decide
> to in
> >> the future we can move the actual code to
> org.apache.hadoop.hive.hcatalog
> >> and create shell classes under org.apache.hcatalog.
> >>
> >> 3) The mailing lists
> >> Given that our goal is to merge the projects and not create a subproject
> >> we should merge the mailing lists rather than keep hcat specific lists.
> We
> >> can ask infra to remove hcatalog-*@incubator.apache.org and forward any
> >> new mail to the appropriate Hive lists. We need to find out if they can
> >> auto-subscribe people from the hcat lists to the hive lists. Given that
> >> traffic on the Hive lists is an order of magnitude higher we should warn
> >> people before we auto-subscribe them and allow them a chance to get off.
> >>
> >> 4) JIRA
> >> We can create an hcatalog component in Hive's JIRA. All new HCat issues
> >> could be filed there. I don't know if there's a way to upload existing
> >> JIRAs into Hive's JIRA, but I think it would be better to leave them
> where
> >> they are. We should see if infra can turn off the ability to create new
> >> JIRAs in hcatalog.
> >>
> >> 5) Website
> >> We will need to integrate HCatalog's website with Hive's. This should
> be
> >> easy except for the documentation. HCat uses forrest for docs, Hive
> uses
> >> wiki. We will need to put links under 'Documentation' for older
> versions
> >> of HCat docs so users can find them. As far as how docs are handled for
> >> the next version of HCatalog, I think that depends on the answer to
> >> question 7 (next release of HCat), but I propose that HCat needs to
> conform
> >> to the way Hive does docs on wiki. Though I would strongly encourage
> the
> >> HCat docs to be version specific (that is, have a set of wiki pages for
> >> each version). incubator.apache.org/hcatalog should be changed to
> >> forward to hive.apache.org.
> >>
> >> 6) Committer rights
> >> Carl will need to set up committer rights for all the new HCat
> committers.
> >> Based on our discussion of making active HCat committers Hive submodule
> >> committers this would add the following set: Alan, Sushanth, Francis,
> >> Daniel, Vandana, Travis, and Mithun. Ashutosh and Paul are already Hive
> >> committers, and neither Devaraj nor Mac have been active in HCat in
> over a
> >> year.
> >>
> >> 7) Future releases
> >> We need to discuss how future releases will happen, as I think this will
> >> help developers and users know how to respond to the merge. I propose
> that
> >> HCat will simply become part of future Hive releases. Thus Hive 0.11
> (or
> >> whatever the next major release is) will include HCatalog. If there are
> >> issues found we may need to make HCatalog 0.5.x releases from Hive,
> which
> >> should be fine. But I propose there would not be an HCat 0.6. To be
> clear
> >> I am not proposing that HCat functionality would be subsumed into Hive
> >> jars. Just that the existing hcat jars would become part of Hive's
> release.
> >>
> >> 8) Communicate all of this
> >> We should put up an FAQ page that has this information, as well as
> tracks
> >> our progress while we work on getting these things done.
> >>
> >> Alan.
> >
> >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
>