You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Owen O'Malley <ow...@gmail.com> on 2017/07/01 00:05:12 UTC
Re: [DISCUSS] Separating out the metastore as its own TLP
On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <su...@apache.org> wrote:
> and maybe a different project name?
>
Yes, it certainly needs a new name. I'd like to suggest Riven.
.. Owen
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Eugene Koifman <ek...@hortonworks.com>.
This rings very true to me
On 7/3/17, 6:20 AM, "Edward Capriolo" <ed...@gmail.com> wrote:
I am going to give a hypothetical but real world situation. Suppose I want
to add the statement "CREATE permanent macro xyz", this feature I believe
would cross cut calcite, hive, and hive metastore. To build this feature I
would need to orchestrate the change across 3 separate groups of hive
'subcommittees' for lack of a better word. 3 git repos, 3 Jira's 3
releases. That is not counting if we run into some bug or misfeature (maybe
with Tez or something else) so that brings in 4-5 releases of upstream to
add a feature to hive. This does not take into account normal processes
mess ups. For example say you get the metastore done, but now the people
doing the calcite/antlr suggest the feature have different syntax because
they did not read the 3-4 linked tickets when the process started? Now, you
have to loop back around the process. Finding 1 person in 1 project to
usher along the feature you want is difficult, having to find and clear
time with 3 people across three projects is going to be a difficult along
with then 'pushing' them all to kick out a release so you can finally use
said feature.
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Alan Gates <al...@gmail.com>.
On Wed, Jul 5, 2017 at 12:33 PM, Edward Capriolo <ed...@gmail.com>
wrote:
>
> I hate to draw in something else but I feel it is related:
>
> 8 December 2016 : release 2.1.1 available
> 07 April 2017 : release 1.2.2 available
> hive-dev [DISCUSS] Supporting Hadoop-1 and experimental features
> hive-dev Re: release chaos?
>
> I have been vocal about not liking certain branching strategies and
> proposals that take us away from releasable trunk. We have steadily headed
> in a direction where we are pulling things out of hive, and we are not able
> to turn out releases. We even had a thread "release chaos" talking about
> our 5 active branches (with friends I say "jumped the shark"). Pulling out
> the metastore is only going to make this worse. I do not even see the model
> as successful. You may say it is great that calcite lets people share our
> sql dialect or the ORC TLP has 5 committers, but if Hive can not get a
> release out the door I do not see us optimizing for the proper thing.
>
I don’t see the relationship between these issues. I agree Hive is not
releasing frequently enough. I agree that breaking out the metastore won’t
fix that. It isn’t intended to fix that. But as stated elsewhere in this
thread there are ways to make sure it won’t make it any worse either.
Alan.
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Alan Gates <al...@gmail.com>.
The proposal does build in a time delay by doing the separation first in
the Hive project. This will take several months. We should not put an
absolute bound on it (6 months or whatever). The reasoning behind doing
the separation in Hive before moving the code out is to make sure it’s
feasible and to better understand the issues the separation will create.
Alan.
On Wed, Jul 5, 2017 at 1:16 PM, Xuefu Zhang <xu...@apache.org> wrote:
> I think Edward's concern is valid. While I voiced my support for this
> proposal, which was more from the benefits of the whole Hadoop ecosystem, I
> don't see the equal benefits for Hive. Instead, it may even create more
> overhead for Hive. I'd really like to take time to see what are the road
> blocks for other projects to use HMS as it is. The issue of Spark including
> a Hive fork, which was brought up some time back, is certainly not one of
> them.
>
> Thanks,
> Xuefu
>
> On Wed, Jul 5, 2017 at 12:33 PM, Edward Capriolo <ed...@gmail.com>
> wrote:
>
> > On Wed, Jul 5, 2017 at 1:51 PM, Alan Gates <al...@gmail.com> wrote:
> >
> > > On Mon, Jul 3, 2017 at 6:20 AM, Edward Capriolo <edlinuxguru@gmail.com
> >
> > > wrote:
> > >
> > > >
> > > > We already have things in the meta-store not directly tied to
> language
> > > > features. For example hive metastore has a "retention" property which
> > is
> > > > not actively in use by anything. In reality, we rarely say 'no' or -1
> > to
> > > > much. Which in part is why I believe our release process is grinding
> > > > slower: we have so many things in flight I do not feel that any one
> > > person
> > > > can keep track. You are working on porting the metastore to hbase.
> > > > https://issues.apache.org/jira/browse/HIVE-9452 did you get a -1 or
> > 'No'
> > > > along the way? When I first noticed this I pointed out that someone
> has
> > > > already ported the metastore to Cassandra
> > > > https://github.com/riptano/brisk/blob/master/src/java/
> > > > src/org/apache/cassandra/hadoop/hive/metastore/SchemaManager
> > > Service.java,
> > > > but I was more exciting/rational for this multi-year approach using
> > hbase
> > > > so I let everyone 'have at it'.
> > > >
> > > Your example and mine are not equivalent. The HBase metastore is
> still a
> > > Hive feature, even if some thought it not worth while. That is
> different
> > > than people bringing features that will never interest Hive or that
> Hive
> > > could never use (e.g. Dain’s desire for the metastore to support Presto
> > > style views).
> > >
> > > I forgot to mention the issue these would be non-Hive contributors have
> > > with releases if they contribute their features to the metastore while
> > it’s
> > > inside Hive. Is Hive going to do a release just to push out features
> in
> > > the metastore that it doesn’t care about?
> > >
> > > You seem to be asserting that doing this doesn’t really help non-Hive
> > based
> > > systems that are using or would like to use the metastore. But it is
> > > interesting that people from three of those systems have commented in
> the
> > > thread so far, and all are positive (Dmitrias from Impala, Dain from
> > > Presto, and Sriharsha from the schema registry project).
> > >
> > >
> > > > I am going to give a hypothetical but real world situation. Suppose I
> > > want
> > > > to add the statement "CREATE permanent macro xyz", this feature I
> > believe
> > > > would cross cut calcite, hive, and hive metastore. To build this
> > feature
> > > I
> > > > would need to orchestrate the change across 3 separate groups of hive
> > > > 'subcommittees' for lack of a better word. 3 git repos, 3 Jira's 3
> > > > releases. That is not counting if we run into some bug or misfeature
> > > (maybe
> > > > with Tez or something else) so that brings in 4-5 releases of
> upstream
> > to
> > > > add a feature to hive. This does not take into account normal
> processes
> > > > mess ups. For example say you get the metastore done, but now the
> > people
> > > > doing the calcite/antlr suggest the feature have different syntax
> > because
> > > > they did not read the 3-4 linked tickets when the process started?
> Now,
> > > you
> > > > have to loop back around the process. Finding 1 person in 1 project
> to
> > > > usher along the feature you want is difficult, having to find and
> clear
> > > > time with 3 people across three projects is going to be a difficult
> > along
> > > > with then 'pushing' them all to kick out a release so you can finally
> > use
> > > > said feature.
> > > >
> > >
> > > I partially agree with you. On the reviews, JIRAs, etc. I don’t think
> it
> > > adds much, if any, overhead. Hive is a big project and no one person
> > knows
> > > all the code anymore. If you wanted to add a permanent macros feature
> > you
> > > would need reviews from someone who knows the parser (probably
> > Pengcheng),
> > > people who know the optimizer (Jesus, Ashutosh, …), and someone who
> knows
> > > the metastore (me, Thejas, …). And any large feature is going to be
> > > implemented over multiple JIRAs, all of which are linkable regardless
> of
> > > whether the JIRAs start with METASTORE- or HIVE-. I also don’t think
> it
> > > makes the feature disagreement any worse. If the optimizer team
> > absolutely
> > > insists it has to have some feature and the metastore team insists that
> > it
> > > can’t have that feature you’re going to have to work through the issue
> > > whether they all are in Hive or in two separate projects.
> > >
> > > Where I agree the split adds cost is releases. Before your macro
> feature
> > > could go live you need releases from each of the components. And while
> > in
> > > development the components need to use snapshot versions of the other
> > > components. My assertion is that the benefits out weigh this cost.
> > >
> > > Alan.
> > >
> >
> >
> > "You seem to be asserting that doing this doesn’t really help non-Hive
> > based
> > systems that are using or would like to use the metastore. But it is
> > interesting that people from three of those systems have commented in the
> > thread so far, and all are positive (Dmitrias from Impala, Dain from
> > Presto, and Sriharsha from the schema registry project)."
> >
> > I notice that impala has a syntax for caching.
> >
> > https://www.cloudera.com/documentation/enterprise/5-8-x/topi
> > cs/impala_perf_hdfs_caching.html
> >
> > Notice how the cache syntax did not way into Hive? It would make sense if
> > this feature trickled it's way into hive and use HDFS caching for
> example.
> > I have heard many people claim that using hive metastore is such a
> because
> > it is packaged weird (like with ORC), but again besides claim/complaining
> > no one has stepped up to deal with that.
> >
> > What I would suggest is going forward for maybe a trial period of 6
> months,
> > labeling JIRA tickets with a tag that would be
> > "SeeThisProvesWeNeedATLPMetastore". Because right now I do not enough
> > active use cases of people giving anything back to justify hurting our
> > workflow so much.
> >
> >
> > "I partially agree with you. On the reviews, JIRAs, etc. I don’t think
> it
> > adds much, if any, overhead. Hive is a big project and no one person
> knows
> > all the code anymore. If you wanted to add a permanent macros feature
> you
> > would need reviews from someone who knows the parser (probably
> Pengcheng),
> > people who know the optimizer (Jesus, Ashutosh, …), and someone who knows
> > the metastore (me, Thejas, …). And any large feature is going to be
> > implemented over multiple JIRAs, all of which are linkable regardless of
> > whether the JIRAs start with METASTORE- or HIVE-. I also don’t think it
> > makes the feature disagreement any worse. If the optimizer team
> absolutely
> > insists it has to have some feature and the metastore team insists that
> it
> > can’t have that feature you’re going to have to work through the issue
> > whether they all are in Hive or in two separate projects"
> >
> > Macro was done in 1 patch and reviewed by 2 people. With 2-3 follow on
> > bugs.
> >
> > https://issues.apache.org/jira/browse/HIVE-2655
> >
> > I think your perception is different then mine because of circumstances.
> I
> > have waited weeks/months for reviews/merges (in Hive and other apache
> > projects) from mundane udfs to cassandra-storage-handlers. You obviously
> > work in a large company and you can more easily align objectives, go to
> the
> > water cooler and say "hey bob you know it would be cool if you can
> release
> > x so I can do y". When you are not in that situation its like, "hey
> mailing
> > list, my patch was done for three months now and like I have had to
> rebase
> > it three times and like I notice like other stuff is getting committed."
> >
> > If you look at it tactically, "create permanent macro xzy". I go over to
> > calcite and suggest some changes there, if this concept is not "game
> > changer" it is probably going to sit unreviewed. If it is "game changer"
> > exciting that is 72 hours for release voting. Next go to hive-metastore
> > repeat the process, but remember now I have to "wow" the metastore people
> > with the "game changer" and if that crew is super focused on something
> > about kafka well now Hive features are second fiddle. Now lets say a hive
> > release is coming up, and I really want my feature in it.
> > hive-metastore-tlp might currently have a broken trunk because mongo
> wants
> > to add spaceships to wombats feature has a bug and frankly that should
> not
> > effect us.
> >
> > I hate to draw in something else but I feel it is related:
> >
> > 8 December 2016 : release 2.1.1 available
> > 07 April 2017 : release 1.2.2 available
> > hive-dev [DISCUSS] Supporting Hadoop-1 and experimental features
> > hive-dev Re: release chaos?
> >
> > I have been vocal about not liking certain branching strategies and
> > proposals that take us away from releasable trunk. We have steadily
> headed
> > in a direction where we are pulling things out of hive, and we are not
> able
> > to turn out releases. We even had a thread "release chaos" talking about
> > our 5 active branches (with friends I say "jumped the shark"). Pulling
> out
> > the metastore is only going to make this worse. I do not even see the
> model
> > as successful. You may say it is great that calcite lets people share our
> > sql dialect or the ORC TLP has 5 committers, but if Hive can not get a
> > release out the door I do not see us optimizing for the proper thing.
> >
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Alan Gates <al...@gmail.com>.
+1 to having an always releasable head of master.
+1 to having test verified API compliance. I was thinking that the project
should set up verification tests where it runs against supported versions
of Hive, Imapala, Spark, … (and obviously open to others to add their tests
as well) on a nightly basis so that we guarantee API stability.
Alan.
On Thu, Jul 6, 2017 at 2:10 AM, Peter Vary <pv...@cloudera.com> wrote:
> Hi folks,
>
> I agree with most of the things Edward said. I have faced similar issues
> in smaller scale when integrated Hive with Yetus. We are forced to keep
> patched Yetus files in Hive repo until they push their next release. Also
> followed one more serious problem when a patch was committed to Hive,
> Impala and Spark, and just few days before the release all of them was
> reverted from the projects due to concerns raised by the Spark committee
> (after the changes was already committed to Spark as well)
>
> Having said all of these, I still think that separating the HMS to a new
> top level project could be a step to the right direction with the following
> constraints. The new project should have:
> - Strict, stability oriented branching strategy following Edward's
> suggestions, so if a downstream project - for example Hive - needs some fix
> or easy change that could be incorporated, and released almost immediately.
> So we have to have these:
> - Always releasable head
> - Every multi commit feature should be added as a feature branch
> - Strict, enforced, stability oriented API strategy. So we will not be
> surprised by features added by other projects and break Hive compatibility.
> To avoid this situation we need to design for it, have pre-commit tests in
> place for catch the in-adverted changes, and most importantly have a clear
> commitment for it.
>
> I think, since the current HMS is already used by numerous other projects,
> we already should have these in mind when modifying anything in HMS related
> code. This is not the main focus of Hive, so we do not concentrate on this
> and there are often interoperability issues, problems. We can do this
> inside Hive as well, but the current approach followed by Hive, and the one
> required by the HMS are requiring a different mindset. We need a clear,
> well defined boundary and separating the 2 projects could help in this. We
> can focus on the different needs and goal and eventually we might have
> different culture as well which suits the specific needs of the specific
> part of the code.
>
> I think keeping these rules in the new to level HMS we can mitigate most
> of the issues mentioned below, and we will be better of overall.
> What do you think Edward?
>
> Thanks,
> Peter
>
>
> > On Jul 5, 2017, at 10:16 PM, Xuefu Zhang <xu...@apache.org> wrote:
> >
> > I think Edward's concern is valid. While I voiced my support for this
> > proposal, which was more from the benefits of the whole Hadoop
> ecosystem, I
> > don't see the equal benefits for Hive. Instead, it may even create more
> > overhead for Hive. I'd really like to take time to see what are the road
> > blocks for other projects to use HMS as it is. The issue of Spark
> including
> > a Hive fork, which was brought up some time back, is certainly not one of
> > them.
> >
> > Thanks,
> > Xuefu
> >
> > On Wed, Jul 5, 2017 at 12:33 PM, Edward Capriolo <ed...@gmail.com>
> > wrote:
> >
> >> On Wed, Jul 5, 2017 at 1:51 PM, Alan Gates <al...@gmail.com>
> wrote:
> >>
> >>> On Mon, Jul 3, 2017 at 6:20 AM, Edward Capriolo <edlinuxguru@gmail.com
> >
> >>> wrote:
> >>>
> >>>>
> >>>> We already have things in the meta-store not directly tied to language
> >>>> features. For example hive metastore has a "retention" property which
> >> is
> >>>> not actively in use by anything. In reality, we rarely say 'no' or -1
> >> to
> >>>> much. Which in part is why I believe our release process is grinding
> >>>> slower: we have so many things in flight I do not feel that any one
> >>> person
> >>>> can keep track. You are working on porting the metastore to hbase.
> >>>> https://issues.apache.org/jira/browse/HIVE-9452 did you get a -1 or
> >> 'No'
> >>>> along the way? When I first noticed this I pointed out that someone
> has
> >>>> already ported the metastore to Cassandra
> >>>> https://github.com/riptano/brisk/blob/master/src/java/
> >>>> src/org/apache/cassandra/hadoop/hive/metastore/SchemaManager
> >>> Service.java,
> >>>> but I was more exciting/rational for this multi-year approach using
> >> hbase
> >>>> so I let everyone 'have at it'.
> >>>>
> >>> Your example and mine are not equivalent. The HBase metastore is
> still a
> >>> Hive feature, even if some thought it not worth while. That is
> different
> >>> than people bringing features that will never interest Hive or that
> Hive
> >>> could never use (e.g. Dain’s desire for the metastore to support Presto
> >>> style views).
> >>>
> >>> I forgot to mention the issue these would be non-Hive contributors have
> >>> with releases if they contribute their features to the metastore while
> >> it’s
> >>> inside Hive. Is Hive going to do a release just to push out features
> in
> >>> the metastore that it doesn’t care about?
> >>>
> >>> You seem to be asserting that doing this doesn’t really help non-Hive
> >> based
> >>> systems that are using or would like to use the metastore. But it is
> >>> interesting that people from three of those systems have commented in
> the
> >>> thread so far, and all are positive (Dmitrias from Impala, Dain from
> >>> Presto, and Sriharsha from the schema registry project).
> >>>
> >>>
> >>>> I am going to give a hypothetical but real world situation. Suppose I
> >>> want
> >>>> to add the statement "CREATE permanent macro xyz", this feature I
> >> believe
> >>>> would cross cut calcite, hive, and hive metastore. To build this
> >> feature
> >>> I
> >>>> would need to orchestrate the change across 3 separate groups of hive
> >>>> 'subcommittees' for lack of a better word. 3 git repos, 3 Jira's 3
> >>>> releases. That is not counting if we run into some bug or misfeature
> >>> (maybe
> >>>> with Tez or something else) so that brings in 4-5 releases of upstream
> >> to
> >>>> add a feature to hive. This does not take into account normal
> processes
> >>>> mess ups. For example say you get the metastore done, but now the
> >> people
> >>>> doing the calcite/antlr suggest the feature have different syntax
> >> because
> >>>> they did not read the 3-4 linked tickets when the process started?
> Now,
> >>> you
> >>>> have to loop back around the process. Finding 1 person in 1 project to
> >>>> usher along the feature you want is difficult, having to find and
> clear
> >>>> time with 3 people across three projects is going to be a difficult
> >> along
> >>>> with then 'pushing' them all to kick out a release so you can finally
> >> use
> >>>> said feature.
> >>>>
> >>>
> >>> I partially agree with you. On the reviews, JIRAs, etc. I don’t think
> it
> >>> adds much, if any, overhead. Hive is a big project and no one person
> >> knows
> >>> all the code anymore. If you wanted to add a permanent macros feature
> >> you
> >>> would need reviews from someone who knows the parser (probably
> >> Pengcheng),
> >>> people who know the optimizer (Jesus, Ashutosh, …), and someone who
> knows
> >>> the metastore (me, Thejas, …). And any large feature is going to be
> >>> implemented over multiple JIRAs, all of which are linkable regardless
> of
> >>> whether the JIRAs start with METASTORE- or HIVE-. I also don’t think
> it
> >>> makes the feature disagreement any worse. If the optimizer team
> >> absolutely
> >>> insists it has to have some feature and the metastore team insists that
> >> it
> >>> can’t have that feature you’re going to have to work through the issue
> >>> whether they all are in Hive or in two separate projects.
> >>>
> >>> Where I agree the split adds cost is releases. Before your macro
> feature
> >>> could go live you need releases from each of the components. And while
> >> in
> >>> development the components need to use snapshot versions of the other
> >>> components. My assertion is that the benefits out weigh this cost.
> >>>
> >>> Alan.
> >>>
> >>
> >>
> >> "You seem to be asserting that doing this doesn’t really help non-Hive
> >> based
> >> systems that are using or would like to use the metastore. But it is
> >> interesting that people from three of those systems have commented in
> the
> >> thread so far, and all are positive (Dmitrias from Impala, Dain from
> >> Presto, and Sriharsha from the schema registry project)."
> >>
> >> I notice that impala has a syntax for caching.
> >>
> >> https://www.cloudera.com/documentation/enterprise/5-8-x/topi
> >> cs/impala_perf_hdfs_caching.html
> >>
> >> Notice how the cache syntax did not way into Hive? It would make sense
> if
> >> this feature trickled it's way into hive and use HDFS caching for
> example.
> >> I have heard many people claim that using hive metastore is such a
> because
> >> it is packaged weird (like with ORC), but again besides
> claim/complaining
> >> no one has stepped up to deal with that.
> >>
> >> What I would suggest is going forward for maybe a trial period of 6
> months,
> >> labeling JIRA tickets with a tag that would be
> >> "SeeThisProvesWeNeedATLPMetastore". Because right now I do not enough
> >> active use cases of people giving anything back to justify hurting our
> >> workflow so much.
> >>
> >>
> >> "I partially agree with you. On the reviews, JIRAs, etc. I don’t think
> it
> >> adds much, if any, overhead. Hive is a big project and no one person
> knows
> >> all the code anymore. If you wanted to add a permanent macros feature
> you
> >> would need reviews from someone who knows the parser (probably
> Pengcheng),
> >> people who know the optimizer (Jesus, Ashutosh, …), and someone who
> knows
> >> the metastore (me, Thejas, …). And any large feature is going to be
> >> implemented over multiple JIRAs, all of which are linkable regardless of
> >> whether the JIRAs start with METASTORE- or HIVE-. I also don’t think
> it
> >> makes the feature disagreement any worse. If the optimizer team
> absolutely
> >> insists it has to have some feature and the metastore team insists that
> it
> >> can’t have that feature you’re going to have to work through the issue
> >> whether they all are in Hive or in two separate projects"
> >>
> >> Macro was done in 1 patch and reviewed by 2 people. With 2-3 follow on
> >> bugs.
> >>
> >> https://issues.apache.org/jira/browse/HIVE-2655
> >>
> >> I think your perception is different then mine because of
> circumstances. I
> >> have waited weeks/months for reviews/merges (in Hive and other apache
> >> projects) from mundane udfs to cassandra-storage-handlers. You obviously
> >> work in a large company and you can more easily align objectives, go to
> the
> >> water cooler and say "hey bob you know it would be cool if you can
> release
> >> x so I can do y". When you are not in that situation its like, "hey
> mailing
> >> list, my patch was done for three months now and like I have had to
> rebase
> >> it three times and like I notice like other stuff is getting committed."
> >>
> >> If you look at it tactically, "create permanent macro xzy". I go over to
> >> calcite and suggest some changes there, if this concept is not "game
> >> changer" it is probably going to sit unreviewed. If it is "game changer"
> >> exciting that is 72 hours for release voting. Next go to hive-metastore
> >> repeat the process, but remember now I have to "wow" the metastore
> people
> >> with the "game changer" and if that crew is super focused on something
> >> about kafka well now Hive features are second fiddle. Now lets say a
> hive
> >> release is coming up, and I really want my feature in it.
> >> hive-metastore-tlp might currently have a broken trunk because mongo
> wants
> >> to add spaceships to wombats feature has a bug and frankly that should
> not
> >> effect us.
> >>
> >> I hate to draw in something else but I feel it is related:
> >>
> >> 8 December 2016 : release 2.1.1 available
> >> 07 April 2017 : release 1.2.2 available
> >> hive-dev [DISCUSS] Supporting Hadoop-1 and experimental features
> >> hive-dev Re: release chaos?
> >>
> >> I have been vocal about not liking certain branching strategies and
> >> proposals that take us away from releasable trunk. We have steadily
> headed
> >> in a direction where we are pulling things out of hive, and we are not
> able
> >> to turn out releases. We even had a thread "release chaos" talking about
> >> our 5 active branches (with friends I say "jumped the shark"). Pulling
> out
> >> the metastore is only going to make this worse. I do not even see the
> model
> >> as successful. You may say it is great that calcite lets people share
> our
> >> sql dialect or the ORC TLP has 5 committers, but if Hive can not get a
> >> release out the door I do not see us optimizing for the proper thing.
> >>
>
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Vihang Karajgaonkar <vi...@cloudera.com>.
I can understand the concerns from Edward and Xuefu and I think they are
valid as well. I think having a regular cadence of release will help
alleviate the concerns related to features making into releases to a
certain extent. Having quarterly or semi-annual releases would be a good
thing in general for both Hive as well as Metastore (if we decide to
separate it). It would help that metastore has the same PMC as Hive since
you would most likely have to get reviews and approvals from the same set
of people that you would now. For features in dev branches, we will have to
come up with a release strategy so that features spanning metastore and
hive would work well while in development. May be using snapshot libraries
of metastore which builds from the latest code and making a metastore
release before releasing Hive so that they are always in sync.
As far as concerns related to features spanning multiple projects are
concerned, it is true even today (Hive, Spark, Tez are all separate
projects), although I agree that it may be a lesser problem for most
features. In my opinion, while it is true that many other projects like
Impala, Presto, Spark use Hive Metastore, they do so primarily to maintain
compatibility with Hive. When their adoption rises what stops them from
creating their own metadata service which suits their needs better? If or
when that happens, it would lead to fragmentation as far as metadata stores
are concerned. If we separate HMS, we can strive to make it a general
purpose metadata service which the other projects would like to adopt
banking on the advantages which it brings now like compatibility with Hive.
I think as long as metastore is within Hive, it will always be Hive's
metastore and other projects will be cautious to adopt it fearing changes
which might break their code and always having to play well with Hive.
Thanks,
Vihang
On Thu, Jul 6, 2017 at 2:10 AM, Peter Vary <pv...@cloudera.com> wrote:
> Hi folks,
>
> I agree with most of the things Edward said. I have faced similar issues
> in smaller scale when integrated Hive with Yetus. We are forced to keep
> patched Yetus files in Hive repo until they push their next release. Also
> followed one more serious problem when a patch was committed to Hive,
> Impala and Spark, and just few days before the release all of them was
> reverted from the projects due to concerns raised by the Spark committee
> (after the changes was already committed to Spark as well)
>
> Having said all of these, I still think that separating the HMS to a new
> top level project could be a step to the right direction with the following
> constraints. The new project should have:
> - Strict, stability oriented branching strategy following Edward's
> suggestions, so if a downstream project - for example Hive - needs some fix
> or easy change that could be incorporated, and released almost immediately.
> So we have to have these:
> - Always releasable head
> - Every multi commit feature should be added as a feature branch
> - Strict, enforced, stability oriented API strategy. So we will not be
> surprised by features added by other projects and break Hive compatibility.
> To avoid this situation we need to design for it, have pre-commit tests in
> place for catch the in-adverted changes, and most importantly have a clear
> commitment for it.
>
> I think, since the current HMS is already used by numerous other projects,
> we already should have these in mind when modifying anything in HMS related
> code. This is not the main focus of Hive, so we do not concentrate on this
> and there are often interoperability issues, problems. We can do this
> inside Hive as well, but the current approach followed by Hive, and the one
> required by the HMS are requiring a different mindset. We need a clear,
> well defined boundary and separating the 2 projects could help in this. We
> can focus on the different needs and goal and eventually we might have
> different culture as well which suits the specific needs of the specific
> part of the code.
>
> I think keeping these rules in the new to level HMS we can mitigate most
> of the issues mentioned below, and we will be better of overall.
> What do you think Edward?
>
> Thanks,
> Peter
>
>
> > On Jul 5, 2017, at 10:16 PM, Xuefu Zhang <xu...@apache.org> wrote:
> >
> > I think Edward's concern is valid. While I voiced my support for this
> > proposal, which was more from the benefits of the whole Hadoop
> ecosystem, I
> > don't see the equal benefits for Hive. Instead, it may even create more
> > overhead for Hive. I'd really like to take time to see what are the road
> > blocks for other projects to use HMS as it is. The issue of Spark
> including
> > a Hive fork, which was brought up some time back, is certainly not one of
> > them.
> >
> > Thanks,
> > Xuefu
> >
> > On Wed, Jul 5, 2017 at 12:33 PM, Edward Capriolo <ed...@gmail.com>
> > wrote:
> >
> >> On Wed, Jul 5, 2017 at 1:51 PM, Alan Gates <al...@gmail.com>
> wrote:
> >>
> >>> On Mon, Jul 3, 2017 at 6:20 AM, Edward Capriolo <edlinuxguru@gmail.com
> >
> >>> wrote:
> >>>
> >>>>
> >>>> We already have things in the meta-store not directly tied to language
> >>>> features. For example hive metastore has a "retention" property which
> >> is
> >>>> not actively in use by anything. In reality, we rarely say 'no' or -1
> >> to
> >>>> much. Which in part is why I believe our release process is grinding
> >>>> slower: we have so many things in flight I do not feel that any one
> >>> person
> >>>> can keep track. You are working on porting the metastore to hbase.
> >>>> https://issues.apache.org/jira/browse/HIVE-9452 did you get a -1 or
> >> 'No'
> >>>> along the way? When I first noticed this I pointed out that someone
> has
> >>>> already ported the metastore to Cassandra
> >>>> https://github.com/riptano/brisk/blob/master/src/java/
> >>>> src/org/apache/cassandra/hadoop/hive/metastore/SchemaManager
> >>> Service.java,
> >>>> but I was more exciting/rational for this multi-year approach using
> >> hbase
> >>>> so I let everyone 'have at it'.
> >>>>
> >>> Your example and mine are not equivalent. The HBase metastore is
> still a
> >>> Hive feature, even if some thought it not worth while. That is
> different
> >>> than people bringing features that will never interest Hive or that
> Hive
> >>> could never use (e.g. Dain’s desire for the metastore to support Presto
> >>> style views).
> >>>
> >>> I forgot to mention the issue these would be non-Hive contributors have
> >>> with releases if they contribute their features to the metastore while
> >> it’s
> >>> inside Hive. Is Hive going to do a release just to push out features
> in
> >>> the metastore that it doesn’t care about?
> >>>
> >>> You seem to be asserting that doing this doesn’t really help non-Hive
> >> based
> >>> systems that are using or would like to use the metastore. But it is
> >>> interesting that people from three of those systems have commented in
> the
> >>> thread so far, and all are positive (Dmitrias from Impala, Dain from
> >>> Presto, and Sriharsha from the schema registry project).
> >>>
> >>>
> >>>> I am going to give a hypothetical but real world situation. Suppose I
> >>> want
> >>>> to add the statement "CREATE permanent macro xyz", this feature I
> >> believe
> >>>> would cross cut calcite, hive, and hive metastore. To build this
> >> feature
> >>> I
> >>>> would need to orchestrate the change across 3 separate groups of hive
> >>>> 'subcommittees' for lack of a better word. 3 git repos, 3 Jira's 3
> >>>> releases. That is not counting if we run into some bug or misfeature
> >>> (maybe
> >>>> with Tez or something else) so that brings in 4-5 releases of upstream
> >> to
> >>>> add a feature to hive. This does not take into account normal
> processes
> >>>> mess ups. For example say you get the metastore done, but now the
> >> people
> >>>> doing the calcite/antlr suggest the feature have different syntax
> >> because
> >>>> they did not read the 3-4 linked tickets when the process started?
> Now,
> >>> you
> >>>> have to loop back around the process. Finding 1 person in 1 project to
> >>>> usher along the feature you want is difficult, having to find and
> clear
> >>>> time with 3 people across three projects is going to be a difficult
> >> along
> >>>> with then 'pushing' them all to kick out a release so you can finally
> >> use
> >>>> said feature.
> >>>>
> >>>
> >>> I partially agree with you. On the reviews, JIRAs, etc. I don’t think
> it
> >>> adds much, if any, overhead. Hive is a big project and no one person
> >> knows
> >>> all the code anymore. If you wanted to add a permanent macros feature
> >> you
> >>> would need reviews from someone who knows the parser (probably
> >> Pengcheng),
> >>> people who know the optimizer (Jesus, Ashutosh, …), and someone who
> knows
> >>> the metastore (me, Thejas, …). And any large feature is going to be
> >>> implemented over multiple JIRAs, all of which are linkable regardless
> of
> >>> whether the JIRAs start with METASTORE- or HIVE-. I also don’t think
> it
> >>> makes the feature disagreement any worse. If the optimizer team
> >> absolutely
> >>> insists it has to have some feature and the metastore team insists that
> >> it
> >>> can’t have that feature you’re going to have to work through the issue
> >>> whether they all are in Hive or in two separate projects.
> >>>
> >>> Where I agree the split adds cost is releases. Before your macro
> feature
> >>> could go live you need releases from each of the components. And while
> >> in
> >>> development the components need to use snapshot versions of the other
> >>> components. My assertion is that the benefits out weigh this cost.
> >>>
> >>> Alan.
> >>>
> >>
> >>
> >> "You seem to be asserting that doing this doesn’t really help non-Hive
> >> based
> >> systems that are using or would like to use the metastore. But it is
> >> interesting that people from three of those systems have commented in
> the
> >> thread so far, and all are positive (Dmitrias from Impala, Dain from
> >> Presto, and Sriharsha from the schema registry project)."
> >>
> >> I notice that impala has a syntax for caching.
> >>
> >> https://www.cloudera.com/documentation/enterprise/5-8-x/topi
> >> cs/impala_perf_hdfs_caching.html
> >>
> >> Notice how the cache syntax did not way into Hive? It would make sense
> if
> >> this feature trickled it's way into hive and use HDFS caching for
> example.
> >> I have heard many people claim that using hive metastore is such a
> because
> >> it is packaged weird (like with ORC), but again besides
> claim/complaining
> >> no one has stepped up to deal with that.
> >>
> >> What I would suggest is going forward for maybe a trial period of 6
> months,
> >> labeling JIRA tickets with a tag that would be
> >> "SeeThisProvesWeNeedATLPMetastore". Because right now I do not enough
> >> active use cases of people giving anything back to justify hurting our
> >> workflow so much.
> >>
> >>
> >> "I partially agree with you. On the reviews, JIRAs, etc. I don’t think
> it
> >> adds much, if any, overhead. Hive is a big project and no one person
> knows
> >> all the code anymore. If you wanted to add a permanent macros feature
> you
> >> would need reviews from someone who knows the parser (probably
> Pengcheng),
> >> people who know the optimizer (Jesus, Ashutosh, …), and someone who
> knows
> >> the metastore (me, Thejas, …). And any large feature is going to be
> >> implemented over multiple JIRAs, all of which are linkable regardless of
> >> whether the JIRAs start with METASTORE- or HIVE-. I also don’t think
> it
> >> makes the feature disagreement any worse. If the optimizer team
> absolutely
> >> insists it has to have some feature and the metastore team insists that
> it
> >> can’t have that feature you’re going to have to work through the issue
> >> whether they all are in Hive or in two separate projects"
> >>
> >> Macro was done in 1 patch and reviewed by 2 people. With 2-3 follow on
> >> bugs.
> >>
> >> https://issues.apache.org/jira/browse/HIVE-2655
> >>
> >> I think your perception is different then mine because of
> circumstances. I
> >> have waited weeks/months for reviews/merges (in Hive and other apache
> >> projects) from mundane udfs to cassandra-storage-handlers. You obviously
> >> work in a large company and you can more easily align objectives, go to
> the
> >> water cooler and say "hey bob you know it would be cool if you can
> release
> >> x so I can do y". When you are not in that situation its like, "hey
> mailing
> >> list, my patch was done for three months now and like I have had to
> rebase
> >> it three times and like I notice like other stuff is getting committed."
> >>
> >> If you look at it tactically, "create permanent macro xzy". I go over to
> >> calcite and suggest some changes there, if this concept is not "game
> >> changer" it is probably going to sit unreviewed. If it is "game changer"
> >> exciting that is 72 hours for release voting. Next go to hive-metastore
> >> repeat the process, but remember now I have to "wow" the metastore
> people
> >> with the "game changer" and if that crew is super focused on something
> >> about kafka well now Hive features are second fiddle. Now lets say a
> hive
> >> release is coming up, and I really want my feature in it.
> >> hive-metastore-tlp might currently have a broken trunk because mongo
> wants
> >> to add spaceships to wombats feature has a bug and frankly that should
> not
> >> effect us.
> >>
> >> I hate to draw in something else but I feel it is related:
> >>
> >> 8 December 2016 : release 2.1.1 available
> >> 07 April 2017 : release 1.2.2 available
> >> hive-dev [DISCUSS] Supporting Hadoop-1 and experimental features
> >> hive-dev Re: release chaos?
> >>
> >> I have been vocal about not liking certain branching strategies and
> >> proposals that take us away from releasable trunk. We have steadily
> headed
> >> in a direction where we are pulling things out of hive, and we are not
> able
> >> to turn out releases. We even had a thread "release chaos" talking about
> >> our 5 active branches (with friends I say "jumped the shark"). Pulling
> out
> >> the metastore is only going to make this worse. I do not even see the
> model
> >> as successful. You may say it is great that calcite lets people share
> our
> >> sql dialect or the ORC TLP has 5 committers, but if Hive can not get a
> >> release out the door I do not see us optimizing for the proper thing.
> >>
>
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Peter Vary <pv...@cloudera.com>.
Hi folks,
I agree with most of the things Edward said. I have faced similar issues in smaller scale when integrated Hive with Yetus. We are forced to keep patched Yetus files in Hive repo until they push their next release. Also followed one more serious problem when a patch was committed to Hive, Impala and Spark, and just few days before the release all of them was reverted from the projects due to concerns raised by the Spark committee (after the changes was already committed to Spark as well)
Having said all of these, I still think that separating the HMS to a new top level project could be a step to the right direction with the following constraints. The new project should have:
- Strict, stability oriented branching strategy following Edward's suggestions, so if a downstream project - for example Hive - needs some fix or easy change that could be incorporated, and released almost immediately. So we have to have these:
- Always releasable head
- Every multi commit feature should be added as a feature branch
- Strict, enforced, stability oriented API strategy. So we will not be surprised by features added by other projects and break Hive compatibility. To avoid this situation we need to design for it, have pre-commit tests in place for catch the in-adverted changes, and most importantly have a clear commitment for it.
I think, since the current HMS is already used by numerous other projects, we already should have these in mind when modifying anything in HMS related code. This is not the main focus of Hive, so we do not concentrate on this and there are often interoperability issues, problems. We can do this inside Hive as well, but the current approach followed by Hive, and the one required by the HMS are requiring a different mindset. We need a clear, well defined boundary and separating the 2 projects could help in this. We can focus on the different needs and goal and eventually we might have different culture as well which suits the specific needs of the specific part of the code.
I think keeping these rules in the new to level HMS we can mitigate most of the issues mentioned below, and we will be better of overall.
What do you think Edward?
Thanks,
Peter
> On Jul 5, 2017, at 10:16 PM, Xuefu Zhang <xu...@apache.org> wrote:
>
> I think Edward's concern is valid. While I voiced my support for this
> proposal, which was more from the benefits of the whole Hadoop ecosystem, I
> don't see the equal benefits for Hive. Instead, it may even create more
> overhead for Hive. I'd really like to take time to see what are the road
> blocks for other projects to use HMS as it is. The issue of Spark including
> a Hive fork, which was brought up some time back, is certainly not one of
> them.
>
> Thanks,
> Xuefu
>
> On Wed, Jul 5, 2017 at 12:33 PM, Edward Capriolo <ed...@gmail.com>
> wrote:
>
>> On Wed, Jul 5, 2017 at 1:51 PM, Alan Gates <al...@gmail.com> wrote:
>>
>>> On Mon, Jul 3, 2017 at 6:20 AM, Edward Capriolo <ed...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> We already have things in the meta-store not directly tied to language
>>>> features. For example hive metastore has a "retention" property which
>> is
>>>> not actively in use by anything. In reality, we rarely say 'no' or -1
>> to
>>>> much. Which in part is why I believe our release process is grinding
>>>> slower: we have so many things in flight I do not feel that any one
>>> person
>>>> can keep track. You are working on porting the metastore to hbase.
>>>> https://issues.apache.org/jira/browse/HIVE-9452 did you get a -1 or
>> 'No'
>>>> along the way? When I first noticed this I pointed out that someone has
>>>> already ported the metastore to Cassandra
>>>> https://github.com/riptano/brisk/blob/master/src/java/
>>>> src/org/apache/cassandra/hadoop/hive/metastore/SchemaManager
>>> Service.java,
>>>> but I was more exciting/rational for this multi-year approach using
>> hbase
>>>> so I let everyone 'have at it'.
>>>>
>>> Your example and mine are not equivalent. The HBase metastore is still a
>>> Hive feature, even if some thought it not worth while. That is different
>>> than people bringing features that will never interest Hive or that Hive
>>> could never use (e.g. Dain’s desire for the metastore to support Presto
>>> style views).
>>>
>>> I forgot to mention the issue these would be non-Hive contributors have
>>> with releases if they contribute their features to the metastore while
>> it’s
>>> inside Hive. Is Hive going to do a release just to push out features in
>>> the metastore that it doesn’t care about?
>>>
>>> You seem to be asserting that doing this doesn’t really help non-Hive
>> based
>>> systems that are using or would like to use the metastore. But it is
>>> interesting that people from three of those systems have commented in the
>>> thread so far, and all are positive (Dmitrias from Impala, Dain from
>>> Presto, and Sriharsha from the schema registry project).
>>>
>>>
>>>> I am going to give a hypothetical but real world situation. Suppose I
>>> want
>>>> to add the statement "CREATE permanent macro xyz", this feature I
>> believe
>>>> would cross cut calcite, hive, and hive metastore. To build this
>> feature
>>> I
>>>> would need to orchestrate the change across 3 separate groups of hive
>>>> 'subcommittees' for lack of a better word. 3 git repos, 3 Jira's 3
>>>> releases. That is not counting if we run into some bug or misfeature
>>> (maybe
>>>> with Tez or something else) so that brings in 4-5 releases of upstream
>> to
>>>> add a feature to hive. This does not take into account normal processes
>>>> mess ups. For example say you get the metastore done, but now the
>> people
>>>> doing the calcite/antlr suggest the feature have different syntax
>> because
>>>> they did not read the 3-4 linked tickets when the process started? Now,
>>> you
>>>> have to loop back around the process. Finding 1 person in 1 project to
>>>> usher along the feature you want is difficult, having to find and clear
>>>> time with 3 people across three projects is going to be a difficult
>> along
>>>> with then 'pushing' them all to kick out a release so you can finally
>> use
>>>> said feature.
>>>>
>>>
>>> I partially agree with you. On the reviews, JIRAs, etc. I don’t think it
>>> adds much, if any, overhead. Hive is a big project and no one person
>> knows
>>> all the code anymore. If you wanted to add a permanent macros feature
>> you
>>> would need reviews from someone who knows the parser (probably
>> Pengcheng),
>>> people who know the optimizer (Jesus, Ashutosh, …), and someone who knows
>>> the metastore (me, Thejas, …). And any large feature is going to be
>>> implemented over multiple JIRAs, all of which are linkable regardless of
>>> whether the JIRAs start with METASTORE- or HIVE-. I also don’t think it
>>> makes the feature disagreement any worse. If the optimizer team
>> absolutely
>>> insists it has to have some feature and the metastore team insists that
>> it
>>> can’t have that feature you’re going to have to work through the issue
>>> whether they all are in Hive or in two separate projects.
>>>
>>> Where I agree the split adds cost is releases. Before your macro feature
>>> could go live you need releases from each of the components. And while
>> in
>>> development the components need to use snapshot versions of the other
>>> components. My assertion is that the benefits out weigh this cost.
>>>
>>> Alan.
>>>
>>
>>
>> "You seem to be asserting that doing this doesn’t really help non-Hive
>> based
>> systems that are using or would like to use the metastore. But it is
>> interesting that people from three of those systems have commented in the
>> thread so far, and all are positive (Dmitrias from Impala, Dain from
>> Presto, and Sriharsha from the schema registry project)."
>>
>> I notice that impala has a syntax for caching.
>>
>> https://www.cloudera.com/documentation/enterprise/5-8-x/topi
>> cs/impala_perf_hdfs_caching.html
>>
>> Notice how the cache syntax did not way into Hive? It would make sense if
>> this feature trickled it's way into hive and use HDFS caching for example.
>> I have heard many people claim that using hive metastore is such a because
>> it is packaged weird (like with ORC), but again besides claim/complaining
>> no one has stepped up to deal with that.
>>
>> What I would suggest is going forward for maybe a trial period of 6 months,
>> labeling JIRA tickets with a tag that would be
>> "SeeThisProvesWeNeedATLPMetastore". Because right now I do not enough
>> active use cases of people giving anything back to justify hurting our
>> workflow so much.
>>
>>
>> "I partially agree with you. On the reviews, JIRAs, etc. I don’t think it
>> adds much, if any, overhead. Hive is a big project and no one person knows
>> all the code anymore. If you wanted to add a permanent macros feature you
>> would need reviews from someone who knows the parser (probably Pengcheng),
>> people who know the optimizer (Jesus, Ashutosh, …), and someone who knows
>> the metastore (me, Thejas, …). And any large feature is going to be
>> implemented over multiple JIRAs, all of which are linkable regardless of
>> whether the JIRAs start with METASTORE- or HIVE-. I also don’t think it
>> makes the feature disagreement any worse. If the optimizer team absolutely
>> insists it has to have some feature and the metastore team insists that it
>> can’t have that feature you’re going to have to work through the issue
>> whether they all are in Hive or in two separate projects"
>>
>> Macro was done in 1 patch and reviewed by 2 people. With 2-3 follow on
>> bugs.
>>
>> https://issues.apache.org/jira/browse/HIVE-2655
>>
>> I think your perception is different then mine because of circumstances. I
>> have waited weeks/months for reviews/merges (in Hive and other apache
>> projects) from mundane udfs to cassandra-storage-handlers. You obviously
>> work in a large company and you can more easily align objectives, go to the
>> water cooler and say "hey bob you know it would be cool if you can release
>> x so I can do y". When you are not in that situation its like, "hey mailing
>> list, my patch was done for three months now and like I have had to rebase
>> it three times and like I notice like other stuff is getting committed."
>>
>> If you look at it tactically, "create permanent macro xzy". I go over to
>> calcite and suggest some changes there, if this concept is not "game
>> changer" it is probably going to sit unreviewed. If it is "game changer"
>> exciting that is 72 hours for release voting. Next go to hive-metastore
>> repeat the process, but remember now I have to "wow" the metastore people
>> with the "game changer" and if that crew is super focused on something
>> about kafka well now Hive features are second fiddle. Now lets say a hive
>> release is coming up, and I really want my feature in it.
>> hive-metastore-tlp might currently have a broken trunk because mongo wants
>> to add spaceships to wombats feature has a bug and frankly that should not
>> effect us.
>>
>> I hate to draw in something else but I feel it is related:
>>
>> 8 December 2016 : release 2.1.1 available
>> 07 April 2017 : release 1.2.2 available
>> hive-dev [DISCUSS] Supporting Hadoop-1 and experimental features
>> hive-dev Re: release chaos?
>>
>> I have been vocal about not liking certain branching strategies and
>> proposals that take us away from releasable trunk. We have steadily headed
>> in a direction where we are pulling things out of hive, and we are not able
>> to turn out releases. We even had a thread "release chaos" talking about
>> our 5 active branches (with friends I say "jumped the shark"). Pulling out
>> the metastore is only going to make this worse. I do not even see the model
>> as successful. You may say it is great that calcite lets people share our
>> sql dialect or the ORC TLP has 5 committers, but if Hive can not get a
>> release out the door I do not see us optimizing for the proper thing.
>>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Xuefu Zhang <xu...@apache.org>.
I think Edward's concern is valid. While I voiced my support for this
proposal, which was more from the benefits of the whole Hadoop ecosystem, I
don't see the equal benefits for Hive. Instead, it may even create more
overhead for Hive. I'd really like to take time to see what are the road
blocks for other projects to use HMS as it is. The issue of Spark including
a Hive fork, which was brought up some time back, is certainly not one of
them.
Thanks,
Xuefu
On Wed, Jul 5, 2017 at 12:33 PM, Edward Capriolo <ed...@gmail.com>
wrote:
> On Wed, Jul 5, 2017 at 1:51 PM, Alan Gates <al...@gmail.com> wrote:
>
> > On Mon, Jul 3, 2017 at 6:20 AM, Edward Capriolo <ed...@gmail.com>
> > wrote:
> >
> > >
> > > We already have things in the meta-store not directly tied to language
> > > features. For example hive metastore has a "retention" property which
> is
> > > not actively in use by anything. In reality, we rarely say 'no' or -1
> to
> > > much. Which in part is why I believe our release process is grinding
> > > slower: we have so many things in flight I do not feel that any one
> > person
> > > can keep track. You are working on porting the metastore to hbase.
> > > https://issues.apache.org/jira/browse/HIVE-9452 did you get a -1 or
> 'No'
> > > along the way? When I first noticed this I pointed out that someone has
> > > already ported the metastore to Cassandra
> > > https://github.com/riptano/brisk/blob/master/src/java/
> > > src/org/apache/cassandra/hadoop/hive/metastore/SchemaManager
> > Service.java,
> > > but I was more exciting/rational for this multi-year approach using
> hbase
> > > so I let everyone 'have at it'.
> > >
> > Your example and mine are not equivalent. The HBase metastore is still a
> > Hive feature, even if some thought it not worth while. That is different
> > than people bringing features that will never interest Hive or that Hive
> > could never use (e.g. Dain’s desire for the metastore to support Presto
> > style views).
> >
> > I forgot to mention the issue these would be non-Hive contributors have
> > with releases if they contribute their features to the metastore while
> it’s
> > inside Hive. Is Hive going to do a release just to push out features in
> > the metastore that it doesn’t care about?
> >
> > You seem to be asserting that doing this doesn’t really help non-Hive
> based
> > systems that are using or would like to use the metastore. But it is
> > interesting that people from three of those systems have commented in the
> > thread so far, and all are positive (Dmitrias from Impala, Dain from
> > Presto, and Sriharsha from the schema registry project).
> >
> >
> > > I am going to give a hypothetical but real world situation. Suppose I
> > want
> > > to add the statement "CREATE permanent macro xyz", this feature I
> believe
> > > would cross cut calcite, hive, and hive metastore. To build this
> feature
> > I
> > > would need to orchestrate the change across 3 separate groups of hive
> > > 'subcommittees' for lack of a better word. 3 git repos, 3 Jira's 3
> > > releases. That is not counting if we run into some bug or misfeature
> > (maybe
> > > with Tez or something else) so that brings in 4-5 releases of upstream
> to
> > > add a feature to hive. This does not take into account normal processes
> > > mess ups. For example say you get the metastore done, but now the
> people
> > > doing the calcite/antlr suggest the feature have different syntax
> because
> > > they did not read the 3-4 linked tickets when the process started? Now,
> > you
> > > have to loop back around the process. Finding 1 person in 1 project to
> > > usher along the feature you want is difficult, having to find and clear
> > > time with 3 people across three projects is going to be a difficult
> along
> > > with then 'pushing' them all to kick out a release so you can finally
> use
> > > said feature.
> > >
> >
> > I partially agree with you. On the reviews, JIRAs, etc. I don’t think it
> > adds much, if any, overhead. Hive is a big project and no one person
> knows
> > all the code anymore. If you wanted to add a permanent macros feature
> you
> > would need reviews from someone who knows the parser (probably
> Pengcheng),
> > people who know the optimizer (Jesus, Ashutosh, …), and someone who knows
> > the metastore (me, Thejas, …). And any large feature is going to be
> > implemented over multiple JIRAs, all of which are linkable regardless of
> > whether the JIRAs start with METASTORE- or HIVE-. I also don’t think it
> > makes the feature disagreement any worse. If the optimizer team
> absolutely
> > insists it has to have some feature and the metastore team insists that
> it
> > can’t have that feature you’re going to have to work through the issue
> > whether they all are in Hive or in two separate projects.
> >
> > Where I agree the split adds cost is releases. Before your macro feature
> > could go live you need releases from each of the components. And while
> in
> > development the components need to use snapshot versions of the other
> > components. My assertion is that the benefits out weigh this cost.
> >
> > Alan.
> >
>
>
> "You seem to be asserting that doing this doesn’t really help non-Hive
> based
> systems that are using or would like to use the metastore. But it is
> interesting that people from three of those systems have commented in the
> thread so far, and all are positive (Dmitrias from Impala, Dain from
> Presto, and Sriharsha from the schema registry project)."
>
> I notice that impala has a syntax for caching.
>
> https://www.cloudera.com/documentation/enterprise/5-8-x/topi
> cs/impala_perf_hdfs_caching.html
>
> Notice how the cache syntax did not way into Hive? It would make sense if
> this feature trickled it's way into hive and use HDFS caching for example.
> I have heard many people claim that using hive metastore is such a because
> it is packaged weird (like with ORC), but again besides claim/complaining
> no one has stepped up to deal with that.
>
> What I would suggest is going forward for maybe a trial period of 6 months,
> labeling JIRA tickets with a tag that would be
> "SeeThisProvesWeNeedATLPMetastore". Because right now I do not enough
> active use cases of people giving anything back to justify hurting our
> workflow so much.
>
>
> "I partially agree with you. On the reviews, JIRAs, etc. I don’t think it
> adds much, if any, overhead. Hive is a big project and no one person knows
> all the code anymore. If you wanted to add a permanent macros feature you
> would need reviews from someone who knows the parser (probably Pengcheng),
> people who know the optimizer (Jesus, Ashutosh, …), and someone who knows
> the metastore (me, Thejas, …). And any large feature is going to be
> implemented over multiple JIRAs, all of which are linkable regardless of
> whether the JIRAs start with METASTORE- or HIVE-. I also don’t think it
> makes the feature disagreement any worse. If the optimizer team absolutely
> insists it has to have some feature and the metastore team insists that it
> can’t have that feature you’re going to have to work through the issue
> whether they all are in Hive or in two separate projects"
>
> Macro was done in 1 patch and reviewed by 2 people. With 2-3 follow on
> bugs.
>
> https://issues.apache.org/jira/browse/HIVE-2655
>
> I think your perception is different then mine because of circumstances. I
> have waited weeks/months for reviews/merges (in Hive and other apache
> projects) from mundane udfs to cassandra-storage-handlers. You obviously
> work in a large company and you can more easily align objectives, go to the
> water cooler and say "hey bob you know it would be cool if you can release
> x so I can do y". When you are not in that situation its like, "hey mailing
> list, my patch was done for three months now and like I have had to rebase
> it three times and like I notice like other stuff is getting committed."
>
> If you look at it tactically, "create permanent macro xzy". I go over to
> calcite and suggest some changes there, if this concept is not "game
> changer" it is probably going to sit unreviewed. If it is "game changer"
> exciting that is 72 hours for release voting. Next go to hive-metastore
> repeat the process, but remember now I have to "wow" the metastore people
> with the "game changer" and if that crew is super focused on something
> about kafka well now Hive features are second fiddle. Now lets say a hive
> release is coming up, and I really want my feature in it.
> hive-metastore-tlp might currently have a broken trunk because mongo wants
> to add spaceships to wombats feature has a bug and frankly that should not
> effect us.
>
> I hate to draw in something else but I feel it is related:
>
> 8 December 2016 : release 2.1.1 available
> 07 April 2017 : release 1.2.2 available
> hive-dev [DISCUSS] Supporting Hadoop-1 and experimental features
> hive-dev Re: release chaos?
>
> I have been vocal about not liking certain branching strategies and
> proposals that take us away from releasable trunk. We have steadily headed
> in a direction where we are pulling things out of hive, and we are not able
> to turn out releases. We even had a thread "release chaos" talking about
> our 5 active branches (with friends I say "jumped the shark"). Pulling out
> the metastore is only going to make this worse. I do not even see the model
> as successful. You may say it is great that calcite lets people share our
> sql dialect or the ORC TLP has 5 committers, but if Hive can not get a
> release out the door I do not see us optimizing for the proper thing.
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Edward Capriolo <ed...@gmail.com>.
On Wed, Jul 5, 2017 at 1:51 PM, Alan Gates <al...@gmail.com> wrote:
> On Mon, Jul 3, 2017 at 6:20 AM, Edward Capriolo <ed...@gmail.com>
> wrote:
>
> >
> > We already have things in the meta-store not directly tied to language
> > features. For example hive metastore has a "retention" property which is
> > not actively in use by anything. In reality, we rarely say 'no' or -1 to
> > much. Which in part is why I believe our release process is grinding
> > slower: we have so many things in flight I do not feel that any one
> person
> > can keep track. You are working on porting the metastore to hbase.
> > https://issues.apache.org/jira/browse/HIVE-9452 did you get a -1 or 'No'
> > along the way? When I first noticed this I pointed out that someone has
> > already ported the metastore to Cassandra
> > https://github.com/riptano/brisk/blob/master/src/java/
> > src/org/apache/cassandra/hadoop/hive/metastore/SchemaManager
> Service.java,
> > but I was more exciting/rational for this multi-year approach using hbase
> > so I let everyone 'have at it'.
> >
> Your example and mine are not equivalent. The HBase metastore is still a
> Hive feature, even if some thought it not worth while. That is different
> than people bringing features that will never interest Hive or that Hive
> could never use (e.g. Dain’s desire for the metastore to support Presto
> style views).
>
> I forgot to mention the issue these would be non-Hive contributors have
> with releases if they contribute their features to the metastore while it’s
> inside Hive. Is Hive going to do a release just to push out features in
> the metastore that it doesn’t care about?
>
> You seem to be asserting that doing this doesn’t really help non-Hive based
> systems that are using or would like to use the metastore. But it is
> interesting that people from three of those systems have commented in the
> thread so far, and all are positive (Dmitrias from Impala, Dain from
> Presto, and Sriharsha from the schema registry project).
>
>
> > I am going to give a hypothetical but real world situation. Suppose I
> want
> > to add the statement "CREATE permanent macro xyz", this feature I believe
> > would cross cut calcite, hive, and hive metastore. To build this feature
> I
> > would need to orchestrate the change across 3 separate groups of hive
> > 'subcommittees' for lack of a better word. 3 git repos, 3 Jira's 3
> > releases. That is not counting if we run into some bug or misfeature
> (maybe
> > with Tez or something else) so that brings in 4-5 releases of upstream to
> > add a feature to hive. This does not take into account normal processes
> > mess ups. For example say you get the metastore done, but now the people
> > doing the calcite/antlr suggest the feature have different syntax because
> > they did not read the 3-4 linked tickets when the process started? Now,
> you
> > have to loop back around the process. Finding 1 person in 1 project to
> > usher along the feature you want is difficult, having to find and clear
> > time with 3 people across three projects is going to be a difficult along
> > with then 'pushing' them all to kick out a release so you can finally use
> > said feature.
> >
>
> I partially agree with you. On the reviews, JIRAs, etc. I don’t think it
> adds much, if any, overhead. Hive is a big project and no one person knows
> all the code anymore. If you wanted to add a permanent macros feature you
> would need reviews from someone who knows the parser (probably Pengcheng),
> people who know the optimizer (Jesus, Ashutosh, …), and someone who knows
> the metastore (me, Thejas, …). And any large feature is going to be
> implemented over multiple JIRAs, all of which are linkable regardless of
> whether the JIRAs start with METASTORE- or HIVE-. I also don’t think it
> makes the feature disagreement any worse. If the optimizer team absolutely
> insists it has to have some feature and the metastore team insists that it
> can’t have that feature you’re going to have to work through the issue
> whether they all are in Hive or in two separate projects.
>
> Where I agree the split adds cost is releases. Before your macro feature
> could go live you need releases from each of the components. And while in
> development the components need to use snapshot versions of the other
> components. My assertion is that the benefits out weigh this cost.
>
> Alan.
>
"You seem to be asserting that doing this doesn’t really help non-Hive based
systems that are using or would like to use the metastore. But it is
interesting that people from three of those systems have commented in the
thread so far, and all are positive (Dmitrias from Impala, Dain from
Presto, and Sriharsha from the schema registry project)."
I notice that impala has a syntax for caching.
https://www.cloudera.com/documentation/enterprise/5-8-x/topi
cs/impala_perf_hdfs_caching.html
Notice how the cache syntax did not way into Hive? It would make sense if
this feature trickled it's way into hive and use HDFS caching for example.
I have heard many people claim that using hive metastore is such a because
it is packaged weird (like with ORC), but again besides claim/complaining
no one has stepped up to deal with that.
What I would suggest is going forward for maybe a trial period of 6 months,
labeling JIRA tickets with a tag that would be
"SeeThisProvesWeNeedATLPMetastore". Because right now I do not enough
active use cases of people giving anything back to justify hurting our
workflow so much.
"I partially agree with you. On the reviews, JIRAs, etc. I don’t think it
adds much, if any, overhead. Hive is a big project and no one person knows
all the code anymore. If you wanted to add a permanent macros feature you
would need reviews from someone who knows the parser (probably Pengcheng),
people who know the optimizer (Jesus, Ashutosh, …), and someone who knows
the metastore (me, Thejas, …). And any large feature is going to be
implemented over multiple JIRAs, all of which are linkable regardless of
whether the JIRAs start with METASTORE- or HIVE-. I also don’t think it
makes the feature disagreement any worse. If the optimizer team absolutely
insists it has to have some feature and the metastore team insists that it
can’t have that feature you’re going to have to work through the issue
whether they all are in Hive or in two separate projects"
Macro was done in 1 patch and reviewed by 2 people. With 2-3 follow on
bugs.
https://issues.apache.org/jira/browse/HIVE-2655
I think your perception is different then mine because of circumstances. I
have waited weeks/months for reviews/merges (in Hive and other apache
projects) from mundane udfs to cassandra-storage-handlers. You obviously
work in a large company and you can more easily align objectives, go to the
water cooler and say "hey bob you know it would be cool if you can release
x so I can do y". When you are not in that situation its like, "hey mailing
list, my patch was done for three months now and like I have had to rebase
it three times and like I notice like other stuff is getting committed."
If you look at it tactically, "create permanent macro xzy". I go over to
calcite and suggest some changes there, if this concept is not "game
changer" it is probably going to sit unreviewed. If it is "game changer"
exciting that is 72 hours for release voting. Next go to hive-metastore
repeat the process, but remember now I have to "wow" the metastore people
with the "game changer" and if that crew is super focused on something
about kafka well now Hive features are second fiddle. Now lets say a hive
release is coming up, and I really want my feature in it.
hive-metastore-tlp might currently have a broken trunk because mongo wants
to add spaceships to wombats feature has a bug and frankly that should not
effect us.
I hate to draw in something else but I feel it is related:
8 December 2016 : release 2.1.1 available
07 April 2017 : release 1.2.2 available
hive-dev [DISCUSS] Supporting Hadoop-1 and experimental features
hive-dev Re: release chaos?
I have been vocal about not liking certain branching strategies and
proposals that take us away from releasable trunk. We have steadily headed
in a direction where we are pulling things out of hive, and we are not able
to turn out releases. We even had a thread "release chaos" talking about
our 5 active branches (with friends I say "jumped the shark"). Pulling out
the metastore is only going to make this worse. I do not even see the model
as successful. You may say it is great that calcite lets people share our
sql dialect or the ORC TLP has 5 committers, but if Hive can not get a
release out the door I do not see us optimizing for the proper thing.
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Alan Gates <al...@gmail.com>.
On Mon, Jul 3, 2017 at 6:20 AM, Edward Capriolo <ed...@gmail.com>
wrote:
>
> We already have things in the meta-store not directly tied to language
> features. For example hive metastore has a "retention" property which is
> not actively in use by anything. In reality, we rarely say 'no' or -1 to
> much. Which in part is why I believe our release process is grinding
> slower: we have so many things in flight I do not feel that any one person
> can keep track. You are working on porting the metastore to hbase.
> https://issues.apache.org/jira/browse/HIVE-9452 did you get a -1 or 'No'
> along the way? When I first noticed this I pointed out that someone has
> already ported the metastore to Cassandra
> https://github.com/riptano/brisk/blob/master/src/java/
> src/org/apache/cassandra/hadoop/hive/metastore/SchemaManagerService.java,
> but I was more exciting/rational for this multi-year approach using hbase
> so I let everyone 'have at it'.
>
Your example and mine are not equivalent. The HBase metastore is still a
Hive feature, even if some thought it not worth while. That is different
than people bringing features that will never interest Hive or that Hive
could never use (e.g. Dain’s desire for the metastore to support Presto
style views).
I forgot to mention the issue these would be non-Hive contributors have
with releases if they contribute their features to the metastore while it’s
inside Hive. Is Hive going to do a release just to push out features in
the metastore that it doesn’t care about?
You seem to be asserting that doing this doesn’t really help non-Hive based
systems that are using or would like to use the metastore. But it is
interesting that people from three of those systems have commented in the
thread so far, and all are positive (Dmitrias from Impala, Dain from
Presto, and Sriharsha from the schema registry project).
> I am going to give a hypothetical but real world situation. Suppose I want
> to add the statement "CREATE permanent macro xyz", this feature I believe
> would cross cut calcite, hive, and hive metastore. To build this feature I
> would need to orchestrate the change across 3 separate groups of hive
> 'subcommittees' for lack of a better word. 3 git repos, 3 Jira's 3
> releases. That is not counting if we run into some bug or misfeature (maybe
> with Tez or something else) so that brings in 4-5 releases of upstream to
> add a feature to hive. This does not take into account normal processes
> mess ups. For example say you get the metastore done, but now the people
> doing the calcite/antlr suggest the feature have different syntax because
> they did not read the 3-4 linked tickets when the process started? Now, you
> have to loop back around the process. Finding 1 person in 1 project to
> usher along the feature you want is difficult, having to find and clear
> time with 3 people across three projects is going to be a difficult along
> with then 'pushing' them all to kick out a release so you can finally use
> said feature.
>
I partially agree with you. On the reviews, JIRAs, etc. I don’t think it
adds much, if any, overhead. Hive is a big project and no one person knows
all the code anymore. If you wanted to add a permanent macros feature you
would need reviews from someone who knows the parser (probably Pengcheng),
people who know the optimizer (Jesus, Ashutosh, …), and someone who knows
the metastore (me, Thejas, …). And any large feature is going to be
implemented over multiple JIRAs, all of which are linkable regardless of
whether the JIRAs start with METASTORE- or HIVE-. I also don’t think it
makes the feature disagreement any worse. If the optimizer team absolutely
insists it has to have some feature and the metastore team insists that it
can’t have that feature you’re going to have to work through the issue
whether they all are in Hive or in two separate projects.
Where I agree the split adds cost is releases. Before your macro feature
could go live you need releases from each of the components. And while in
development the components need to use snapshot versions of the other
components. My assertion is that the benefits out weigh this cost.
Alan.
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Edward Capriolo <ed...@gmail.com>.
On Sun, Jul 2, 2017 at 10:15 PM, Alan Gates <al...@gmail.com> wrote:
> Comments inlined.
>
> On Sun, Jul 2, 2017 at 3:22 PM, Edward Capriolo <ed...@gmail.com>
> wrote:
>
> > I am not sure I am on the fence with this.
> >
> > I am -1, and I offer this -1 with the hope of being convinced otherwise
> >
> Thank you for being open to reconsider.
>
> >
> >
> > "By making it a separate project we will enable other projects to join us
> > in
> > innovating on the metastore. "
> >
> > The relevant questions I have are,
> >
> > "What is stopping others from joining us now?"
> > "What does being a TLP do for us that we do not have now?"
> >
>
> Walking through a use case will help answer these. This is a real world
> situation, not a hypothetical. I’ve been talking with a team building a
> schema registry for Kafka[1]. I’d like them to use the Hive metastore
> rather than reinvent the wheel. I believe this would be good for users
> (all their tools can work together on a shared understanding of the data)
> and admins (just one metadata store to administer) and for the ecosytems
> (tools can work across stored data and streaming data).
>
> This system has some requirements on metadata that Hive does not. To take
> one example, it would like a schema to be a top level concept instead of a
> concept tied to tables or partitions. This is not a problem for Hive, but
> neither is it interesting. So if they come with patches for this, would we
> accept them? As the Hive PMC our answer will be no, because it doesn’t
> help Hive’s metadata. Even if we accept their patches will we make them
> committers when we know they don’t care about Hive as Hive, but only the
> metastore. Again, the right answer for the Hive PMC is no.
>
> And we cannot say that Hive should support a generic metadata system within
> itself. That turns Hive into an umbrella project, which Apache has
> repeatedly worked to avoid. So Hive will either need to reject non-Hive
> centric features and contributors or end up in a place Apache has worked to
> avoid.
>
> And finally, why would other teams want to mess with all of Hive when they
> only want the metastore? Hive is a large and complex system. If we break
> the metastore out it is much more approachable by non-Hive contributors.
>
> Obviously the Hive team doesn’t want to see their metastore turn into
> something unusable by Hive, which is why we were specific in saying we
> wanted it to continue to support high performance SQL systems.
>
> My experience in watching ORC move out of Hive is that the adoption has
> increased significantly. It is reasonable to assume that moving the
> metastore out will also increase adoption and make it easier for others to
> get involved.
>
>
> > I see a lot of downsides:
> > 1) We have to maintain two sites
> > 2) we have to maintain two committer lists
> >
> > A large problem I see is this: Hive is already being pulled in too many
> > different directions. There is some grumbling about the state of
> > hive-on-spark.
> >
>
> I believe this argues in favor of the split, not against. By pulling out
> the metastore we are releiving pressure on Hive itself. Let Hive focus on
> being a SQL engine. Let another team focus on runtime metadata.
>
> On your committer questions in later emails, the point of going to a TLP
> has nothing to do with adding new committers. Traditionally new projects
> start in the incubator. But given that all of the PMC of this new project
> are already experienced Hive PMC members I see no reason to go through
> incubator. I agree with you that we would not throw any new people into
> the mix. People join the project in the same way as always, by
> contributing.
>
> Alan.
>
> 1. https://github.com/hortonworks/registry
>
>
> > Most importantly, our release process seems 'injured' by too many
> branches
> > going off in different ways. If the metastore lives outside of Hive we
> are
> > going to compound this issue. I would strongly suggest we do not
> undertake
> > this until we can at least turn out 2 usable releases in a 6 month
> period.
> >
>
So if they come with patches for this, would we
accept them? As the Hive PMC our answer will be no, because it doesn’t
help Hive’s metadata.
We already have things in the meta-store not directly tied to language
features. For example hive metastore has a "retention" property which is
not actively in use by anything. In reality, we rarely say 'no' or -1 to
much. Which in part is why I believe our release process is grinding
slower: we have so many things in flight I do not feel that any one person
can keep track. You are working on porting the metastore to hbase.
https://issues.apache.org/jira/browse/HIVE-9452 did you get a -1 or 'No'
along the way? When I first noticed this I pointed out that someone has
already ported the metastore to Cassandra
https://github.com/riptano/brisk/blob/master/src/java/src/org/apache/cassandra/hadoop/hive/metastore/SchemaManagerService.java,
but I was more exciting/rational for this multi-year approach using hbase
so I let everyone 'have at it'.
I am going to give a hypothetical but real world situation. Suppose I want
to add the statement "CREATE permanent macro xyz", this feature I believe
would cross cut calcite, hive, and hive metastore. To build this feature I
would need to orchestrate the change across 3 separate groups of hive
'subcommittees' for lack of a better word. 3 git repos, 3 Jira's 3
releases. That is not counting if we run into some bug or misfeature (maybe
with Tez or something else) so that brings in 4-5 releases of upstream to
add a feature to hive. This does not take into account normal processes
mess ups. For example say you get the metastore done, but now the people
doing the calcite/antlr suggest the feature have different syntax because
they did not read the 3-4 linked tickets when the process started? Now, you
have to loop back around the process. Finding 1 person in 1 project to
usher along the feature you want is difficult, having to find and clear
time with 3 people across three projects is going to be a difficult along
with then 'pushing' them all to kick out a release so you can finally use
said feature.
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Alan Gates <al...@gmail.com>.
Comments inlined.
On Sun, Jul 2, 2017 at 3:22 PM, Edward Capriolo <ed...@gmail.com>
wrote:
> I am not sure I am on the fence with this.
>
> I am -1, and I offer this -1 with the hope of being convinced otherwise
>
Thank you for being open to reconsider.
>
>
> "By making it a separate project we will enable other projects to join us
> in
> innovating on the metastore. "
>
> The relevant questions I have are,
>
> "What is stopping others from joining us now?"
> "What does being a TLP do for us that we do not have now?"
>
Walking through a use case will help answer these. This is a real world
situation, not a hypothetical. I’ve been talking with a team building a
schema registry for Kafka[1]. I’d like them to use the Hive metastore
rather than reinvent the wheel. I believe this would be good for users
(all their tools can work together on a shared understanding of the data)
and admins (just one metadata store to administer) and for the ecosytems
(tools can work across stored data and streaming data).
This system has some requirements on metadata that Hive does not. To take
one example, it would like a schema to be a top level concept instead of a
concept tied to tables or partitions. This is not a problem for Hive, but
neither is it interesting. So if they come with patches for this, would we
accept them? As the Hive PMC our answer will be no, because it doesn’t
help Hive’s metadata. Even if we accept their patches will we make them
committers when we know they don’t care about Hive as Hive, but only the
metastore. Again, the right answer for the Hive PMC is no.
And we cannot say that Hive should support a generic metadata system within
itself. That turns Hive into an umbrella project, which Apache has
repeatedly worked to avoid. So Hive will either need to reject non-Hive
centric features and contributors or end up in a place Apache has worked to
avoid.
And finally, why would other teams want to mess with all of Hive when they
only want the metastore? Hive is a large and complex system. If we break
the metastore out it is much more approachable by non-Hive contributors.
Obviously the Hive team doesn’t want to see their metastore turn into
something unusable by Hive, which is why we were specific in saying we
wanted it to continue to support high performance SQL systems.
My experience in watching ORC move out of Hive is that the adoption has
increased significantly. It is reasonable to assume that moving the
metastore out will also increase adoption and make it easier for others to
get involved.
> I see a lot of downsides:
> 1) We have to maintain two sites
> 2) we have to maintain two committer lists
>
> A large problem I see is this: Hive is already being pulled in too many
> different directions. There is some grumbling about the state of
> hive-on-spark.
>
I believe this argues in favor of the split, not against. By pulling out
the metastore we are releiving pressure on Hive itself. Let Hive focus on
being a SQL engine. Let another team focus on runtime metadata.
On your committer questions in later emails, the point of going to a TLP
has nothing to do with adding new committers. Traditionally new projects
start in the incubator. But given that all of the PMC of this new project
are already experienced Hive PMC members I see no reason to go through
incubator. I agree with you that we would not throw any new people into
the mix. People join the project in the same way as always, by
contributing.
Alan.
1. https://github.com/hortonworks/registry
> Most importantly, our release process seems 'injured' by too many branches
> going off in different ways. If the metastore lives outside of Hive we are
> going to compound this issue. I would strongly suggest we do not undertake
> this until we can at least turn out 2 usable releases in a 6 month period.
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Edward Capriolo <ed...@gmail.com>.
I am not sure I am on the fence with this.
On Sat, Jul 1, 2017 at 6:38 AM, Chaoyu Tang <ct...@gmail.com> wrote:
> +1. A nice move which might help HMS to be more easily adopted by more
> other components.
>
> On Sat, Jul 1, 2017 at 12:41 AM, Sushanth Sowmyan <kh...@gmail.com>
> wrote:
>
> > +1
> >
> > On Jun 30, 2017 17:05, "Owen O'Malley" <ow...@gmail.com> wrote:
> >
> > > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <su...@apache.org> wrote:
> > >
> > > > and maybe a different project name?
> > > >
> > >
> > > Yes, it certainly needs a new name. I'd like to suggest Riven.
> > >
> > > .. Owen
> > >
> >
>
I am -1, and I offer this -1 with the hope of being convinced otherwise
"By making it a separate project we will enable other projects to join us in
innovating on the metastore. "
The relevant questions I have are,
"What is stopping others from joining us now?"
"What does being a TLP do for us that we do not have now?"
I see a lot of downsides:
1) We have to maintain two sites
2) we have to maintain two committer lists
A large problem I see is this: Hive is already being pulled in too many
different directions. There is some grumbling about the state of
hive-on-spark.
Most importantly, our release process seems 'injured' by too many branches
going off in different ways. If the metastore lives outside of Hive we are
going to compound this issue. I would strongly suggest we do not undertake
this until we can at least turn out 2 usable releases in a 6 month period.
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Zoltan Haindrich <zh...@hortonworks.com>.
I think it would be great to move the metastore into a separate project - It seems to me that some security related features/apis which are specific to the metastore are outside of it…so separating the metastore would definetly mean a cleanup in this area as well - which will in turn make the whole system more easily understandable.
I feel that the "storage-api” may also find a better home in the new metastore project - since it defines an api used by Orc/etc to enable the usage of different storage drivers in a somewhat similar way - (however: this might not be a good idea, in case we want to use the storage-api module as the ‘declared’ api to support further storage drivers)
> On 2017Jul 1,, at 12:38 PM, Chaoyu Tang <ct...@gmail.com> wrote:
>
> +1. A nice move which might help HMS to be more easily adopted by more
> other components.
>
> On Sat, Jul 1, 2017 at 12:41 AM, Sushanth Sowmyan <kh...@gmail.com>
> wrote:
>
>> +1
>>
>> On Jun 30, 2017 17:05, "Owen O'Malley" <ow...@gmail.com> wrote:
>>
>>> On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <su...@apache.org> wrote:
>>>
>>>> and maybe a different project name?
>>>>
>>>
>>> Yes, it certainly needs a new name. I'd like to suggest Riven.
>>>
>>> .. Owen
>>>
>>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Chaoyu Tang <ct...@gmail.com>.
+1. A nice move which might help HMS to be more easily adopted by more
other components.
On Sat, Jul 1, 2017 at 12:41 AM, Sushanth Sowmyan <kh...@gmail.com>
wrote:
> +1
>
> On Jun 30, 2017 17:05, "Owen O'Malley" <ow...@gmail.com> wrote:
>
> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <su...@apache.org> wrote:
> >
> > > and maybe a different project name?
> > >
> >
> > Yes, it certainly needs a new name. I'd like to suggest Riven.
> >
> > .. Owen
> >
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Sushanth Sowmyan <kh...@gmail.com>.
+1
On Jun 30, 2017 17:05, "Owen O'Malley" <ow...@gmail.com> wrote:
> On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <su...@apache.org> wrote:
>
> > and maybe a different project name?
> >
>
> Yes, it certainly needs a new name. I'd like to suggest Riven.
>
> .. Owen
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Lefty Leverenz <le...@gmail.com>.
Johndee (and everyone else), wiki edit privileges are easy to get: About
This Wiki -- How to get permission to edit
<https://cwiki.apache.org/confluence/display/Hive/AboutThisWiki#AboutThisWiki-Howtogetpermissiontoedit>
.
-- Lefty
On Thu, Jul 27, 2017 at 2:32 PM, Johndee Cloudera <jo...@cloudera.com>
wrote:
> Well if I cannot get Metastore, how about Hadoop Metastore it is simple and
> self explanatory to a degree.
>
> @Alan,
>
> Sorry to make the name suggestion here but I could not comment or edit the
> page you created.
>
> On Mon, Jul 24, 2017 at 7:17 PM, Gopal Vijayaraghavan <go...@apache.org>
> wrote:
>
> > Hi,
> >
> >
> > Changing the name isn't really optional or "being google-able" [2].
> >
> > The naming is a crucial part of trademark protection [1], which is the
> > only protection ASF has against hostile Embrace & Extends.
> >
> > Fragmented forks with the same name is particularly bad, especially if
> the
> > feature in question can be only used by a proprietary tool (like Dain's
> > suggestion about Presto view metadata, except it only works with a
> per-cpu
> > license).
> >
> > The safe path isn't pretty, it still ends up with IcedTea and IceWeasel …
> > but at least, those are clearly weird.
> >
> > Cheers,
> > Gopal
> >
> > [1] - https://en.wikipedia.org/wiki/A_moron_in_a_hurry#United_States
> > [2] - https://packages.debian.org/jessie/misc/metastore
> >
> > On 7/24/17, 3:04 PM, "Carl Steinbach" <cw...@gmail.com> wrote:
> >
> > +1 to Vihang's suggestion. Changing the name will only cause
> confusion.
> >
> > On Mon, Jul 24, 2017 at 2:28 PM, Johndee Cloudera <
> > johndee@cloudera.com>
> > wrote:
> >
> > > +1 Vihang, I do not really like Catalog as it could create
> confusion
> > with
> > > the Catalog daemon from impala.
> > >
> > > On Mon, Jul 24, 2017 at 5:20 PM, Vihang Karajgaonkar <
> > vihang@cloudera.com>
> > > wrote:
> > >
> > > > Before we see a flood of name suggestions :) Why not just keep it
> > > > Metastore? Its already well-known in the community and easy to
> > relate to.
> > > >
> > > > On Mon, Jul 24, 2017 at 2:13 PM, Alan Gates <
> alanfgates@gmail.com>
> > > wrote:
> > > >
> > > > > In the same vein Carter and Gunther suggested Omegastore. Pick
> > your
> > > > > alphabet and whether it’s a catalog or a store I guess.
> > > > >
> > > > > Alan.
> > > > >
> > > > > On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin <
> > > > sergey@hortonworks.com>
> > > > > wrote:
> > > > >
> > > > > > I’d like to suggest ZCatalog.
> > > > > >
> > > > > > On 17/7/11, 15:41, "Lefty Leverenz" <leftyleverenz@gmail.com
> >
> > wrote:
> > > > > >
> > > > > > >>> I'd like to suggest Riven. (Owen O'Malley)
> > > > > > >
> > > > > > >> How about "Flora"? (Andrew Sherman)
> > > > > > >
> > > > > > >Nice idea and thanks for introducing me to that book,
> Andrew.
> > > > > > >
> > > > > > >Along the same lines, how about "Honeycomb"?
> > > > > > >
> > > > > > >But since the idea is to make the metastore useful for many
> > > projects,
> > > > a
> > > > > > >generic name that starts with "Meta" would be less confusing
> > ...
> > > even
> > > > > > >though it breaks the tradition of Apache projects having
> > quirky
> > > names.
> > > > > > >Unfortunately "Metalog" is already in use. "Metamorph" has
> > other
> > > > > > >connotations, but it's cool.
> > > > > > >
> > > > > > >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of
> > splitting
> > > > off
> > > > > > >the metastore into a new project: -0.5 for the sake of Hive
> > and
> > > +0.5
> > > > > for
> > > > > > >the greater good. Wishy-washy, that's me.
> > > > > > >
> > > > > > >-- Lefty
> > > > > > >
> > > > > > >
> > > > > > >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <
> > > > asherman@cloudera.com>
> > > > > > >wrote:
> > > > > > >
> > > > > > >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <
> > > > > owen.omalley@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <
> > sunchao@apache.org>
> > > > > wrote:
> > > > > > >> >
> > > > > > >> > > and maybe a different project name?
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> > Yes, it certainly needs a new name. I'd like to suggest
> > Riven.
> > > > > > >> >
> > > > > > >> > .. Owen
> > > > > > >> >
> > > > > > >>
> > > > > > >> How about "Flora"?
> > > > > > >>
> > > > > > >> (Flora is the protagonist of The Bees by Laline Paull)
> > > > > > >>
> > > > > > >> -Andrew
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > - JRB
> > >
> >
> >
> >
> >
>
>
> --
> - JRB
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Johndee Cloudera <jo...@cloudera.com>.
Well if I cannot get Metastore, how about Hadoop Metastore it is simple and
self explanatory to a degree.
@Alan,
Sorry to make the name suggestion here but I could not comment or edit the
page you created.
On Mon, Jul 24, 2017 at 7:17 PM, Gopal Vijayaraghavan <go...@apache.org>
wrote:
> Hi,
>
>
> Changing the name isn't really optional or "being google-able" [2].
>
> The naming is a crucial part of trademark protection [1], which is the
> only protection ASF has against hostile Embrace & Extends.
>
> Fragmented forks with the same name is particularly bad, especially if the
> feature in question can be only used by a proprietary tool (like Dain's
> suggestion about Presto view metadata, except it only works with a per-cpu
> license).
>
> The safe path isn't pretty, it still ends up with IcedTea and IceWeasel …
> but at least, those are clearly weird.
>
> Cheers,
> Gopal
>
> [1] - https://en.wikipedia.org/wiki/A_moron_in_a_hurry#United_States
> [2] - https://packages.debian.org/jessie/misc/metastore
>
> On 7/24/17, 3:04 PM, "Carl Steinbach" <cw...@gmail.com> wrote:
>
> +1 to Vihang's suggestion. Changing the name will only cause confusion.
>
> On Mon, Jul 24, 2017 at 2:28 PM, Johndee Cloudera <
> johndee@cloudera.com>
> wrote:
>
> > +1 Vihang, I do not really like Catalog as it could create confusion
> with
> > the Catalog daemon from impala.
> >
> > On Mon, Jul 24, 2017 at 5:20 PM, Vihang Karajgaonkar <
> vihang@cloudera.com>
> > wrote:
> >
> > > Before we see a flood of name suggestions :) Why not just keep it
> > > Metastore? Its already well-known in the community and easy to
> relate to.
> > >
> > > On Mon, Jul 24, 2017 at 2:13 PM, Alan Gates <al...@gmail.com>
> > wrote:
> > >
> > > > In the same vein Carter and Gunther suggested Omegastore. Pick
> your
> > > > alphabet and whether it’s a catalog or a store I guess.
> > > >
> > > > Alan.
> > > >
> > > > On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin <
> > > sergey@hortonworks.com>
> > > > wrote:
> > > >
> > > > > I’d like to suggest ZCatalog.
> > > > >
> > > > > On 17/7/11, 15:41, "Lefty Leverenz" <le...@gmail.com>
> wrote:
> > > > >
> > > > > >>> I'd like to suggest Riven. (Owen O'Malley)
> > > > > >
> > > > > >> How about "Flora"? (Andrew Sherman)
> > > > > >
> > > > > >Nice idea and thanks for introducing me to that book, Andrew.
> > > > > >
> > > > > >Along the same lines, how about "Honeycomb"?
> > > > > >
> > > > > >But since the idea is to make the metastore useful for many
> > projects,
> > > a
> > > > > >generic name that starts with "Meta" would be less confusing
> ...
> > even
> > > > > >though it breaks the tradition of Apache projects having
> quirky
> > names.
> > > > > >Unfortunately "Metalog" is already in use. "Metamorph" has
> other
> > > > > >connotations, but it's cool.
> > > > > >
> > > > > >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of
> splitting
> > > off
> > > > > >the metastore into a new project: -0.5 for the sake of Hive
> and
> > +0.5
> > > > for
> > > > > >the greater good. Wishy-washy, that's me.
> > > > > >
> > > > > >-- Lefty
> > > > > >
> > > > > >
> > > > > >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <
> > > asherman@cloudera.com>
> > > > > >wrote:
> > > > > >
> > > > > >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <
> > > > owen.omalley@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <
> sunchao@apache.org>
> > > > wrote:
> > > > > >> >
> > > > > >> > > and maybe a different project name?
> > > > > >> > >
> > > > > >> >
> > > > > >> > Yes, it certainly needs a new name. I'd like to suggest
> Riven.
> > > > > >> >
> > > > > >> > .. Owen
> > > > > >> >
> > > > > >>
> > > > > >> How about "Flora"?
> > > > > >>
> > > > > >> (Flora is the protagonist of The Bees by Laline Paull)
> > > > > >>
> > > > > >> -Andrew
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > - JRB
> >
>
>
>
>
--
- JRB
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Gopal Vijayaraghavan <go...@apache.org>.
On 7/25/17, 4:45 PM, "cwsteinbach@gmail.com on behalf of Carl Steinbach" <cwsteinbach@gmail.com on behalf of cws@apache.org> wrote:
> "IceWeasel" and "MetaStore" are both examples of English compound words.
> What exactly makes the former any safer than the latter?
Usually descriptive words are considered weaker for trademarks - if the words describe what it does, then it might be weaker.
"PainKiller" is a weak one, while "Aspirin" isn't.
Uniqueness is useful, because an active defense is necessary to retain possession of a trademark - as a tautology, the more unique the phrase, the fewer occurrences there are to tackle.
But, in the case of Aspirin, Bayer did not defend the use lowercase "aspirin" and now only has a TM on the upper-case one "Aspirin".
IceWeasel is an infamous precedent of trademark dispute in the open source community
"The end of the Iceweasel Age" - https://lwn.net/Articles/676799/
+
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=354622
Cheers,
Gopal
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Alan Gates <al...@gmail.com>.
I think the concerns with Metastore are twofold. One, it’s a common term
in software, and thus would be hard to defend as a trademark. Hence
Gopal’s link to the Debian package already named metastore. IANAL but as I
understand trademark law the test is whether a name could cause a
reasonable person to be confused as to who is offering a good or service.
So a McDonalds tire store is ok, a reasonable person knows McDonalds sells
food, not tires, but a restaurant named McDonnies that serves hamburgers
isn’t. Whether IceWeasal (or any of the suggested names besides metastore)
passes that test I don’t know. The board will require us to go through a
namesearch as part of the TLP process.
If I understand Gopal’s second point it is that it will cause confusion for
users as to whether this is still part of Hive or something separate. I
think stressing the continuity is exactly what Vihang and Carl like about
keeping the name.
My suggestion would be that the project needs a name other than metastore,
but we can call the module metastore. Having a more unique name is good
for trademarks and helping users find your stuff via google, etc. (Go to
Google and search on “Hive" to see what I mean here.) If we pick X as the
project name, we can then refer to it as the X metastore, the maven modules
can be x-metastore, etc. I think this strikes the balance between the
benefits of a unique-ish name and telling users what it does and where it
came from.
Plus, IceWeasal is a _way_ cooler name for a project that metastore. :)
Alan.
On Tue, Jul 25, 2017 at 4:45 PM, Carl Steinbach <cw...@apache.org> wrote:
> "IceWeasel" and "MetaStore" are both examples of English compound words.
> What exactly makes the former any safer than the latter?
>
> On Mon, Jul 24, 2017 at 4:17 PM, Gopal Vijayaraghavan <go...@apache.org>
> wrote:
>
> > Hi,
> >
> >
> > Changing the name isn't really optional or "being google-able" [2].
> >
> > The naming is a crucial part of trademark protection [1], which is the
> > only protection ASF has against hostile Embrace & Extends.
> >
> > Fragmented forks with the same name is particularly bad, especially if
> the
> > feature in question can be only used by a proprietary tool (like Dain's
> > suggestion about Presto view metadata, except it only works with a
> per-cpu
> > license).
> >
> > The safe path isn't pretty, it still ends up with IcedTea and IceWeasel …
> > but at least, those are clearly weird.
> >
> > Cheers,
> > Gopal
> >
> > [1] - https://en.wikipedia.org/wiki/A_moron_in_a_hurry#United_States
> > [2] - https://packages.debian.org/jessie/misc/metastore
> >
> > On 7/24/17, 3:04 PM, "Carl Steinbach" <cw...@gmail.com> wrote:
> >
> > +1 to Vihang's suggestion. Changing the name will only cause
> confusion.
> >
> > On Mon, Jul 24, 2017 at 2:28 PM, Johndee Cloudera <
> > johndee@cloudera.com>
> > wrote:
> >
> > > +1 Vihang, I do not really like Catalog as it could create
> confusion
> > with
> > > the Catalog daemon from impala.
> > >
> > > On Mon, Jul 24, 2017 at 5:20 PM, Vihang Karajgaonkar <
> > vihang@cloudera.com>
> > > wrote:
> > >
> > > > Before we see a flood of name suggestions :) Why not just keep it
> > > > Metastore? Its already well-known in the community and easy to
> > relate to.
> > > >
> > > > On Mon, Jul 24, 2017 at 2:13 PM, Alan Gates <
> alanfgates@gmail.com>
> > > wrote:
> > > >
> > > > > In the same vein Carter and Gunther suggested Omegastore. Pick
> > your
> > > > > alphabet and whether it’s a catalog or a store I guess.
> > > > >
> > > > > Alan.
> > > > >
> > > > > On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin <
> > > > sergey@hortonworks.com>
> > > > > wrote:
> > > > >
> > > > > > I’d like to suggest ZCatalog.
> > > > > >
> > > > > > On 17/7/11, 15:41, "Lefty Leverenz" <leftyleverenz@gmail.com
> >
> > wrote:
> > > > > >
> > > > > > >>> I'd like to suggest Riven. (Owen O'Malley)
> > > > > > >
> > > > > > >> How about "Flora"? (Andrew Sherman)
> > > > > > >
> > > > > > >Nice idea and thanks for introducing me to that book,
> Andrew.
> > > > > > >
> > > > > > >Along the same lines, how about "Honeycomb"?
> > > > > > >
> > > > > > >But since the idea is to make the metastore useful for many
> > > projects,
> > > > a
> > > > > > >generic name that starts with "Meta" would be less confusing
> > ...
> > > even
> > > > > > >though it breaks the tradition of Apache projects having
> > quirky
> > > names.
> > > > > > >Unfortunately "Metalog" is already in use. "Metamorph" has
> > other
> > > > > > >connotations, but it's cool.
> > > > > > >
> > > > > > >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of
> > splitting
> > > > off
> > > > > > >the metastore into a new project: -0.5 for the sake of Hive
> > and
> > > +0.5
> > > > > for
> > > > > > >the greater good. Wishy-washy, that's me.
> > > > > > >
> > > > > > >-- Lefty
> > > > > > >
> > > > > > >
> > > > > > >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <
> > > > asherman@cloudera.com>
> > > > > > >wrote:
> > > > > > >
> > > > > > >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <
> > > > > owen.omalley@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <
> > sunchao@apache.org>
> > > > > wrote:
> > > > > > >> >
> > > > > > >> > > and maybe a different project name?
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> > Yes, it certainly needs a new name. I'd like to suggest
> > Riven.
> > > > > > >> >
> > > > > > >> > .. Owen
> > > > > > >> >
> > > > > > >>
> > > > > > >> How about "Flora"?
> > > > > > >>
> > > > > > >> (Flora is the protagonist of The Bees by Laline Paull)
> > > > > > >>
> > > > > > >> -Andrew
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > - JRB
> > >
> >
> >
> >
> >
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Carl Steinbach <cw...@apache.org>.
"IceWeasel" and "MetaStore" are both examples of English compound words.
What exactly makes the former any safer than the latter?
On Mon, Jul 24, 2017 at 4:17 PM, Gopal Vijayaraghavan <go...@apache.org>
wrote:
> Hi,
>
>
> Changing the name isn't really optional or "being google-able" [2].
>
> The naming is a crucial part of trademark protection [1], which is the
> only protection ASF has against hostile Embrace & Extends.
>
> Fragmented forks with the same name is particularly bad, especially if the
> feature in question can be only used by a proprietary tool (like Dain's
> suggestion about Presto view metadata, except it only works with a per-cpu
> license).
>
> The safe path isn't pretty, it still ends up with IcedTea and IceWeasel …
> but at least, those are clearly weird.
>
> Cheers,
> Gopal
>
> [1] - https://en.wikipedia.org/wiki/A_moron_in_a_hurry#United_States
> [2] - https://packages.debian.org/jessie/misc/metastore
>
> On 7/24/17, 3:04 PM, "Carl Steinbach" <cw...@gmail.com> wrote:
>
> +1 to Vihang's suggestion. Changing the name will only cause confusion.
>
> On Mon, Jul 24, 2017 at 2:28 PM, Johndee Cloudera <
> johndee@cloudera.com>
> wrote:
>
> > +1 Vihang, I do not really like Catalog as it could create confusion
> with
> > the Catalog daemon from impala.
> >
> > On Mon, Jul 24, 2017 at 5:20 PM, Vihang Karajgaonkar <
> vihang@cloudera.com>
> > wrote:
> >
> > > Before we see a flood of name suggestions :) Why not just keep it
> > > Metastore? Its already well-known in the community and easy to
> relate to.
> > >
> > > On Mon, Jul 24, 2017 at 2:13 PM, Alan Gates <al...@gmail.com>
> > wrote:
> > >
> > > > In the same vein Carter and Gunther suggested Omegastore. Pick
> your
> > > > alphabet and whether it’s a catalog or a store I guess.
> > > >
> > > > Alan.
> > > >
> > > > On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin <
> > > sergey@hortonworks.com>
> > > > wrote:
> > > >
> > > > > I’d like to suggest ZCatalog.
> > > > >
> > > > > On 17/7/11, 15:41, "Lefty Leverenz" <le...@gmail.com>
> wrote:
> > > > >
> > > > > >>> I'd like to suggest Riven. (Owen O'Malley)
> > > > > >
> > > > > >> How about "Flora"? (Andrew Sherman)
> > > > > >
> > > > > >Nice idea and thanks for introducing me to that book, Andrew.
> > > > > >
> > > > > >Along the same lines, how about "Honeycomb"?
> > > > > >
> > > > > >But since the idea is to make the metastore useful for many
> > projects,
> > > a
> > > > > >generic name that starts with "Meta" would be less confusing
> ...
> > even
> > > > > >though it breaks the tradition of Apache projects having
> quirky
> > names.
> > > > > >Unfortunately "Metalog" is already in use. "Metamorph" has
> other
> > > > > >connotations, but it's cool.
> > > > > >
> > > > > >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of
> splitting
> > > off
> > > > > >the metastore into a new project: -0.5 for the sake of Hive
> and
> > +0.5
> > > > for
> > > > > >the greater good. Wishy-washy, that's me.
> > > > > >
> > > > > >-- Lefty
> > > > > >
> > > > > >
> > > > > >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <
> > > asherman@cloudera.com>
> > > > > >wrote:
> > > > > >
> > > > > >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <
> > > > owen.omalley@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <
> sunchao@apache.org>
> > > > wrote:
> > > > > >> >
> > > > > >> > > and maybe a different project name?
> > > > > >> > >
> > > > > >> >
> > > > > >> > Yes, it certainly needs a new name. I'd like to suggest
> Riven.
> > > > > >> >
> > > > > >> > .. Owen
> > > > > >> >
> > > > > >>
> > > > > >> How about "Flora"?
> > > > > >>
> > > > > >> (Flora is the protagonist of The Bees by Laline Paull)
> > > > > >>
> > > > > >> -Andrew
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > - JRB
> >
>
>
>
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Alan Gates <al...@gmail.com>.
I’ve collected the naming suggestions on the wiki at
https://cwiki.apache.org/confluence/display/Hive/Metastore+TLP+Proposal
Please feel free to add additional ones there.
Alan.
On Mon, Jul 24, 2017 at 4:17 PM, Gopal Vijayaraghavan <go...@apache.org>
wrote:
> Hi,
>
>
> Changing the name isn't really optional or "being google-able" [2].
>
> The naming is a crucial part of trademark protection [1], which is the
> only protection ASF has against hostile Embrace & Extends.
>
> Fragmented forks with the same name is particularly bad, especially if the
> feature in question can be only used by a proprietary tool (like Dain's
> suggestion about Presto view metadata, except it only works with a per-cpu
> license).
>
> The safe path isn't pretty, it still ends up with IcedTea and IceWeasel …
> but at least, those are clearly weird.
>
> Cheers,
> Gopal
>
> [1] - https://en.wikipedia.org/wiki/A_moron_in_a_hurry#United_States
> [2] - https://packages.debian.org/jessie/misc/metastore
>
> On 7/24/17, 3:04 PM, "Carl Steinbach" <cw...@gmail.com> wrote:
>
> +1 to Vihang's suggestion. Changing the name will only cause confusion.
>
> On Mon, Jul 24, 2017 at 2:28 PM, Johndee Cloudera <
> johndee@cloudera.com>
> wrote:
>
> > +1 Vihang, I do not really like Catalog as it could create confusion
> with
> > the Catalog daemon from impala.
> >
> > On Mon, Jul 24, 2017 at 5:20 PM, Vihang Karajgaonkar <
> vihang@cloudera.com>
> > wrote:
> >
> > > Before we see a flood of name suggestions :) Why not just keep it
> > > Metastore? Its already well-known in the community and easy to
> relate to.
> > >
> > > On Mon, Jul 24, 2017 at 2:13 PM, Alan Gates <al...@gmail.com>
> > wrote:
> > >
> > > > In the same vein Carter and Gunther suggested Omegastore. Pick
> your
> > > > alphabet and whether it’s a catalog or a store I guess.
> > > >
> > > > Alan.
> > > >
> > > > On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin <
> > > sergey@hortonworks.com>
> > > > wrote:
> > > >
> > > > > I’d like to suggest ZCatalog.
> > > > >
> > > > > On 17/7/11, 15:41, "Lefty Leverenz" <le...@gmail.com>
> wrote:
> > > > >
> > > > > >>> I'd like to suggest Riven. (Owen O'Malley)
> > > > > >
> > > > > >> How about "Flora"? (Andrew Sherman)
> > > > > >
> > > > > >Nice idea and thanks for introducing me to that book, Andrew.
> > > > > >
> > > > > >Along the same lines, how about "Honeycomb"?
> > > > > >
> > > > > >But since the idea is to make the metastore useful for many
> > projects,
> > > a
> > > > > >generic name that starts with "Meta" would be less confusing
> ...
> > even
> > > > > >though it breaks the tradition of Apache projects having
> quirky
> > names.
> > > > > >Unfortunately "Metalog" is already in use. "Metamorph" has
> other
> > > > > >connotations, but it's cool.
> > > > > >
> > > > > >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of
> splitting
> > > off
> > > > > >the metastore into a new project: -0.5 for the sake of Hive
> and
> > +0.5
> > > > for
> > > > > >the greater good. Wishy-washy, that's me.
> > > > > >
> > > > > >-- Lefty
> > > > > >
> > > > > >
> > > > > >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <
> > > asherman@cloudera.com>
> > > > > >wrote:
> > > > > >
> > > > > >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <
> > > > owen.omalley@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <
> sunchao@apache.org>
> > > > wrote:
> > > > > >> >
> > > > > >> > > and maybe a different project name?
> > > > > >> > >
> > > > > >> >
> > > > > >> > Yes, it certainly needs a new name. I'd like to suggest
> Riven.
> > > > > >> >
> > > > > >> > .. Owen
> > > > > >> >
> > > > > >>
> > > > > >> How about "Flora"?
> > > > > >>
> > > > > >> (Flora is the protagonist of The Bees by Laline Paull)
> > > > > >>
> > > > > >> -Andrew
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > - JRB
> >
>
>
>
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Gopal Vijayaraghavan <go...@apache.org>.
Hi,
Changing the name isn't really optional or "being google-able" [2].
The naming is a crucial part of trademark protection [1], which is the only protection ASF has against hostile Embrace & Extends.
Fragmented forks with the same name is particularly bad, especially if the feature in question can be only used by a proprietary tool (like Dain's suggestion about Presto view metadata, except it only works with a per-cpu license).
The safe path isn't pretty, it still ends up with IcedTea and IceWeasel … but at least, those are clearly weird.
Cheers,
Gopal
[1] - https://en.wikipedia.org/wiki/A_moron_in_a_hurry#United_States
[2] - https://packages.debian.org/jessie/misc/metastore
On 7/24/17, 3:04 PM, "Carl Steinbach" <cw...@gmail.com> wrote:
+1 to Vihang's suggestion. Changing the name will only cause confusion.
On Mon, Jul 24, 2017 at 2:28 PM, Johndee Cloudera <jo...@cloudera.com>
wrote:
> +1 Vihang, I do not really like Catalog as it could create confusion with
> the Catalog daemon from impala.
>
> On Mon, Jul 24, 2017 at 5:20 PM, Vihang Karajgaonkar <vi...@cloudera.com>
> wrote:
>
> > Before we see a flood of name suggestions :) Why not just keep it
> > Metastore? Its already well-known in the community and easy to relate to.
> >
> > On Mon, Jul 24, 2017 at 2:13 PM, Alan Gates <al...@gmail.com>
> wrote:
> >
> > > In the same vein Carter and Gunther suggested Omegastore. Pick your
> > > alphabet and whether it’s a catalog or a store I guess.
> > >
> > > Alan.
> > >
> > > On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin <
> > sergey@hortonworks.com>
> > > wrote:
> > >
> > > > I’d like to suggest ZCatalog.
> > > >
> > > > On 17/7/11, 15:41, "Lefty Leverenz" <le...@gmail.com> wrote:
> > > >
> > > > >>> I'd like to suggest Riven. (Owen O'Malley)
> > > > >
> > > > >> How about "Flora"? (Andrew Sherman)
> > > > >
> > > > >Nice idea and thanks for introducing me to that book, Andrew.
> > > > >
> > > > >Along the same lines, how about "Honeycomb"?
> > > > >
> > > > >But since the idea is to make the metastore useful for many
> projects,
> > a
> > > > >generic name that starts with "Meta" would be less confusing ...
> even
> > > > >though it breaks the tradition of Apache projects having quirky
> names.
> > > > >Unfortunately "Metalog" is already in use. "Metamorph" has other
> > > > >connotations, but it's cool.
> > > > >
> > > > >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of splitting
> > off
> > > > >the metastore into a new project: -0.5 for the sake of Hive and
> +0.5
> > > for
> > > > >the greater good. Wishy-washy, that's me.
> > > > >
> > > > >-- Lefty
> > > > >
> > > > >
> > > > >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <
> > asherman@cloudera.com>
> > > > >wrote:
> > > > >
> > > > >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <
> > > owen.omalley@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <su...@apache.org>
> > > wrote:
> > > > >> >
> > > > >> > > and maybe a different project name?
> > > > >> > >
> > > > >> >
> > > > >> > Yes, it certainly needs a new name. I'd like to suggest Riven.
> > > > >> >
> > > > >> > .. Owen
> > > > >> >
> > > > >>
> > > > >> How about "Flora"?
> > > > >>
> > > > >> (Flora is the protagonist of The Bees by Laline Paull)
> > > > >>
> > > > >> -Andrew
> > > > >>
> > > >
> > > >
> > >
> >
>
>
>
> --
> - JRB
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Carl Steinbach <cw...@gmail.com>.
+1 to Vihang's suggestion. Changing the name will only cause confusion.
On Mon, Jul 24, 2017 at 2:28 PM, Johndee Cloudera <jo...@cloudera.com>
wrote:
> +1 Vihang, I do not really like Catalog as it could create confusion with
> the Catalog daemon from impala.
>
> On Mon, Jul 24, 2017 at 5:20 PM, Vihang Karajgaonkar <vi...@cloudera.com>
> wrote:
>
> > Before we see a flood of name suggestions :) Why not just keep it
> > Metastore? Its already well-known in the community and easy to relate to.
> >
> > On Mon, Jul 24, 2017 at 2:13 PM, Alan Gates <al...@gmail.com>
> wrote:
> >
> > > In the same vein Carter and Gunther suggested Omegastore. Pick your
> > > alphabet and whether it’s a catalog or a store I guess.
> > >
> > > Alan.
> > >
> > > On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin <
> > sergey@hortonworks.com>
> > > wrote:
> > >
> > > > I’d like to suggest ZCatalog.
> > > >
> > > > On 17/7/11, 15:41, "Lefty Leverenz" <le...@gmail.com> wrote:
> > > >
> > > > >>> I'd like to suggest Riven. (Owen O'Malley)
> > > > >
> > > > >> How about "Flora"? (Andrew Sherman)
> > > > >
> > > > >Nice idea and thanks for introducing me to that book, Andrew.
> > > > >
> > > > >Along the same lines, how about "Honeycomb"?
> > > > >
> > > > >But since the idea is to make the metastore useful for many
> projects,
> > a
> > > > >generic name that starts with "Meta" would be less confusing ...
> even
> > > > >though it breaks the tradition of Apache projects having quirky
> names.
> > > > >Unfortunately "Metalog" is already in use. "Metamorph" has other
> > > > >connotations, but it's cool.
> > > > >
> > > > >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of splitting
> > off
> > > > >the metastore into a new project: -0.5 for the sake of Hive and
> +0.5
> > > for
> > > > >the greater good. Wishy-washy, that's me.
> > > > >
> > > > >-- Lefty
> > > > >
> > > > >
> > > > >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <
> > asherman@cloudera.com>
> > > > >wrote:
> > > > >
> > > > >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <
> > > owen.omalley@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <su...@apache.org>
> > > wrote:
> > > > >> >
> > > > >> > > and maybe a different project name?
> > > > >> > >
> > > > >> >
> > > > >> > Yes, it certainly needs a new name. I'd like to suggest Riven.
> > > > >> >
> > > > >> > .. Owen
> > > > >> >
> > > > >>
> > > > >> How about "Flora"?
> > > > >>
> > > > >> (Flora is the protagonist of The Bees by Laline Paull)
> > > > >>
> > > > >> -Andrew
> > > > >>
> > > >
> > > >
> > >
> >
>
>
>
> --
> - JRB
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Johndee Cloudera <jo...@cloudera.com>.
+1 Vihang, I do not really like Catalog as it could create confusion with
the Catalog daemon from impala.
On Mon, Jul 24, 2017 at 5:20 PM, Vihang Karajgaonkar <vi...@cloudera.com>
wrote:
> Before we see a flood of name suggestions :) Why not just keep it
> Metastore? Its already well-known in the community and easy to relate to.
>
> On Mon, Jul 24, 2017 at 2:13 PM, Alan Gates <al...@gmail.com> wrote:
>
> > In the same vein Carter and Gunther suggested Omegastore. Pick your
> > alphabet and whether it’s a catalog or a store I guess.
> >
> > Alan.
> >
> > On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin <
> sergey@hortonworks.com>
> > wrote:
> >
> > > I’d like to suggest ZCatalog.
> > >
> > > On 17/7/11, 15:41, "Lefty Leverenz" <le...@gmail.com> wrote:
> > >
> > > >>> I'd like to suggest Riven. (Owen O'Malley)
> > > >
> > > >> How about "Flora"? (Andrew Sherman)
> > > >
> > > >Nice idea and thanks for introducing me to that book, Andrew.
> > > >
> > > >Along the same lines, how about "Honeycomb"?
> > > >
> > > >But since the idea is to make the metastore useful for many projects,
> a
> > > >generic name that starts with "Meta" would be less confusing ... even
> > > >though it breaks the tradition of Apache projects having quirky names.
> > > >Unfortunately "Metalog" is already in use. "Metamorph" has other
> > > >connotations, but it's cool.
> > > >
> > > >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of splitting
> off
> > > >the metastore into a new project: -0.5 for the sake of Hive and +0.5
> > for
> > > >the greater good. Wishy-washy, that's me.
> > > >
> > > >-- Lefty
> > > >
> > > >
> > > >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <
> asherman@cloudera.com>
> > > >wrote:
> > > >
> > > >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <
> > owen.omalley@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <su...@apache.org>
> > wrote:
> > > >> >
> > > >> > > and maybe a different project name?
> > > >> > >
> > > >> >
> > > >> > Yes, it certainly needs a new name. I'd like to suggest Riven.
> > > >> >
> > > >> > .. Owen
> > > >> >
> > > >>
> > > >> How about "Flora"?
> > > >>
> > > >> (Flora is the protagonist of The Bees by Laline Paull)
> > > >>
> > > >> -Andrew
> > > >>
> > >
> > >
> >
>
--
- JRB
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Vihang Karajgaonkar <vi...@cloudera.com>.
Before we see a flood of name suggestions :) Why not just keep it
Metastore? Its already well-known in the community and easy to relate to.
On Mon, Jul 24, 2017 at 2:13 PM, Alan Gates <al...@gmail.com> wrote:
> In the same vein Carter and Gunther suggested Omegastore. Pick your
> alphabet and whether it’s a catalog or a store I guess.
>
> Alan.
>
> On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin <se...@hortonworks.com>
> wrote:
>
> > I’d like to suggest ZCatalog.
> >
> > On 17/7/11, 15:41, "Lefty Leverenz" <le...@gmail.com> wrote:
> >
> > >>> I'd like to suggest Riven. (Owen O'Malley)
> > >
> > >> How about "Flora"? (Andrew Sherman)
> > >
> > >Nice idea and thanks for introducing me to that book, Andrew.
> > >
> > >Along the same lines, how about "Honeycomb"?
> > >
> > >But since the idea is to make the metastore useful for many projects, a
> > >generic name that starts with "Meta" would be less confusing ... even
> > >though it breaks the tradition of Apache projects having quirky names.
> > >Unfortunately "Metalog" is already in use. "Metamorph" has other
> > >connotations, but it's cool.
> > >
> > >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of splitting off
> > >the metastore into a new project: -0.5 for the sake of Hive and +0.5
> for
> > >the greater good. Wishy-washy, that's me.
> > >
> > >-- Lefty
> > >
> > >
> > >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <as...@cloudera.com>
> > >wrote:
> > >
> > >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <
> owen.omalley@gmail.com>
> > >> wrote:
> > >>
> > >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <su...@apache.org>
> wrote:
> > >> >
> > >> > > and maybe a different project name?
> > >> > >
> > >> >
> > >> > Yes, it certainly needs a new name. I'd like to suggest Riven.
> > >> >
> > >> > .. Owen
> > >> >
> > >>
> > >> How about "Flora"?
> > >>
> > >> (Flora is the protagonist of The Bees by Laline Paull)
> > >>
> > >> -Andrew
> > >>
> >
> >
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Alan Gates <al...@gmail.com>.
In the same vein Carter and Gunther suggested Omegastore. Pick your
alphabet and whether it’s a catalog or a store I guess.
Alan.
On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin <se...@hortonworks.com>
wrote:
> I’d like to suggest ZCatalog.
>
> On 17/7/11, 15:41, "Lefty Leverenz" <le...@gmail.com> wrote:
>
> >>> I'd like to suggest Riven. (Owen O'Malley)
> >
> >> How about "Flora"? (Andrew Sherman)
> >
> >Nice idea and thanks for introducing me to that book, Andrew.
> >
> >Along the same lines, how about "Honeycomb"?
> >
> >But since the idea is to make the metastore useful for many projects, a
> >generic name that starts with "Meta" would be less confusing ... even
> >though it breaks the tradition of Apache projects having quirky names.
> >Unfortunately "Metalog" is already in use. "Metamorph" has other
> >connotations, but it's cool.
> >
> >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of splitting off
> >the metastore into a new project: -0.5 for the sake of Hive and +0.5 for
> >the greater good. Wishy-washy, that's me.
> >
> >-- Lefty
> >
> >
> >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <as...@cloudera.com>
> >wrote:
> >
> >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <ow...@gmail.com>
> >> wrote:
> >>
> >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <su...@apache.org> wrote:
> >> >
> >> > > and maybe a different project name?
> >> > >
> >> >
> >> > Yes, it certainly needs a new name. I'd like to suggest Riven.
> >> >
> >> > .. Owen
> >> >
> >>
> >> How about "Flora"?
> >>
> >> (Flora is the protagonist of The Bees by Laline Paull)
> >>
> >> -Andrew
> >>
>
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Sergey Shelukhin <se...@hortonworks.com>.
I’d like to suggest ZCatalog.
On 17/7/11, 15:41, "Lefty Leverenz" <le...@gmail.com> wrote:
>>> I'd like to suggest Riven. (Owen O'Malley)
>
>> How about "Flora"? (Andrew Sherman)
>
>Nice idea and thanks for introducing me to that book, Andrew.
>
>Along the same lines, how about "Honeycomb"?
>
>But since the idea is to make the metastore useful for many projects, a
>generic name that starts with "Meta" would be less confusing ... even
>though it breaks the tradition of Apache projects having quirky names.
>Unfortunately "Metalog" is already in use. "Metamorph" has other
>connotations, but it's cool.
>
>Naming enthusiasm notwithstanding, I'm +/-0 on the idea of splitting off
>the metastore into a new project: -0.5 for the sake of Hive and +0.5 for
>the greater good. Wishy-washy, that's me.
>
>-- Lefty
>
>
>On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <as...@cloudera.com>
>wrote:
>
>> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <ow...@gmail.com>
>> wrote:
>>
>> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <su...@apache.org> wrote:
>> >
>> > > and maybe a different project name?
>> > >
>> >
>> > Yes, it certainly needs a new name. I'd like to suggest Riven.
>> >
>> > .. Owen
>> >
>>
>> How about "Flora"?
>>
>> (Flora is the protagonist of The Bees by Laline Paull)
>>
>> -Andrew
>>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Alan Gates <al...@gmail.com>.
It seems we have settled into a consensus that this will be good for the
ecosystem, but there are concerns that this will be a burden on Hive. The
original proposal included a time to work on the separation inside the Hive
project to help address any such issues. This would culminate in a source
only release of the metastore inside Hive. I propose we start working on
that internal separation now. I’ll file an umbrella JIRA for it soon.
Alan.
On Tue, Jul 11, 2017 at 3:41 PM, Lefty Leverenz <le...@gmail.com>
wrote:
> >> I'd like to suggest Riven. (Owen O'Malley)
>
> > How about "Flora"? (Andrew Sherman)
>
> Nice idea and thanks for introducing me to that book, Andrew.
>
> Along the same lines, how about "Honeycomb"?
>
> But since the idea is to make the metastore useful for many projects, a
> generic name that starts with "Meta" would be less confusing ... even
> though it breaks the tradition of Apache projects having quirky names.
> Unfortunately "Metalog" is already in use. "Metamorph" has other
> connotations, but it's cool.
>
> Naming enthusiasm notwithstanding, I'm +/-0 on the idea of splitting off
> the metastore into a new project: -0.5 for the sake of Hive and +0.5 for
> the greater good. Wishy-washy, that's me.
>
> -- Lefty
>
>
> On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <as...@cloudera.com>
> wrote:
>
> > On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <ow...@gmail.com>
> > wrote:
> >
> > > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <su...@apache.org> wrote:
> > >
> > > > and maybe a different project name?
> > > >
> > >
> > > Yes, it certainly needs a new name. I'd like to suggest Riven.
> > >
> > > .. Owen
> > >
> >
> > How about "Flora"?
> >
> > (Flora is the protagonist of The Bees by Laline Paull)
> >
> > -Andrew
> >
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Lefty Leverenz <le...@gmail.com>.
>> I'd like to suggest Riven. (Owen O'Malley)
> How about "Flora"? (Andrew Sherman)
Nice idea and thanks for introducing me to that book, Andrew.
Along the same lines, how about "Honeycomb"?
But since the idea is to make the metastore useful for many projects, a
generic name that starts with "Meta" would be less confusing ... even
though it breaks the tradition of Apache projects having quirky names.
Unfortunately "Metalog" is already in use. "Metamorph" has other
connotations, but it's cool.
Naming enthusiasm notwithstanding, I'm +/-0 on the idea of splitting off
the metastore into a new project: -0.5 for the sake of Hive and +0.5 for
the greater good. Wishy-washy, that's me.
-- Lefty
On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <as...@cloudera.com>
wrote:
> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <ow...@gmail.com>
> wrote:
>
> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <su...@apache.org> wrote:
> >
> > > and maybe a different project name?
> > >
> >
> > Yes, it certainly needs a new name. I'd like to suggest Riven.
> >
> > .. Owen
> >
>
> How about "Flora"?
>
> (Flora is the protagonist of The Bees by Laline Paull)
>
> -Andrew
>
Re: [DISCUSS] Separating out the metastore as its own TLP
Posted by Andrew Sherman <as...@cloudera.com>.
On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <ow...@gmail.com>
wrote:
> On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun <su...@apache.org> wrote:
>
> > and maybe a different project name?
> >
>
> Yes, it certainly needs a new name. I'd like to suggest Riven.
>
> .. Owen
>
How about "Flora"?
(Flora is the protagonist of The Bees by Laline Paull)
-Andrew