You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@incubator.apache.org by Tom White <to...@apache.org> on 2015/12/02 11:01:39 UTC
Impala commit policy
The vote to accept Impala into the incubator has passed
(http://s.apache.org/u6r), however there are still some concerns about
CTR/RTC. My main takeaways from the CTR/RTC thread are that it's not a
binary choice, and that it's entirely reasonable that different
communities have different commit policies at the ASF.
I think Julian Hyde's suggestion that the Impala podling start with no
explicit commit policy is a good one. Incubation should be used as a
time to work out what works best for a project. The initial Impala
community should discuss the commit policy as they go through the
process of setting up ASF infra and start growing the podling. In
particular this will include how Gerrit can be used as a tool to
facilitate reviews, and how that fits with ASF culture, which is
something that other projects are looking at too.
Cheers,
Tom
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Impala commit policy
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Wed, Dec 2, 2015 at 11:01 AM, Tom White <to...@apache.org> wrote:
> ...I think Julian Hyde's suggestion that the Impala podling start with no
> explicit commit policy is a good one. Incubation should be used as a
> time to work out what works best for a project....
Big +1
-Bertrand
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Impala commit policy
Posted by Steve Loughran <st...@hortonworks.com>.
On 2 Dec 2015, at 10:01, Tom White <to...@apache.org>> wrote:
The vote to accept Impala into the incubator has passed
(http://s.apache.org/u6r), however there are still some concerns about
CTR/RTC. My main takeaways from the CTR/RTC thread are that it's not a
binary choice, and that it's entirely reasonable that different
communities have different commit policies at the ASF.
I think Julian Hyde's suggestion that the Impala podling start with no
explicit commit policy is a good one. Incubation should be used as a
time to work out what works best for a project. The initial Impala
community should discuss the commit policy as they go through the
process of setting up ASF infra and start growing the podling. In
particular this will include how Gerrit can be used as a tool to
facilitate reviews, and how that fits with ASF culture, which is
something that other projects are looking at too.
+1
what we might want to think about —for future proposals— is what makes a good recommended process during incubation, to help build that community, for dev & commit
that is: what is best to not only ship something usable, but to pull in outside contributors. This is more than commit policy, it's how to organise discussions, what automated patch checking mechanisms to set up, etc, etc.
That means more than just cargo-cult duplication of existing projects, but allowing incubating projects to innovate in this area, and perhaps having some more rigorous review; successors to [Rigby08] (*)
-Steve
[Rigby08], Open Source Software Peer Review Practices: A Case Study of the Apache Server, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.226.2868&rep=rep1&type=pdf,
related work, https://scholar.google.co.uk/scholar?q=related:Rf0V3sIgPL6vvM:scholar.google.com/
Re: Impala commit policy
Posted by Julien Le Dem <ju...@dremio.com>.
+1 to this as well.
Whether the community changes its mind or not is irrelevant in my opinion.
What is important is it gets to choose for itself and possibly revisits
regularly as it sees fit.
This discussion should be encouraged and people who want to promote the
merits of one approach or another can do so.
On Wed, Dec 2, 2015 at 11:19 AM, Henry Robinson <he...@cloudera.com> wrote:
> What might happen, however, is that the discussion is revisited with a
> particular focus on the concerns that you've raised. So although it might
> be unlikely that the community performs a volte-face and elects for CTR, we
> might say "what can we do to limit the risk that RTC inhibits community
> growth, without abandoning RTC completely?". At the very least, the
> community becomes aware of the concerns, and more sensitive to their
> potential impact.
>
>
> On 2 December 2015 at 11:09, Greg Stein <gs...@gmail.com> wrote:
>
> > Yeah, this is what I meant earlier. Leaving out a commit policy changes
> > nothing. The same people who put together the proposal will be the same
> set
> > as those discussing it as a podling, and they will reach the same
> > conclusion.
> >
> > If the PPMC doubles in size, with fresh faces, then a real discussion can
> > happen. Tho I doubt that will be possible -- I'm unaware of any podling
> > pulling off such growth.
> > On Dec 2, 2015 12:45 PM, "Henry Robinson" <he...@cloudera.com> wrote:
> >
> > > I agree that this is something the Impala community will want to
> discuss
> > > fairly early on in incubation - along with a lot of other project
> > > procedural stuff as we adjust or rethink our workflows to be Apache-Way
> > > compatible.
> > >
> > > Until we have that discussion, I'd expect Impala will continue along
> RTC
> > > lines simply because this is what the existing community is used to
> (and
> > I
> > > believe, prefers), and the workflows are well established and work
> > > smoothly.
> > >
> > > That is, I personally am ok with no explicit commit protocol, but in
> > > practice it is likely to appear as if there is no change, until the
> > > community discussion about policies happens during incubation.
> > >
> > > On 2 December 2015 at 10:08, Henry Saputra <he...@gmail.com>
> > > wrote:
> > >
> > > > Nice +1 =)
> > > >
> > > > On Wed, Dec 2, 2015 at 2:01 AM, Tom White <to...@apache.org>
> wrote:
> > > > > The vote to accept Impala into the incubator has passed
> > > > > (http://s.apache.org/u6r), however there are still some concerns
> > about
> > > > > CTR/RTC. My main takeaways from the CTR/RTC thread are that it's
> not
> > a
> > > > > binary choice, and that it's entirely reasonable that different
> > > > > communities have different commit policies at the ASF.
> > > > >
> > > > > I think Julian Hyde's suggestion that the Impala podling start with
> > no
> > > > > explicit commit policy is a good one. Incubation should be used as
> a
> > > > > time to work out what works best for a project. The initial Impala
> > > > > community should discuss the commit policy as they go through the
> > > > > process of setting up ASF infra and start growing the podling. In
> > > > > particular this will include how Gerrit can be used as a tool to
> > > > > facilitate reviews, and how that fits with ASF culture, which is
> > > > > something that other projects are looking at too.
> > > > >
> > > > > Cheers,
> > > > > Tom
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > >
> > >
> > >
> > > --
> > > Henry Robinson
> > > Software Engineer
> > > Cloudera
> > > 415-994-6679
> > >
> >
>
>
>
> --
> Henry Robinson
> Software Engineer
> Cloudera
> 415-994-6679
>
--
Julien
Re: Impala commit policy
Posted by Greg Stein <gs...@gmail.com>.
Yeup!
On Wed, Dec 2, 2015 at 1:19 PM, Henry Robinson <he...@cloudera.com> wrote:
> What might happen, however, is that the discussion is revisited with a
> particular focus on the concerns that you've raised. So although it might
> be unlikely that the community performs a volte-face and elects for CTR, we
> might say "what can we do to limit the risk that RTC inhibits community
> growth, without abandoning RTC completely?". At the very least, the
> community becomes aware of the concerns, and more sensitive to their
> potential impact.
>
>
> On 2 December 2015 at 11:09, Greg Stein <gs...@gmail.com> wrote:
>
> > Yeah, this is what I meant earlier. Leaving out a commit policy changes
> > nothing. The same people who put together the proposal will be the same
> set
> > as those discussing it as a podling, and they will reach the same
> > conclusion.
> >
> > If the PPMC doubles in size, with fresh faces, then a real discussion can
> > happen. Tho I doubt that will be possible -- I'm unaware of any podling
> > pulling off such growth.
> > On Dec 2, 2015 12:45 PM, "Henry Robinson" <he...@cloudera.com> wrote:
> >
> > > I agree that this is something the Impala community will want to
> discuss
> > > fairly early on in incubation - along with a lot of other project
> > > procedural stuff as we adjust or rethink our workflows to be Apache-Way
> > > compatible.
> > >
> > > Until we have that discussion, I'd expect Impala will continue along
> RTC
> > > lines simply because this is what the existing community is used to
> (and
> > I
> > > believe, prefers), and the workflows are well established and work
> > > smoothly.
> > >
> > > That is, I personally am ok with no explicit commit protocol, but in
> > > practice it is likely to appear as if there is no change, until the
> > > community discussion about policies happens during incubation.
> > >
> > > On 2 December 2015 at 10:08, Henry Saputra <he...@gmail.com>
> > > wrote:
> > >
> > > > Nice +1 =)
> > > >
> > > > On Wed, Dec 2, 2015 at 2:01 AM, Tom White <to...@apache.org>
> wrote:
> > > > > The vote to accept Impala into the incubator has passed
> > > > > (http://s.apache.org/u6r), however there are still some concerns
> > about
> > > > > CTR/RTC. My main takeaways from the CTR/RTC thread are that it's
> not
> > a
> > > > > binary choice, and that it's entirely reasonable that different
> > > > > communities have different commit policies at the ASF.
> > > > >
> > > > > I think Julian Hyde's suggestion that the Impala podling start with
> > no
> > > > > explicit commit policy is a good one. Incubation should be used as
> a
> > > > > time to work out what works best for a project. The initial Impala
> > > > > community should discuss the commit policy as they go through the
> > > > > process of setting up ASF infra and start growing the podling. In
> > > > > particular this will include how Gerrit can be used as a tool to
> > > > > facilitate reviews, and how that fits with ASF culture, which is
> > > > > something that other projects are looking at too.
> > > > >
> > > > > Cheers,
> > > > > Tom
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > >
> > >
> > >
> > > --
> > > Henry Robinson
> > > Software Engineer
> > > Cloudera
> > > 415-994-6679
> > >
> >
>
>
>
> --
> Henry Robinson
> Software Engineer
> Cloudera
> 415-994-6679
>
Re: Impala commit policy
Posted by Henry Robinson <he...@cloudera.com>.
What might happen, however, is that the discussion is revisited with a
particular focus on the concerns that you've raised. So although it might
be unlikely that the community performs a volte-face and elects for CTR, we
might say "what can we do to limit the risk that RTC inhibits community
growth, without abandoning RTC completely?". At the very least, the
community becomes aware of the concerns, and more sensitive to their
potential impact.
On 2 December 2015 at 11:09, Greg Stein <gs...@gmail.com> wrote:
> Yeah, this is what I meant earlier. Leaving out a commit policy changes
> nothing. The same people who put together the proposal will be the same set
> as those discussing it as a podling, and they will reach the same
> conclusion.
>
> If the PPMC doubles in size, with fresh faces, then a real discussion can
> happen. Tho I doubt that will be possible -- I'm unaware of any podling
> pulling off such growth.
> On Dec 2, 2015 12:45 PM, "Henry Robinson" <he...@cloudera.com> wrote:
>
> > I agree that this is something the Impala community will want to discuss
> > fairly early on in incubation - along with a lot of other project
> > procedural stuff as we adjust or rethink our workflows to be Apache-Way
> > compatible.
> >
> > Until we have that discussion, I'd expect Impala will continue along RTC
> > lines simply because this is what the existing community is used to (and
> I
> > believe, prefers), and the workflows are well established and work
> > smoothly.
> >
> > That is, I personally am ok with no explicit commit protocol, but in
> > practice it is likely to appear as if there is no change, until the
> > community discussion about policies happens during incubation.
> >
> > On 2 December 2015 at 10:08, Henry Saputra <he...@gmail.com>
> > wrote:
> >
> > > Nice +1 =)
> > >
> > > On Wed, Dec 2, 2015 at 2:01 AM, Tom White <to...@apache.org> wrote:
> > > > The vote to accept Impala into the incubator has passed
> > > > (http://s.apache.org/u6r), however there are still some concerns
> about
> > > > CTR/RTC. My main takeaways from the CTR/RTC thread are that it's not
> a
> > > > binary choice, and that it's entirely reasonable that different
> > > > communities have different commit policies at the ASF.
> > > >
> > > > I think Julian Hyde's suggestion that the Impala podling start with
> no
> > > > explicit commit policy is a good one. Incubation should be used as a
> > > > time to work out what works best for a project. The initial Impala
> > > > community should discuss the commit policy as they go through the
> > > > process of setting up ASF infra and start growing the podling. In
> > > > particular this will include how Gerrit can be used as a tool to
> > > > facilitate reviews, and how that fits with ASF culture, which is
> > > > something that other projects are looking at too.
> > > >
> > > > Cheers,
> > > > Tom
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
> >
> >
> > --
> > Henry Robinson
> > Software Engineer
> > Cloudera
> > 415-994-6679
> >
>
--
Henry Robinson
Software Engineer
Cloudera
415-994-6679
Re: Impala commit policy
Posted by Greg Stein <gs...@gmail.com>.
Yeah, this is what I meant earlier. Leaving out a commit policy changes
nothing. The same people who put together the proposal will be the same set
as those discussing it as a podling, and they will reach the same
conclusion.
If the PPMC doubles in size, with fresh faces, then a real discussion can
happen. Tho I doubt that will be possible -- I'm unaware of any podling
pulling off such growth.
On Dec 2, 2015 12:45 PM, "Henry Robinson" <he...@cloudera.com> wrote:
> I agree that this is something the Impala community will want to discuss
> fairly early on in incubation - along with a lot of other project
> procedural stuff as we adjust or rethink our workflows to be Apache-Way
> compatible.
>
> Until we have that discussion, I'd expect Impala will continue along RTC
> lines simply because this is what the existing community is used to (and I
> believe, prefers), and the workflows are well established and work
> smoothly.
>
> That is, I personally am ok with no explicit commit protocol, but in
> practice it is likely to appear as if there is no change, until the
> community discussion about policies happens during incubation.
>
> On 2 December 2015 at 10:08, Henry Saputra <he...@gmail.com>
> wrote:
>
> > Nice +1 =)
> >
> > On Wed, Dec 2, 2015 at 2:01 AM, Tom White <to...@apache.org> wrote:
> > > The vote to accept Impala into the incubator has passed
> > > (http://s.apache.org/u6r), however there are still some concerns about
> > > CTR/RTC. My main takeaways from the CTR/RTC thread are that it's not a
> > > binary choice, and that it's entirely reasonable that different
> > > communities have different commit policies at the ASF.
> > >
> > > I think Julian Hyde's suggestion that the Impala podling start with no
> > > explicit commit policy is a good one. Incubation should be used as a
> > > time to work out what works best for a project. The initial Impala
> > > community should discuss the commit policy as they go through the
> > > process of setting up ASF infra and start growing the podling. In
> > > particular this will include how Gerrit can be used as a tool to
> > > facilitate reviews, and how that fits with ASF culture, which is
> > > something that other projects are looking at too.
> > >
> > > Cheers,
> > > Tom
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
>
> --
> Henry Robinson
> Software Engineer
> Cloudera
> 415-994-6679
>
Re: Impala commit policy
Posted by Henry Robinson <he...@cloudera.com>.
I agree that this is something the Impala community will want to discuss
fairly early on in incubation - along with a lot of other project
procedural stuff as we adjust or rethink our workflows to be Apache-Way
compatible.
Until we have that discussion, I'd expect Impala will continue along RTC
lines simply because this is what the existing community is used to (and I
believe, prefers), and the workflows are well established and work
smoothly.
That is, I personally am ok with no explicit commit protocol, but in
practice it is likely to appear as if there is no change, until the
community discussion about policies happens during incubation.
On 2 December 2015 at 10:08, Henry Saputra <he...@gmail.com> wrote:
> Nice +1 =)
>
> On Wed, Dec 2, 2015 at 2:01 AM, Tom White <to...@apache.org> wrote:
> > The vote to accept Impala into the incubator has passed
> > (http://s.apache.org/u6r), however there are still some concerns about
> > CTR/RTC. My main takeaways from the CTR/RTC thread are that it's not a
> > binary choice, and that it's entirely reasonable that different
> > communities have different commit policies at the ASF.
> >
> > I think Julian Hyde's suggestion that the Impala podling start with no
> > explicit commit policy is a good one. Incubation should be used as a
> > time to work out what works best for a project. The initial Impala
> > community should discuss the commit policy as they go through the
> > process of setting up ASF infra and start growing the podling. In
> > particular this will include how Gerrit can be used as a tool to
> > facilitate reviews, and how that fits with ASF culture, which is
> > something that other projects are looking at too.
> >
> > Cheers,
> > Tom
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
Henry Robinson
Software Engineer
Cloudera
415-994-6679
Re: Impala commit policy
Posted by Henry Saputra <he...@gmail.com>.
Nice +1 =)
On Wed, Dec 2, 2015 at 2:01 AM, Tom White <to...@apache.org> wrote:
> The vote to accept Impala into the incubator has passed
> (http://s.apache.org/u6r), however there are still some concerns about
> CTR/RTC. My main takeaways from the CTR/RTC thread are that it's not a
> binary choice, and that it's entirely reasonable that different
> communities have different commit policies at the ASF.
>
> I think Julian Hyde's suggestion that the Impala podling start with no
> explicit commit policy is a good one. Incubation should be used as a
> time to work out what works best for a project. The initial Impala
> community should discuss the commit policy as they go through the
> process of setting up ASF infra and start growing the podling. In
> particular this will include how Gerrit can be used as a tool to
> facilitate reviews, and how that fits with ASF culture, which is
> something that other projects are looking at too.
>
> Cheers,
> Tom
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Impala commit policy
Posted by Josh Elser <el...@apache.org>.
Tom White wrote:
> The vote to accept Impala into the incubator has passed
> (http://s.apache.org/u6r), however there are still some concerns about
> CTR/RTC. My main takeaways from the CTR/RTC thread are that it's not a
> binary choice, and that it's entirely reasonable that different
> communities have different commit policies at the ASF.
>
> I think Julian Hyde's suggestion that the Impala podling start with no
> explicit commit policy is a good one. Incubation should be used as a
> time to work out what works best for a project. The initial Impala
> community should discuss the commit policy as they go through the
> process of setting up ASF infra and start growing the podling. In
> particular this will include how Gerrit can be used as a tool to
> facilitate reviews, and how that fits with ASF culture, which is
> something that other projects are looking at too.
>
> Cheers,
> Tom
>
(Ugh, and then I catch up on the rest of my mail - sorry for the spam on
the RESULT thread)
+1 I'm not entirely sold on saying they have no explicitly policy up
front (I'd be worried about that causing confusion -- the project will
operate how they're comfortable operating), but I'd definitely want to
see _real_ discussion is had after the podling gets on its feet and
grows beyond the initial membership.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Impala commit policy
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Thu, Dec 3, 2015 at 7:41 PM, Ralph Goers <ra...@dslextreme.com> wrote:
> In my opinion the correct approach is to identify the parts of the code that a)
> seem to be most susceptible to bugs, b) are hard to understand well, or c) where
> simple changes can have huge impacts on performance and then use RTC for
> those areas....
I tend to agree but I'd say *maybe, if really needed* use RTC...
And re-evaluate that periodically.
RTC can have a big cost community-wise, so it's important IMO to use
it as sparingly as possible.
-Bertrand
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Impala commit policy
Posted by Ralph Goers <ra...@dslextreme.com>.
Virtually any project you look at is going to have portions that are fairly complex and portions that are pretty straightforward. In my opinion the correct approach is to identify the parts of the code that a) seem to be most susceptible to bugs, b) are hard to understand well, or c) where simple changes can have huge impacts on performance and then use RTC for those areas. In addition, you might want to require RTC for significant new feature additions. Although I expect every project to have code that falls into these categories the ratio of that code will vary from project to project. As an example, I can honestly say that with Flume the File Channel should always be RTC. But almost everything else could be done more effectively with CTR.
Ralph
> On Dec 3, 2015, at 11:20 AM, Henry Robinson <he...@apache.org> wrote:
>
> I'm happy to field technical questions about Impala. You seem to be
> conflating 'complexity' with 'severity of potential bugs' - I see the two
> as separate.
>
> Under the 'severity' heading, Impala both writes and reads data from a
> variety of data stores. So if there's a bug in Impala's write path, data
> can be lost. But because Impala also returns results to client
> applications, there's a significant risk of business impact if the *wrong*
> results are returned. I know, because I have dealt with situations where
> this has happened, and no-one is very happy about it. Our customers
> typically run business-critical analytic workloads through Impala; if it
> stops working correctly that's usually a big problem.
>
> As far as 'complexity' goes, I make no comparative claims about Impala's
> complexity vs any other project. But to give some indication of the moving
> parts inside Impala: there's a component which compiles highly optimised
> versions of each query operator at run time, there's a query planner which
> parses and plans a large portion of the SQL standard, there is the added
> complexity of being a 'massively' (with many deployments in the high 100s
> of nodes) distributed system with the added coordination and consistency
> guarantees that brings to it, and there is also the added complexity of
> running highly concurrent workloads in a single process, with all the
> concurrency headaches etc. that can bring. That's not to mention
> implementations of 'standard' SQL operators like joins, sorts and so on
> that are still the subject of active research in academia and industry.
>
> All this is in the context of Impala's main differentiator, which is that
> it is amongst the very fastest SQL engine for data stored in HDFS and
> friends. That means that small changes can have large unexpected
> consequences, since efficiency is a subtle and capricious thing. It has
> always, therefore, helped us to have more than one set of eyes on every
> change in the past, to ensure that the probability of the introduction of
> subtle performance and functional regressions is reduced. Automated testing
> plays a huge role here as well, but for us it's been most effective in
> concert with code review.
>
> (There are other reasons I vastly prefer RTC as well, but I'm addressing
> your specific points here so as not to kick off another RTCvsCTR thread :)).
>
>
>
>>
>> In this case, the RTC seems to stem from the choice of Gerrit, rather than
>> some innate complexity.
>>
>>
>
> Gerrit does not mandate RTC, since you can just push to refs/heads/<branch>
> and bypass the review creation step.
>
> Historically, the Impala team at Cloudera has used at least three different
> review tools (including Review Board, which is used elsewhere at the ASF).
> The choice of review tool stems completely from pragmatism - we really did
> not like Review Board, and briefly used Rietveld before moving to Gerrit
> which we have preferred. At every step, we used RTC.
>
> Henry
>
>
>
>> I *do* note that possibly committers could choose to commit directly, or
>> choose to use Gerrit when they are unsure. Will the (P)PMC allow those
>> direct commits? Or mandate Gerrit for every commit?
>>
>> Cheers,
>> -g
Re: Impala commit policy
Posted by Henry Robinson <he...@apache.org>.
On 2 December 2015 at 23:04, Greg Stein <gs...@gmail.com> wrote:
> On Wed, Dec 2, 2015 at 8:50 PM, Julian Hyde <jh...@apache.org> wrote:
>
> > Thanks, Roman. For the record, I don’t plan to contribute to Impala or
> > Kudu, and I don’t like strict commit policies such as RTC. But I wanted
> to
> > stand up for “states' rights”, the right of podlings and projects to
> > determine their own processes and cultures.
> >
>
> LOL ... being a Texan, I can certainly get on board with the notion of
> states' rights :-P
>
> But I caution: as I said else-thread, we use the Incubation process because
> we believe the podling needs to *learn* how we like communities to operate.
> Peer respect, inclusivity, open dialog, consensus, etc. By definition, the
> podling is unable to make these decisions within the guides and desires of
> the Foundation. If we trusted them to do so, then we'd just make them a TLP
> and skip incubation.
>
> Josh puts it well:
>
> On Thu, Dec 3, 2015 at 12:26 AM, Josh Elser <el...@apache.org> wrote:
> >...
>
> > +1 I'm not entirely sold on saying they have no explicitly policy up
> front
> > (I'd be worried about that causing confusion -- the project will operate
> > how they're comfortable operating), but I'd definitely want to see _real_
> > discussion is had after the podling gets on its feet and grows beyond the
> > initial membership.
> >
>
> I'd like to see podlings have enough diversity and independence from the
> initial PPMC, to have such a discussion. My fear is that RTC holds back
> growing the diversity of opinion, and that status quo will not allow for
> moving away from Gerrit.
>
> ...
>
> I will also note that one of the primary reasons explained for RTC is "the
> code is too complex to allow for unreviewed changes to be applied". Has
> that basis been justified for Impala? Are we talking data loss? Nope. It's
> a layer over the data substrate. Synchronization across a cluster? Nah.
> Where is the complexity?
>
I'm happy to field technical questions about Impala. You seem to be
conflating 'complexity' with 'severity of potential bugs' - I see the two
as separate.
Under the 'severity' heading, Impala both writes and reads data from a
variety of data stores. So if there's a bug in Impala's write path, data
can be lost. But because Impala also returns results to client
applications, there's a significant risk of business impact if the *wrong*
results are returned. I know, because I have dealt with situations where
this has happened, and no-one is very happy about it. Our customers
typically run business-critical analytic workloads through Impala; if it
stops working correctly that's usually a big problem.
As far as 'complexity' goes, I make no comparative claims about Impala's
complexity vs any other project. But to give some indication of the moving
parts inside Impala: there's a component which compiles highly optimised
versions of each query operator at run time, there's a query planner which
parses and plans a large portion of the SQL standard, there is the added
complexity of being a 'massively' (with many deployments in the high 100s
of nodes) distributed system with the added coordination and consistency
guarantees that brings to it, and there is also the added complexity of
running highly concurrent workloads in a single process, with all the
concurrency headaches etc. that can bring. That's not to mention
implementations of 'standard' SQL operators like joins, sorts and so on
that are still the subject of active research in academia and industry.
All this is in the context of Impala's main differentiator, which is that
it is amongst the very fastest SQL engine for data stored in HDFS and
friends. That means that small changes can have large unexpected
consequences, since efficiency is a subtle and capricious thing. It has
always, therefore, helped us to have more than one set of eyes on every
change in the past, to ensure that the probability of the introduction of
subtle performance and functional regressions is reduced. Automated testing
plays a huge role here as well, but for us it's been most effective in
concert with code review.
(There are other reasons I vastly prefer RTC as well, but I'm addressing
your specific points here so as not to kick off another RTCvsCTR thread :)).
>
> In this case, the RTC seems to stem from the choice of Gerrit, rather than
> some innate complexity.
>
>
Gerrit does not mandate RTC, since you can just push to refs/heads/<branch>
and bypass the review creation step.
Historically, the Impala team at Cloudera has used at least three different
review tools (including Review Board, which is used elsewhere at the ASF).
The choice of review tool stems completely from pragmatism - we really did
not like Review Board, and briefly used Rietveld before moving to Gerrit
which we have preferred. At every step, we used RTC.
Henry
> I *do* note that possibly committers could choose to commit directly, or
> choose to use Gerrit when they are unsure. Will the (P)PMC allow those
> direct commits? Or mandate Gerrit for every commit?
>
> Cheers,
> -g
>
Re: Impala commit policy
Posted by Greg Stein <gs...@gmail.com>.
On Wed, Dec 2, 2015 at 8:50 PM, Julian Hyde <jh...@apache.org> wrote:
> Thanks, Roman. For the record, I don’t plan to contribute to Impala or
> Kudu, and I don’t like strict commit policies such as RTC. But I wanted to
> stand up for “states' rights”, the right of podlings and projects to
> determine their own processes and cultures.
>
LOL ... being a Texan, I can certainly get on board with the notion of
states' rights :-P
But I caution: as I said else-thread, we use the Incubation process because
we believe the podling needs to *learn* how we like communities to operate.
Peer respect, inclusivity, open dialog, consensus, etc. By definition, the
podling is unable to make these decisions within the guides and desires of
the Foundation. If we trusted them to do so, then we'd just make them a TLP
and skip incubation.
Josh puts it well:
On Thu, Dec 3, 2015 at 12:26 AM, Josh Elser <el...@apache.org> wrote:
>...
> +1 I'm not entirely sold on saying they have no explicitly policy up front
> (I'd be worried about that causing confusion -- the project will operate
> how they're comfortable operating), but I'd definitely want to see _real_
> discussion is had after the podling gets on its feet and grows beyond the
> initial membership.
>
I'd like to see podlings have enough diversity and independence from the
initial PPMC, to have such a discussion. My fear is that RTC holds back
growing the diversity of opinion, and that status quo will not allow for
moving away from Gerrit.
...
I will also note that one of the primary reasons explained for RTC is "the
code is too complex to allow for unreviewed changes to be applied". Has
that basis been justified for Impala? Are we talking data loss? Nope. It's
a layer over the data substrate. Synchronization across a cluster? Nah.
Where is the complexity?
In this case, the RTC seems to stem from the choice of Gerrit, rather than
some innate complexity.
I *do* note that possibly committers could choose to commit directly, or
choose to use Gerrit when they are unsure. Will the (P)PMC allow those
direct commits? Or mandate Gerrit for every commit?
Cheers,
-g
Re: Impala commit policy
Posted by Julian Hyde <jh...@apache.org>.
Thanks, Roman. For the record, I don’t plan to contribute to Impala or Kudu, and I don’t like strict commit policies such as RTC. But I wanted to stand up for “states' rights”, the right of podlings and projects to determine their own processes and cultures.
Julian
> On Dec 2, 2015, at 6:42 PM, Roman Shaposhnik <ro...@shaposhnik.org> wrote:
>
> On Wed, Dec 2, 2015 at 4:24 PM, Julian Hyde <jh...@apache.org> wrote:
>> “No explicit commit policy” means that only committers can commit.
>> It is each committer’s discretion whether they ask for others to review
>> the change before they commit it, whether they check in code that doesn’t
>> build, whether they run the test suite before committing.
>>
>> This policy is the bare constitutional minimum. We would all hope and expect
>> that the community would quickly agree on some policies, but that is up the community.
>> They are not stupid, they want to produce high-quality software, and they want to grow
>> their community, and they will figure out a policy that achieves these goals.
>
> If that's the expectation that is internalized by the podling community than
> I guess my concerns are somewhat taken care of to a point where I'd
> be comfortable giving the proposal a +0.
>
> Let me explain why it is a +0 instead of +1. This will also be an opportunity
> for me to clarify my seemingly inconsistent position with Kudu proposal (which
> I deliberately +1ed).
>
> It all comes back to trust and inclusiveness. I thought that Greg was super
> convincing at articulating that CTR policy is an indication, a proxy if you will
> for those qualities. If CTR is there -- I know that the community is
> trusting and
> inclusive. My -1 for Impala was based on their strong resistance to CTR as
> a proxy measure of how ready they are to accept the "Apache Way". Them
> fighting the idea of CTR gave me a strong indication that they are resisting
> to being trusting and inclusive.
>
> Now, of course, as any astute student of logic will know, necessity
> and sufficiency
> is not the same thing. While a presence of CTR policy is a strong indication of
> the community getting the "Apache Way" the lack of it is not necessarily an
> indication of them not getting it. Impala community, however has one extra
> strike against it that wasn't allowing me to give it the same benefit
> of the doubt
> that Kudu community got (hence +1 for Kudu despite their instance on RTC).
>
> Impala and its existing community are *not* new to Open Source. They've been
> out on GitHub since late 2012 and they have operated under a very explicit
> BDFL model. Based on their past attitude towards external contributions
> (personalized via statement of the project lead) I have strong
> reasons to believe
> that they are going to have really tough time adjusting to the Apache governance
> model AND when I saw that strong resistance to CTR that was it for me.
>
> That said, this thread and the expectations articulated by Tom and others
> make me more comfortable. However, the lack of *actionable* suggestion
> for how they are going to integrate with Apache governance model
> makes it +0 and not +1.
>
> Thanks,
> Roman.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Impala commit policy
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Wed, Dec 2, 2015 at 4:24 PM, Julian Hyde <jh...@apache.org> wrote:
> “No explicit commit policy” means that only committers can commit.
> It is each committer’s discretion whether they ask for others to review
> the change before they commit it, whether they check in code that doesn’t
> build, whether they run the test suite before committing.
>
> This policy is the bare constitutional minimum. We would all hope and expect
> that the community would quickly agree on some policies, but that is up the community.
> They are not stupid, they want to produce high-quality software, and they want to grow
> their community, and they will figure out a policy that achieves these goals.
If that's the expectation that is internalized by the podling community than
I guess my concerns are somewhat taken care of to a point where I'd
be comfortable giving the proposal a +0.
Let me explain why it is a +0 instead of +1. This will also be an opportunity
for me to clarify my seemingly inconsistent position with Kudu proposal (which
I deliberately +1ed).
It all comes back to trust and inclusiveness. I thought that Greg was super
convincing at articulating that CTR policy is an indication, a proxy if you will
for those qualities. If CTR is there -- I know that the community is
trusting and
inclusive. My -1 for Impala was based on their strong resistance to CTR as
a proxy measure of how ready they are to accept the "Apache Way". Them
fighting the idea of CTR gave me a strong indication that they are resisting
to being trusting and inclusive.
Now, of course, as any astute student of logic will know, necessity
and sufficiency
is not the same thing. While a presence of CTR policy is a strong indication of
the community getting the "Apache Way" the lack of it is not necessarily an
indication of them not getting it. Impala community, however has one extra
strike against it that wasn't allowing me to give it the same benefit
of the doubt
that Kudu community got (hence +1 for Kudu despite their instance on RTC).
Impala and its existing community are *not* new to Open Source. They've been
out on GitHub since late 2012 and they have operated under a very explicit
BDFL model. Based on their past attitude towards external contributions
(personalized via statement of the project lead) I have strong
reasons to believe
that they are going to have really tough time adjusting to the Apache governance
model AND when I saw that strong resistance to CTR that was it for me.
That said, this thread and the expectations articulated by Tom and others
make me more comfortable. However, the lack of *actionable* suggestion
for how they are going to integrate with Apache governance model
makes it +0 and not +1.
Thanks,
Roman.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Impala commit policy
Posted by Julian Hyde <jh...@apache.org>.
“No explicit commit policy” means that only committers can commit. It is each committer’s discretion whether they ask for others to review the change before they commit it, whether they check in code that doesn’t build, whether they run the test suite before committing.
This policy is the bare constitutional minimum. We would all hope and expect that the community would quickly agree on some policies, but that is up the community. They are not stupid, they want to produce high-quality software, and they want to grow their community, and they will figure out a policy that achieves these goals.
Julian
> On Dec 2, 2015, at 12:40 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> I am not sure what "start with no explicit commit policy" even means. Will
> there be no commits, until the discussion on the subject happens?
>
> How code changes will be going into the source base?
> Cos
>
> On Wed, Dec 02, 2015 at 10:01AM, Tom White wrote:
>> The vote to accept Impala into the incubator has passed
>> (http://s.apache.org/u6r), however there are still some concerns about
>> CTR/RTC. My main takeaways from the CTR/RTC thread are that it's not a
>> binary choice, and that it's entirely reasonable that different
>> communities have different commit policies at the ASF.
>>
>> I think Julian Hyde's suggestion that the Impala podling start with no
>> explicit commit policy is a good one. Incubation should be used as a
>> time to work out what works best for a project. The initial Impala
>> community should discuss the commit policy as they go through the
>> process of setting up ASF infra and start growing the podling. In
>> particular this will include how Gerrit can be used as a tool to
>> facilitate reviews, and how that fits with ASF culture, which is
>> something that other projects are looking at too.
>>
>> Cheers,
>> Tom
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Impala commit policy
Posted by Konstantin Boudnik <co...@apache.org>.
I am not sure what "start with no explicit commit policy" even means. Will
there be no commits, until the discussion on the subject happens?
How code changes will be going into the source base?
Cos
On Wed, Dec 02, 2015 at 10:01AM, Tom White wrote:
> The vote to accept Impala into the incubator has passed
> (http://s.apache.org/u6r), however there are still some concerns about
> CTR/RTC. My main takeaways from the CTR/RTC thread are that it's not a
> binary choice, and that it's entirely reasonable that different
> communities have different commit policies at the ASF.
>
> I think Julian Hyde's suggestion that the Impala podling start with no
> explicit commit policy is a good one. Incubation should be used as a
> time to work out what works best for a project. The initial Impala
> community should discuss the commit policy as they go through the
> process of setting up ASF infra and start growing the podling. In
> particular this will include how Gerrit can be used as a tool to
> facilitate reviews, and how that fits with ASF culture, which is
> something that other projects are looking at too.
>
> Cheers,
> Tom
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>