Off list major development

[Slim Bouguerra <bs...@apache.org>]

Hello everyone and hope you all have very good holidays.

First, this email is not directed on the author or the PR
https://github.com/apache/incubator-druid/pull/6794  it self, but i see
this PR as a perfect example.

One of the foundation of Apache Way or what i would simply call open source
community driven development is that "Technical decisions are discussed,
decided, and archived publicly.
developpement"
Which means that big technical  changes such as the one brought by #/6794
should have started as a proposal and round of discussions about the major
changes designs not as 11K line of code.
I believe such openness will promote a lot of good benefits such as:

- ensures community health and growth.
- ensures everyone can participate not only the authors and his co-workers.
- ensures that the project is driven by the community and not a given
company or an individual.
- ensures that there is consensus (not saying 100% agreement;) however it
means that all individuals will accept the current progress on the project
until some better proposal is put forth.

Personally such BIG offline PR makes me feel excluded and doesn't give me a
sense that i belong to  a community at all.

To prevent such off list development i think as a Druid Community we need
to stick to the apache way “If it didn’t happen on the mailing list, it
didn’t happen.”

I would appreciate if some of the Apache mentor help with this.
Thanks

Re: Off list major development

[Jihoon Son <ji...@apache.org>]

Sorry, this is the right link.
https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E

Jihoon

On Wed, Jan 2, 2019 at 11:30 AM Jihoon Son <ji...@apache.org> wrote:

> Slim, thanks for raising this.
>
> I also raised a similar proposal. Please check
> https://lists.apache.org/list.html?dev@druid.apache.org:2018-10:jihoon%20son
> .
>
> Jihoon
>
> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org> wrote:
>
>> Slim,
>>
>> I agree with your points that offline development is bad for community.
>> But I don’t think you need much mentor help. You have raised valid issues
>> and the Druid community needs to decide what its development practices
>> should be.
>>
>> Julian
>>
>>
>> > On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <bs...@apache.org> wrote:
>> >
>> > Hello everyone and hope you all have very good holidays.
>> >
>> > First, this email is not directed on the author or the PR
>> > https://github.com/apache/incubator-druid/pull/6794  it self, but i see
>> > this PR as a perfect example.
>> >
>> > One of the foundation of Apache Way or what i would simply call open
>> source
>> > community driven development is that "Technical decisions are discussed,
>> > decided, and archived publicly.
>> > developpement"
>> > Which means that big technical  changes such as the one brought by
>> #/6794
>> > should have started as a proposal and round of discussions about the
>> major
>> > changes designs not as 11K line of code.
>> > I believe such openness will promote a lot of good benefits such as:
>> >
>> > - ensures community health and growth.
>> > - ensures everyone can participate not only the authors and his
>> co-workers.
>> > - ensures that the project is driven by the community and not a given
>> > company or an individual.
>> > - ensures that there is consensus (not saying 100% agreement;) however
>> it
>> > means that all individuals will accept the current progress on the
>> project
>> > until some better proposal is put forth.
>> >
>> > Personally such BIG offline PR makes me feel excluded and doesn't give
>> me a
>> > sense that i belong to  a community at all.
>> >
>> > To prevent such off list development i think as a Druid Community we
>> need
>> > to stick to the apache way “If it didn’t happen on the mailing list, it
>> > didn’t happen.”
>> >
>> > I would appreciate if some of the Apache mentor help with this.
>> > Thanks
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>> For additional commands, e-mail: dev-help@druid.apache.org
>>
>>

Re: Off list major development

[Jihoon Son <ji...@apache.org>]

Slim, thanks for raising this.

I also raised a similar proposal. Please check
https://lists.apache.org/list.html?dev@druid.apache.org:2018-10:jihoon%20son
.

Jihoon

On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org> wrote:

> Slim,
>
> I agree with your points that offline development is bad for community.
> But I don’t think you need much mentor help. You have raised valid issues
> and the Druid community needs to decide what its development practices
> should be.
>
> Julian
>
>
> > On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <bs...@apache.org> wrote:
> >
> > Hello everyone and hope you all have very good holidays.
> >
> > First, this email is not directed on the author or the PR
> > https://github.com/apache/incubator-druid/pull/6794  it self, but i see
> > this PR as a perfect example.
> >
> > One of the foundation of Apache Way or what i would simply call open
> source
> > community driven development is that "Technical decisions are discussed,
> > decided, and archived publicly.
> > developpement"
> > Which means that big technical  changes such as the one brought by #/6794
> > should have started as a proposal and round of discussions about the
> major
> > changes designs not as 11K line of code.
> > I believe such openness will promote a lot of good benefits such as:
> >
> > - ensures community health and growth.
> > - ensures everyone can participate not only the authors and his
> co-workers.
> > - ensures that the project is driven by the community and not a given
> > company or an individual.
> > - ensures that there is consensus (not saying 100% agreement;) however it
> > means that all individuals will accept the current progress on the
> project
> > until some better proposal is put forth.
> >
> > Personally such BIG offline PR makes me feel excluded and doesn't give
> me a
> > sense that i belong to  a community at all.
> >
> > To prevent such off list development i think as a Druid Community we need
> > to stick to the apache way “If it didn’t happen on the mailing list, it
> > didn’t happen.”
> >
> > I would appreciate if some of the Apache mentor help with this.
> > Thanks
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> For additional commands, e-mail: dev-help@druid.apache.org
>
>

Re: Off list major development

[Julian Hyde <jh...@apache.org>]

Apache has a procedure for major pieces of code that are developed off-list. It is called “IP clearance”[1]. It’s fairly onerous, and I believe it would apply even if the people who developed the code were committers to the project.

So, another good reason to develop code on-list.

Julian

[1] http://incubator.apache.org/ip-clearance/

> On Jan 3, 2019, at 5:35 PM, Jonathan Wei <jo...@apache.org> wrote:
> 
> Thanks for raising these concerns!
> 
> My initial thoughts:
> - I agree that separation of design review and code-level review for major
> changes would be more efficient
> 
> - I agree that a clear, more formalized process for handling major changes
> would be helpful for contributors:
>  - Define what is considered a major change
>  - Define a standard proposal structure, KIP-style proposal format sounds
> good to me
> 
> - I think it's too rigid to have a policy of "no code at all with the
> initial proposal"
>  - Code samples can be useful references for understanding aspects of a
> design
>  - In some cases it's necessary to run experiments to fully understand a
> problem and determine an appropriate design, or to determine whether
> something is even worth doing before committing to the work of fleshing out
> a proposal, prototype code is a natural outcome of that and I'm not against
> someone providing such code for reference
>  - I tend to view design/code as things that are often developed
> simultaneously in an intertwined way
> 
>> Let's not be naive this is very rare that a contributor will accept that
> his work is to be thrown, usually devs takes coding as personal creation
> and they get attached to it.
> 
> If we have a clear review process that emphasizes the need for early
> consensus building, with separate design and code review, then I feel we've
> done enough and don't need a hard rule against having some code linked with
> the initial proposal. If a potential contributor then still wants to go
> ahead and write a lot of code that may be rejected or change significantly,
> the risks were made clear.
> 
>> Once code is written hard to think abstract.
> 
> I can see the validity of the concern, but I personally don't see it as a
> huge risk. My impression from the Druid PR reviews I've seen is that our
> reviewers are able to keep abstract design vs. implementation details
> separate and consider alternate designs when reviewing.
> 
> To summarize I think it's probably enough to have a policy along the lines
> of:
> - Create more formalized guidelines for proposals and what changes require
> proposals
> - Separate design and code review for major changes, with design review
> first, code-level review after reaching consensus on the design.
> - Code before the design review is completed is just for reference, not
> regarded as a candidate for review/merging.
> 
> - Jon
> 
> 
> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <sl...@gmail.com>
> wrote:
> 
>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <cl...@imply.io> wrote:
>> 
>>> I am definitely biased in this matter as an owner of another large PR
>> that
>>> wasn't preceded by a direct proposal or dev list discussion, and in
>> general
>>> I agree that proposal first is usually better, but I think in some rarer
>>> cases approaching a problem code first *is* the most appropriate way to
>>> have a discussion.
>> 
>> 
>> I am wondering here what is the case where code first is better?
>> In general when you are writing code you have an idea about what you want
>> to change, why you want to change and why you want to change it.
>> I do not see what is wrong with sharing this primitive ideas and thoughts
>> as an abstract proposal (at least to avoid overlapping)
>> 
>> I see nothing wrong with it so long as the author
>>> accepts that the PR is treated as a combined proposal and proof of
>> concept,
>>> and fair game to be radically changed via discussion or even rejected,
>>> which sounds like Gian's attitude on the matter and is mine as well with
>> my
>>> compression stuff.
>> 
>> 
>> Let's not be naive this is very rare that a contributor will accept that
>> his work is to be thrown, usually devs takes coding as personal creation
>> and they get attached to it.
>> To my point you can take a look on some old issue in the Druid forum
>> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
>> and am sure other communities have similar problems.
>> So leaving the door open to some side cases is not a good idea in my
>> opinion and will lead to similar issue in the future.
>> 
>> This seems to me especially likely to happen in cases
>>> where an approach still needs proven to be a viable idea *to the author*,
>>> so that a much more productive discussion can be had in the first place.
>>> 
>>> I think there is a trade off, I don't think we want to discourage
>>> experimentation by walling it off behind mandatory discussions before it
>>> can even start, but I do think formalizing the process for large changes
>> is
>>> a good thing, especially since we probably can't expect the wider
>> community
>>> to have the same attitude towards a large PR getting discarded as a
>>> committer might. I think the Kafka approach is reasonable, a bit more
>>> formal than our design review process but not overbearing.
>> 
>> 
>> Can you please explain what is overbearing ? what can be changed to make it
>> easy ?
>> Most of the points are kind of the actual questions that you want to
>> address before hand anyway isn't ?
>> 
>> 
>>> Going code first
>>> should be in general discouraged, but when it does happen, it seems like
>>> opening DIP/an issue/starting a mailing list thread or whatever we go
>> with
>>> to have a more high level design discussion alongside the reference PR
>>> could alleviate some of these complaints?
>> 
>> 
>> What are the complaints ?
>> 
>> 
>>> +1 for "DIP" heh, I think making
>>> them in the form of github issues is probably appropriate, with a dev
>> list
>>> thread to announce them perhaps?
>>> 
>> 
>> I think  github issue with [Proposal] header like
>> https://github.com/apache/incubator-druid/issues/4349 is good to me,
>> 
>> Thanks!
>> 
>> 
>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <bs...@apache.org> wrote:
>>> 
>>>> Thanks everyone for interacting with this thread.
>>>> 
>>>> The fact that i, Roman, Jihoon  and others in the past (FJ
>>>> 
>>> 
>> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ)
>>>> raised this point indicates that PRs without a proposal are indeed an
>>> issue
>>>> and we need to solve it.
>>>> 
>>>> Something Similar to KIP maybe called DIPs is fine with me.
>>>> What i strive to see is the following:
>>>> 
>>>> [Step 1] formalize what is the kind of work that needs a formal
>>> Proposal, I
>>>> think Roman and Jihoon has already covered that pretty well. am +1 on
>>> that.
>>>> 
>>>> 
>>> 
>> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
>>>> 
>>>> I am strongly in favor of the separation of Proposal Review and (later)
>>>> Code review PRs. My  main reasons:
>>>> Most importantly code reviewing will introduce lot of noise and will
>>>> ultimately make  the GitHub page unreadable.
>>>> Avoid overlapping of work.
>>>> Once code is written hard to think abstract.
>>>> Separate page for Design review later can always be used it as a Design
>>>> document that is readable and code free-ish.
>>>> As i said the goal of this first round is to see if the community agree
>>>> about such change, then make the process of design more inclusive thus
>>>> other contributors can submit a counter proposals.
>>>> 
>>>> [Step 2] IF everybody agree about that point Step 2 is to define which
>>>> medium is used to Publish a primitive form of a CODE FREE Abstract
>>> Proposal
>>>> containing at least the following bullet points.
>>>> - The problem description and motivation
>>>> - Overview of the proposed change
>>>> - Operational impact (compatibility/ plans to upgrades) public API
>>> changes,
>>>> configuration changes, algorithm, and so on
>>>> - Expected benefits and drawbacks
>>>> - Rationale and alternatives
>>>> - Estimate Time to Deliver if possible.
>>>> 
>>>> The way i think this can be is a Github issue where member of the
>>> community
>>>> will interact via comments and the author will be updating the
>>> description
>>>> in the light of comments provided by the community.
>>>> 
>>>> During and near the end of the design discussions the author/s can
>> start
>>>> writing POCs to help guide the review process this naturally will be a
>>> Pull
>>>> request with actual code.
>>>> 
>>>> *Now the most important thing is that we need to agree that any work
>> that
>>>> does not align with this formal process will be ignored and the author
>>> will
>>>> be asked to start with a DIP*
>>>> *That is what i meant with  “If it didn’t happen on the mailing list,
>> it
>>>> didn’t happen.”*
>>>> 
>>>> Thanks and happy coding!
>>>> 
>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <gi...@apache.org> wrote:
>>>> 
>>>>> One of the advantages I see with a more formal process is (like Kafka
>>>> KIPs)
>>>>> is that it levels the playing field a bit and sets some ground rules
>>> for
>>>>> working together. In a way it can help encourage contributions by
>>> making
>>>> it
>>>>> clear what is expected of potential contributors.
>>>>> 
>>>>> We have a design review process today that is not as formal as KIPs,
>>> but
>>>> is
>>>>> somewhat heavier than the one you describe. Maybe we could tweak our
>>>>> current one by starting to do design reviews separately from PRs.
>> i.e.,
>>>> for
>>>>> anything that meets our 'design review' criteria, do that on the dev
>>> list
>>>>> or in a separate issue, and keep the PR focused on code-level stuff.
>>> That
>>>>> way we don't end up trying to do both at once. And it makes it easier
>>> to
>>>>> start talking about design before the code is ready, which would be
>>>> better.
>>>>> 
>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <jh...@apache.org> wrote:
>>>>> 
>>>>>> It’s really hard to say no to a contribution when someone has put
>> in
>>> a
>>>>>> significant amount of work.
>>>>>> 
>>>>>> The following approach is simple and works really well: Before you
>>>> start
>>>>>> work, log a case, describing the problem. When you have some ideas
>>>> about
>>>>>> design, add those to the case. When you have a code branch, add its
>>> URL
>>>>> to
>>>>>> the case. And so forth. At any point in the proceedings, people can
>>>> chime
>>>>>> in with their opinions.
>>>>>> 
>>>>>> In my opinion, a formal “design review” process is not necessary.
>>> Just
>>>>>> build consensus iteratively, by starting the conversation early in
>>> the
>>>>>> process.
>>>>>> 
>>>>>> Julian
>>>>>> 
>>>>>> 
>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <gi...@apache.org>
>> wrote:
>>>>>>> 
>>>>>>> In this particular case: please consider the PR as a proposal.
>>> Don't
>>>>> feel
>>>>>>> like just because there is code there that takes a certain
>>> approach,
>>>>> that
>>>>>>> the approach is somehow sacred. I had to implement something to
>>>>>> crystallize
>>>>>>> my own thinking about how the problem could be approached. I
>> won't
>>> be
>>>>>>> disappointed if, as a community, we decide a different direction
>> is
>>>>>> better
>>>>>>> and the code all gets thrown away. That's one of the reasons
>> that I
>>>>>> removed
>>>>>>> the 0.14.0 milestone that was added to the patch. (I don't want
>> to
>>>> rush
>>>>>> it,
>>>>>>> nor do I think that's a good idea.)
>>>>>>> 
>>>>>>> In general: Sounds like we could do with some more formalization
>>>> around
>>>>>>> what a proposal looks like, which sorts of changes need one, and
>>> when
>>>>> in
>>>>>>> the dev cycle it is appropriate. FWIW I think Kafka's process is
>>> more
>>>>> or
>>>>>>> less fine, and would be okay with adopting it for Druid if people
>>>> like
>>>>>> it.
>>>>>>> Right now our standards for what requires a "design review" are
>>> very
>>>>>>> similar to the Kafka community standards for what requires a KIP,
>>> so
>>>> we
>>>>>>> have some familiarity with those concepts. However we don't
>>> separate
>>>> PR
>>>>>>> review and proposal discussion as strictly as they do, which
>> seems
>>> to
>>>>> be
>>>>>>> the foundation for the feeling of exclusion that is being felt
>>> here.
>>>>>>> 
>>>>>>> Separately: I just redid the description on
>>>>>>> https://github.com/apache/incubator-druid/pull/6794 to be more
>>>>>> proposal-y.
>>>>>>> I followed the KIP style:
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>>>>>> .
>>>>>>> Please refresh the page and see if it looks more useful.
>>>>>>> 
>>>>>>> Gian
>>>>>>> 
>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org>
>>>> wrote:
>>>>>>> 
>>>>>>>> Slim,
>>>>>>>> 
>>>>>>>> I agree with your points that offline development is bad for
>>>>> community.
>>>>>>>> But I don’t think you need much mentor help. You have raised
>> valid
>>>>>> issues
>>>>>>>> and the Druid community needs to decide what its development
>>>> practices
>>>>>>>> should be.
>>>>>>>> 
>>>>>>>> Julian
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <bs...@apache.org>
>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Hello everyone and hope you all have very good holidays.
>>>>>>>>> 
>>>>>>>>> First, this email is not directed on the author or the PR
>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794  it self,
>>> but
>>>> i
>>>>>> see
>>>>>>>>> this PR as a perfect example.
>>>>>>>>> 
>>>>>>>>> One of the foundation of Apache Way or what i would simply call
>>>> open
>>>>>>>> source
>>>>>>>>> community driven development is that "Technical decisions are
>>>>>> discussed,
>>>>>>>>> decided, and archived publicly.
>>>>>>>>> developpement"
>>>>>>>>> Which means that big technical  changes such as the one brought
>>> by
>>>>>> #/6794
>>>>>>>>> should have started as a proposal and round of discussions
>> about
>>>> the
>>>>>>>> major
>>>>>>>>> changes designs not as 11K line of code.
>>>>>>>>> I believe such openness will promote a lot of good benefits
>> such
>>>> as:
>>>>>>>>> 
>>>>>>>>> - ensures community health and growth.
>>>>>>>>> - ensures everyone can participate not only the authors and his
>>>>>>>> co-workers.
>>>>>>>>> - ensures that the project is driven by the community and not a
>>>> given
>>>>>>>>> company or an individual.
>>>>>>>>> - ensures that there is consensus (not saying 100% agreement;)
>>>>> however
>>>>>> it
>>>>>>>>> means that all individuals will accept the current progress on
>>> the
>>>>>>>> project
>>>>>>>>> until some better proposal is put forth.
>>>>>>>>> 
>>>>>>>>> Personally such BIG offline PR makes me feel excluded and
>> doesn't
>>>>> give
>>>>>>>> me a
>>>>>>>>> sense that i belong to  a community at all.
>>>>>>>>> 
>>>>>>>>> To prevent such off list development i think as a Druid
>> Community
>>>> we
>>>>>> need
>>>>>>>>> to stick to the apache way “If it didn’t happen on the mailing
>>>> list,
>>>>> it
>>>>>>>>> didn’t happen.”
>>>>>>>>> 
>>>>>>>>> I would appreciate if some of the Apache mentor help with this.
>>>>>>>>> Thanks
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
>> --
>> 
>> B-Slim
>> _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org

Re: Off list major development

[Eyal Yurman <ey...@verizonmedia.com.INVALID>]

Hi, I have created an Issue together with @jon-wei, if anyone wants to
chime in:
https://github.com/apache/incubator-druid/issues/6949 (Create a proposal
template #6949)

On Tue, Jan 15, 2019 at 12:07 PM Jihoon Son <gh...@gmail.com> wrote:

> Good point.
> If some authors raise PRs without noticing the need for a proposal, we
> shouldn't ask them to close their PRs only because of the absence of the
> proposal.
>
> "Design review" without a proposal for simple PRs would be good if we can
> determine well what PRs need and what don't.
> But, how do we know? Even for the same PR, someone may think it needs a
> proposal but another may not.
>
> If someone don't notice the need for a proposal and raise a PR without it,
> I'm fine with that.
> However, we should still encourage writing a proposal before writing code
> because we can avoid unnecessary effort.
>
> I think this kind of issue usually happens for first time contributors and
> they will be better once they get used to Druid development.
> And I believe someday even first contributors would follow this policy once
> it gets settled down well in the community as Kafka community does.
>
> Jihoon
>
> On Tue, Jan 15, 2019 at 4:31 AM Roman Leventov <le...@gmail.com>
> wrote:
>
> > In such small PRs, authors likely won't be aware that they need to
> create a
> > proposal in the first place. The first reviewer just adds the "Design
> > Review" tag. It's also absolutely not about considering designs and
> gauging
> > the proposal, it's just verifying that a configuration / parameter / HTTP
> > endpoint name is reasonable and aligned with the rest of Druid. So I
> think
> > that a separate proposal issue for such PRs is unnecessary bureaucracy.
> >
> > On Tue, 15 Jan 2019 at 07:45, Jihoon Son <gh...@gmail.com> wrote:
> >
> > > Roman,
> > >
> > > > Jihoon in
> > >
> > >
> >
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> > > and later Gian in this thread suggested that _every_ piece of work that
> > > should be labelled as "Design Review" according to the current rules
> > should
> > > be accompanied by an issue. I don't agree with this, there are some PRs
> > as
> > > small as a few dozens of lines of code, that add some configuration
> > > parameter and therefore should be labelled "Design Review". I don't
> > thing a
> > > separate proposal issue is needed for them, and even for a little
> larger
> > > PRs too.
> > >
> > > What I'm concerned with is how people feel if their design is not
> > accepted
> > > even though they wrote code. Of course, as Clint said, sometimes code
> > helps
> > > better understanding of the proposal. But, I believe this is the case
> > when
> > > the proposal is quite complicated and not easy to understand without
> > code.
> > > Also the authors should be aware of that they might rewrite the entire
> > code
> > > if the design should be changed.
> > >
> > > If writing code is simple, I don't see why the authors don't wait until
> > the
> > > review for their proposal is finished.
> > >
> > > Jihoon
> > >
> > > On Fri, Jan 11, 2019 at 9:51 AM Fangjin Yang <fa...@imply.io> wrote:
> > >
> > > > I agree with Gian, as an Apache committer, your responsibility is for
> > the
> > > > betterment of the project. I agree it is in the best interest of the
> > > > project to stop thinking about what orgs people belong to. We are
> all a
> > > > part of the Apache software foundation, regardless of what our roles
> > and
> > > > titles are outside of it.
> > > >
> > > > On Fri, Jan 11, 2019 at 2:22 AM Roman Leventov <
> leventov.ru@gmail.com>
> > > > wrote:
> > > >
> > > > > It's not that people from one org could abuse the project and push
> > some
> > > > > change, but that they have similar perspective (bubble effect) and
> > some
> > > > > important aspects of a large feature could escape their attention.
> > > > >
> > > > > I suggest it to be not a rigid rule, but a recommendation for
> authors
> > > of
> > > > > large proposals to try to attract reviewers from other orgs.
> > > > >
> > > > > On Fri, 11 Jan 2019 at 02:51, Julian Hyde <jh...@apache.org>
> wrote:
> > > > >
> > > > > > I agree with Gian.
> > > > > >
> > > > > > As an Apache committer, you only have one affiliation: you are
> > > working
> > > > in
> > > > > > the best interests of the project.
> > > > > >
> > > > > > Obviously, in the real world there are other pressures. But we do
> > our
> > > > > best
> > > > > > to compensate for them.
> > > > > >
> > > > > > Also, as a a community we try to design our process so as to
> avoid
> > > > undue
> > > > > > influences. For instance, when I advocate for logging cases
> early,
> > I
> > > am
> > > > > > trying to mitigate the effect of product managers and VPs of
> > > > engineering,
> > > > > > who like to have their say in meeting rooms rather than on public
> > > > mailing
> > > > > > lists. That’s just one example; if we see other influences at
> play,
> > > > let’s
> > > > > > evolve our process to try to level the playing field.
> > > > > >
> > > > > > Julian
> > > > > >
> > > > > >
> > > > > > > On Jan 10, 2019, at 10:40 AM, Gian Merlino <gi...@apache.org>
> > > wrote:
> > > > > > >
> > > > > > >>> What do community members think about also making a
> requirement
> > > > that
> > > > > > >>> "Design Review" PRs and proposals are reviewed by at least
> two
> > > > people
> > > > > > > with
> > > > > > >>> different affiliation?
> > > > > > >> This seems pretty reasonable to me. I haven't found anything
> in
> > > > Apache
> > > > > > >> voting procedure docs (
> > > > https://www.apache.org/foundation/voting.html)
> > > > > > that
> > > > > > >> seems to explicitly forbid something like this yet at least.
> > > > > > >
> > > > > > > On the other hand, my understanding of the Apache Way would
> mean
> > > that
> > > > > > this
> > > > > > > kind of rule doesn't make sense. In particular from
> > > > > > > https://www.apache.org/foundation/how-it-works.html: "We
> firmly
> > > > > believe
> > > > > > in
> > > > > > > hats. Your role at the ASF is one assigned to you personally,
> and
> > > is
> > > > > > > bestowed on you by your peers. It is not tied to your job or
> > > current
> > > > > > > employer or company." That sentiment seems incompatible with
> > making
> > > > > > > explicit rules about organizational diversity in voting. I have
> > > also
> > > > > > heard
> > > > > > > a few people say things like: people are supposed to represent
> > > > > > themselves,
> > > > > > > not their employers.
> > > > > > >
> > > > > > > Obviously, though, people's actions and opinions are influenced
> > by
> > > > > their
> > > > > > > employer. IMO, a better way to approach a potential problem
> there
> > > is
> > > > > that
> > > > > > > if people from a particular organization end up behaving
> > abusively,
> > > > > then
> > > > > > > PMC members from other organizations (or in extreme situations
> > the
> > > > > Apache
> > > > > > > board itself) should tell them to knock it off. And then do
> more
> > > > > serious
> > > > > > > things if the inappropriate behavior continues. This kind
> > > > > > >
> > > > > > > The other thing you brought up, promoting more shared ownership
> > and
> > > > > > > awareness, I am hopeful that separating proposals from PRs will
> > > help
> > > > > with
> > > > > > > that. One reason is that it takes much less time to understand
> a
> > > well
> > > > > > > written proposal than it takes to understand a PR. Code is very
> > > clear
> > > > > but
> > > > > > > it is also very verbose and takes a while to read and
> understand.
> > > So
> > > > in
> > > > > > > theory changing how we operate, in this way, should promote
> more
> > > > > > > understanding of more people across the code base.
> > > > > > >
> > > > > > > On Thu, Jan 10, 2019 at 1:34 AM Clint Wylie <
> > clint.wylie@imply.io>
> > > > > > wrote:
> > > > > > >
> > > > > > >>>
> > > > > > >>> What do community members think about also making a
> requirement
> > > > that
> > > > > > >>> "Design Review" PRs and proposals are reviewed by at least
> two
> > > > people
> > > > > > >> with
> > > > > > >>> different affiliation?
> > > > > > >>
> > > > > > >>
> > > > > > >> This seems pretty reasonable to me. I haven't found anything
> in
> > > > Apache
> > > > > > >> voting procedure docs (
> > > > https://www.apache.org/foundation/voting.html)
> > > > > > that
> > > > > > >> seems to explicitly forbid something like this yet at least.
> > > > > > >>
> > > > > > >> On Wed, Jan 9, 2019 at 8:52 PM Roman Leventov <
> > > > leventov.ru@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> What do community members think about also making a
> requirement
> > > > that
> > > > > > >>> "Design Review" PRs and proposals are reviewed by at least
> two
> > > > people
> > > > > > >> with
> > > > > > >>> different affiliation? IMO it's a good idea, because it
> ensures
> > > > that
> > > > > > >>> different interests are taken into account. Also it pushes
> > people
> > > > to
> > > > > > >> engage
> > > > > > >>> with work done in other parts of Druid, improving shared code
> > > > owning
> > > > > > and
> > > > > > >>> awareness.
> > > > > > >>>
> > > > > > >>> Or it's against the Apache Way because it's assumed that
> there
> > > are
> > > > no
> > > > > > >>> company boundaries within the community?
> > > > > > >>>
> > > > > > >>> On Thu, 10 Jan 2019 at 11:45, Roman Leventov <
> > > > leventov.ru@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>> I see two important reasons why it makes sense to file an
> > issue
> > > > and
> > > > > > >>>> probably announce it in the mailing list, before writing a
> lot
> > > of
> > > > > > code,
> > > > > > >>>> despite not having a clear picture of what it will be and
> any
> > > > > > >> performance
> > > > > > >>>> data:
> > > > > > >>>> 1) somebody could already work on this problem privately in
> > > > > parallel,
> > > > > > >> it
> > > > > > >>>> allows to avoid clash in people's works
> > > > > > >>>> 2) some people could quickly think about the problem field
> and
> > > > share
> > > > > > >>>> high-level ideas that could wildly change the direction in
> > which
> > > > the
> > > > > > >>> author
> > > > > > >>>> (the person who is going to write code) will move in his
> work
> > > from
> > > > > > >> early
> > > > > > >>> on.
> > > > > > >>>>
> > > > > > >>>> Jihoon in
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> > > > > > >>> and
> > > > > > >>>> later Gian in this thread suggested that _every_ piece of
> work
> > > > that
> > > > > > >>> should
> > > > > > >>>> be labelled as "Design Review" according to the current
> rules
> > > > should
> > > > > > be
> > > > > > >>>> accompanied by an issue. I don't agree with this, there are
> > some
> > > > PRs
> > > > > > as
> > > > > > >>>> small as a few dozens of lines of code, that add some
> > > > configuration
> > > > > > >>>> parameter and therefore should be labelled "Design Review".
> I
> > > > don't
> > > > > > >>> thing a
> > > > > > >>>> separate proposal issue is needed for them, and even for a
> > > little
> > > > > > >> larger
> > > > > > >>>> PRs too.
> > > > > > >>>>
> > > > > > >>>> For the same reason, I also don't see the point of renaming
> > > > "Design
> > > > > > >>>> Review" into "Proposal", as well as separating "Design
> Review"
> > > > into
> > > > > > >>>> "Proposal" and something like "API Review". I think a single
> > > > "Design
> > > > > > >>>> Review" tag handles it well.
> > > > > > >>>>
> > > > > > >>>> Gian mentioned an idea that PRs that follow a "Design
> Review"
> > > > > proposal
> > > > > > >>>> issue shouldn't be "Design Review" themselves. I don't agree
> > > with
> > > > > > >> this, I
> > > > > > >>>> think that actual code and performance data are important
> > inputs
> > > > > that
> > > > > > >>>> should be re-evaluated at least by two people. I even think
> > that
> > > > > it's
> > > > > > >>> very
> > > > > > >>>> desirable that at least two people read _each line of
> > production
> > > > > code_
> > > > > > >> in
> > > > > > >>>> large PRs, although it's not what was done historically in
> > > Druid,
> > > > > > >> because
> > > > > > >>>> large bodies of newly added code, with whole new classes and
> > > > > > subsystems
> > > > > > >>>> added, are also coincidentally tested worse than already
> > > existing
> > > > > > >> classes
> > > > > > >>>> and subsystems, including in production. It seems to me that
> > > those
> > > > > > huge
> > > > > > >>>> code influxes is a major source of bugs, that could later
> take
> > > > years
> > > > > > to
> > > > > > >>>> squeeze out from the codebase.
> > > > > > >>>>
> > > > > > >>>> On Wed, 9 Jan 2019 at 08:24, Clint Wylie <
> > clint.wylie@imply.io>
> > > > > > wrote:
> > > > > > >>>>
> > > > > > >>>>> Apologies for the delayed response.
> > > > > > >>>>>
> > > > > > >>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > > > > > >>> slim.bouguerra@gmail.com>
> > > > > > >>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> I am wondering here what is the case where code first is
> > > better?
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>> I don't think it's wrong to share ideas as early as
> possible,
> > > and
> > > > > > >> after
> > > > > > >>>>> this discussion I think I am in favor of it too. I just
> meant
> > > > that
> > > > > I
> > > > > > >>> don't
> > > > > > >>>>> think it's always necessarily the most productive
> discussion
> > > > until
> > > > > > >> code
> > > > > > >>>>> exists sometimes, with the types of thing I am thinking of
> > are
> > > > > almost
> > > > > > >>>>> entirely limited to cases where things might sound good to
> > > anyone
> > > > > on
> > > > > > >>> paper
> > > > > > >>>>> but in reality need a large amount of experiments conducted
> > and
> > > > > > >>>>> observations collected to determine that something is
> > actually
> > > > > worth
> > > > > > >>>>> doing,
> > > > > > >>>>> which I imagine is mainly things like reworking internals
> for
> > > > > > >>> performance
> > > > > > >>>>> improvements.
> > > > > > >>>>>
> > > > > > >>>>> In the case of my combined proposal PR, I needed to prove
> > that
> > > > the
> > > > > > >>> thing I
> > > > > > >>>>> was working on was a good idea... and it wasn't directly.
> > But I
> > > > > came
> > > > > > >> up
> > > > > > >>>>> with another idea during the course of experiment turned
> into
> > > > > > >> something
> > > > > > >>>>> compelling, so an initial proposal would have looked quite
> a
> > > lot
> > > > > > >>> different
> > > > > > >>>>> than what I ended up with. Once I had proven to myself that
> > it
> > > > was
> > > > > a
> > > > > > >>> good
> > > > > > >>>>> idea, then I was comfortable sharing with the wider
> > community.
> > > > I'm
> > > > > > not
> > > > > > >>>>> certain how this would play out in an always proposal first
> > > > model,
> > > > > > >> maybe
> > > > > > >>>>> the first proposal exists, I personally reject it after
> > > updating
> > > > > with
> > > > > > >>>>> experiment results show it's a bad idea, continue
> > experimenting
> > > > and
> > > > > > >>> raise
> > > > > > >>>>> a
> > > > > > >>>>> new one after the experiments start looking promising?
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>>> Let's not be naive this is very rare that a contributor
> will
> > > > > accept
> > > > > > >>> that
> > > > > > >>>>>> his work is to be thrown, usually devs takes coding as
> > > personal
> > > > > > >>> creation
> > > > > > >>>>>> and they get attached to it.
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>> I agree, just because a handful of the committers have this
> > > > > attitude,
> > > > > > >> it
> > > > > > >>>>> isn't fair to expect the wider community to also, that's
> why
> > I
> > > am
> > > > > in
> > > > > > >>> favor
> > > > > > >>>>> of formalizing the process.
> > > > > > >>>>>
> > > > > > >>>>> Can you please explain what is overbearing ? what can be
> > > changed
> > > > to
> > > > > > >> make
> > > > > > >>>>> it
> > > > > > >>>>>> easy ?
> > > > > > >>>>>> Most of the points are kind of the actual questions that
> you
> > > > want
> > > > > to
> > > > > > >>>>>> address before hand anyway isn't ?
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>> Sorry for the confusion, I said it's "not overbearing", I
> > think
> > > > > it's
> > > > > > >>> fine.
> > > > > > >>>>>
> > > > > > >>>>> What are the complaints ?
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> Is this and other previous threads not a complaint about
> > > opening
> > > > a
> > > > > > >> large
> > > > > > >>>>> PR
> > > > > > >>>>> without a proposal? :) I just mean that formalizing the
> > > process,
> > > > > even
> > > > > > >>> if a
> > > > > > >>>>> proposal has a reference PR opened with it near
> concurrently,
> > > > could
> > > > > > >>>>> prevent
> > > > > > >>>>> these discussions from happening in the future because
> ground
> > > > rules
> > > > > > >> have
> > > > > > >>>>> been set and we are all on the same page I guess.
> > > > > > >>>>>
> > > > > > >>>>> I think we should also probably consider renaming the
> "design
> > > > > review"
> > > > > > >>>>> label
> > > > > > >>>>> to "proposal" or something to make it more clear that a PR
> is
> > > > > > >> associated
> > > > > > >>>>> with a proposal. It might also be worth considering using
> > > github
> > > > > > >>> projects
> > > > > > >>>>> for particularly large things that involve multiple follow
> up
> > > > PRs,
> > > > > > >> but I
> > > > > > >>>>> haven't used them in much depth to know if they add
> anything.
> > > > > > >>>>>
> > > > > > >>>>> It seems like we are all converging on agreement to do a
> > github
> > > > > issue
> > > > > > >>>>> proposal for larger changes (I would vote for announcing
> them
> > > on
> > > > > the
> > > > > > >> dev
> > > > > > >>>>> list too for more visibility), so that design review is
> > > separated
> > > > > > from
> > > > > > >>>>> code
> > > > > > >>>>> review. I guess my main concern was not wanting to
> discourage
> > > > > > >>>>> experimentation by walling it off behind mandatory
> > discussions,
> > > > but
> > > > > > >> the
> > > > > > >>>>> more I think about it,  it doesn't technically change much
> > > about
> > > > > this
> > > > > > >>>>> process, it just requires a more formal proposal to
> accompany
> > > any
> > > > > > >>>>> experiments that are shared as a PR.
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> On Tue, Jan 8, 2019 at 12:28 PM Gian Merlino <
> > gian@apache.org>
> > > > > > wrote:
> > > > > > >>>>>
> > > > > > >>>>>> I think for us, choosing to use GitHub issues as
> discussion
> > > > > threads
> > > > > > >>> for
> > > > > > >>>>>> potential 'major' contributions would be a good idea,
> > > especially
> > > > > if
> > > > > > >> we
> > > > > > >>>>>> encourage people to start them before PRs show up.
> > Definitely
> > > > > agree
> > > > > > >>> that
> > > > > > >>>>>> all contributors should go through the same process -- I
> > > > couldn't
> > > > > > >> see
> > > > > > >>> it
> > > > > > >>>>>> working well any other way.
> > > > > > >>>>>>
> > > > > > >>>>>> On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <
> > jhyde@apache.org
> > > >
> > > > > > >> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> Statically, yes, GitHub PRs are the same as GitHub cases.
> > But
> > > > > > >>>>>> dynamically,
> > > > > > >>>>>>> they are different, because you can only log a PR when
> you
> > > have
> > > > > > >>>>> finished
> > > > > > >>>>>>> work.
> > > > > > >>>>>>>
> > > > > > >>>>>>> A lot of other Apache projects use JIRA, so there is a
> > clear
> > > > > > >>>>> distinction
> > > > > > >>>>>>> between cases and contributions. JIRA cases, especially
> > when
> > > > > > >> logged
> > > > > > >>>>> early
> > > > > > >>>>>>> in the lifecycle of a contribution, become long-running
> > > > > > >> conversation
> > > > > > >>>>>>> threads with a lot of community participation. If the
> Druid
> > > > chose
> > > > > > >> to
> > > > > > >>>>> do
> > > > > > >>>>>> so,
> > > > > > >>>>>>> GitHub cases could be the same.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Be careful that you do not treat “potential contributors”
> > (by
> > > > > > >> which
> > > > > > >>> I
> > > > > > >>>>>>> presume you mean non-committers) differently from
> > committers
> > > > and
> > > > > > >> PMC
> > > > > > >>>>>>> members. Anyone starting a major piece of work should
> > follow
> > > > the
> > > > > > >>> same
> > > > > > >>>>>>> process. (Experienced committers probably have a somewhat
> > > > better
> > > > > > >>> idea
> > > > > > >>>>>> what
> > > > > > >>>>>>> work will turn out to be “major”, so they get a little
> more
> > > > > > >> leeway.)
> > > > > > >>>>>>>
> > > > > > >>>>>>> Julian
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>> On Jan 7, 2019, at 12:10 PM, Gian Merlino <
> > gian@apache.org>
> > > > > > >>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I don't think there's a need to raise issues for every
> > > > change: a
> > > > > > >>>>> small
> > > > > > >>>>>>> bug
> > > > > > >>>>>>>> fix or doc fix should just go straight to PR. (GitHub
> PRs
> > > show
> > > > > > >> up
> > > > > > >>> as
> > > > > > >>>>>>> issues
> > > > > > >>>>>>>> in the issue-search UI/API, so it's not like this means
> > the
> > > > > > >> patch
> > > > > > >>>>> has
> > > > > > >>>>>> no
> > > > > > >>>>>>>> corresponding issue -- in a sense the PR _is_ the
> issue.)
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I do think it makes sense to encourage potential
> > > contributors
> > > > to
> > > > > > >>>>> write
> > > > > > >>>>>> to
> > > > > > >>>>>>>> the dev list or raise an issue if they aren't sure if
> > > > something
> > > > > > >>>>> would
> > > > > > >>>>>>> need
> > > > > > >>>>>>>> to go through a more heavy weight process.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Fwiw we do have a set of 'design review' criteria
> already
> > > (we
> > > > > > >> had
> > > > > > >>> a
> > > > > > >>>>>>>> discussion about this a couple years ago) at:
> > > > > > >>>>>>>>
> http://druid.io/community/#getting-your-changes-accepted.
> > > So
> > > > we
> > > > > > >>>>>>> wouldn't be
> > > > > > >>>>>>>> starting from zero on defining that. We set it up back
> > when
> > > we
> > > > > > >>> were
> > > > > > >>>>>>> trying
> > > > > > >>>>>>>> to _streamline_ our process -- we used to require two
> > > > non-author
> > > > > > >>> +1s
> > > > > > >>>>>> for
> > > > > > >>>>>>>> _every_ change, even minor ones. The introduction of
> > design
> > > > > > >> review
> > > > > > >>>>>>> criteria
> > > > > > >>>>>>>> was meant to classify which PRs need that level of
> review
> > > and
> > > > > > >>> which
> > > > > > >>>>>> ones
> > > > > > >>>>>>>> are minor and can be merged with less review. I do think
> > it
> > > > > > >> helped
> > > > > > >>>>> with
> > > > > > >>>>>>>> getting minor PRs merged more quickly. The list of
> > criteria
> > > > is,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> - Major architectural changes or API changes
> > > > > > >>>>>>>> - HTTP requests and responses (e. g. a new HTTP
> endpoint)
> > > > > > >>>>>>>> - Interfaces for extensions
> > > > > > >>>>>>>> - Server configuration (e. g. altering the behavior of a
> > > > config
> > > > > > >>>>>> property)
> > > > > > >>>>>>>> - Emitted metrics
> > > > > > >>>>>>>> - Other major changes, judged by the discretion of Druid
> > > > > > >>> committers
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Some of it is subjective, but it has been in place for a
> > > > while,
> > > > > > >> so
> > > > > > >>>>> it's
> > > > > > >>>>>>> at
> > > > > > >>>>>>>> least something we are relatively familiar with.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <
> > > jhyde@apache.org
> > > > >
> > > > > > >>>>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Small contributions don’t need any design review,
> whereas
> > > > large
> > > > > > >>>>>>>>> contributions need significant review. I don’t think we
> > > > should
> > > > > > >>>>> require
> > > > > > >>>>>>> an
> > > > > > >>>>>>>>> additional step for those (many) small contributions.
> But
> > > who
> > > > > > >>>>> decides
> > > > > > >>>>>>>>> whether a contribution fits into the small or large
> > > category?
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> I think the solution is for authors to log a case (or
> > send
> > > an
> > > > > > >>>>> email to
> > > > > > >>>>>>>>> dev) before they start work on any contribution. Then
> > > > > > >> committers
> > > > > > >>>>> can
> > > > > > >>>>>>>>> request a more heavy-weight process if they think it is
> > > > needed.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Julian
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <
> > > gian@apache.org>
> > > > > > >>>>> wrote:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> It sounds like splitting design from code review is a
> > > common
> > > > > > >>> theme
> > > > > > >>>>>> in a
> > > > > > >>>>>>>>> few
> > > > > > >>>>>>>>>> of the posts here. How does everyone feel about
> making a
> > > > point
> > > > > > >>> of
> > > > > > >>>>>>>>>> encouraging design reviews to be done as issues,
> > separate
> > > > from
> > > > > > >>> the
> > > > > > >>>>>> pull
> > > > > > >>>>>>>>>> request, with the expectations that (1) the design
> > review
> > > > > > >> issue
> > > > > > >>>>>>>>>> ("proposal") should generally appear somewhat _before_
> > the
> > > > > > >> pull
> > > > > > >>>>>>> request;
> > > > > > >>>>>>>>>> (2) pull requests should _not_ have design review
> happen
> > > on
> > > > > > >>> them,
> > > > > > >>>>>>> meaning
> > > > > > >>>>>>>>>> there should no longer be PRs with design review tags,
> > and
> > > > we
> > > > > > >>>>> should
> > > > > > >>>>>>> move
> > > > > > >>>>>>>>>> the design review approval process to the issue rather
> > > than
> > > > > > >> the
> > > > > > >>>>> PR.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> For (1), even if we encourage design review
> discussions
> > to
> > > > > > >> start
> > > > > > >>>>>>> before a
> > > > > > >>>>>>>>>> pull request appears, I don't see an issue with them
> > > running
> > > > > > >>>>>>> concurrently
> > > > > > >>>>>>>>>> for a while at some point.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <
> > > > > > >> jonwei@apache.org>
> > > > > > >>>>>> wrote:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Thanks for raising these concerns!
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> My initial thoughts:
> > > > > > >>>>>>>>>>> - I agree that separation of design review and
> > code-level
> > > > > > >>> review
> > > > > > >>>>> for
> > > > > > >>>>>>>>> major
> > > > > > >>>>>>>>>>> changes would be more efficient
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> - I agree that a clear, more formalized process for
> > > > handling
> > > > > > >>>>> major
> > > > > > >>>>>>>>> changes
> > > > > > >>>>>>>>>>> would be helpful for contributors:
> > > > > > >>>>>>>>>>> - Define what is considered a major change
> > > > > > >>>>>>>>>>> - Define a standard proposal structure, KIP-style
> > > proposal
> > > > > > >>> format
> > > > > > >>>>>>>>> sounds
> > > > > > >>>>>>>>>>> good to me
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> - I think it's too rigid to have a policy of "no code
> > at
> > > > all
> > > > > > >>> with
> > > > > > >>>>>> the
> > > > > > >>>>>>>>>>> initial proposal"
> > > > > > >>>>>>>>>>> - Code samples can be useful references for
> > understanding
> > > > > > >>> aspects
> > > > > > >>>>>> of a
> > > > > > >>>>>>>>>>> design
> > > > > > >>>>>>>>>>> - In some cases it's necessary to run experiments to
> > > fully
> > > > > > >>>>>> understand
> > > > > > >>>>>>> a
> > > > > > >>>>>>>>>>> problem and determine an appropriate design, or to
> > > > determine
> > > > > > >>>>> whether
> > > > > > >>>>>>>>>>> something is even worth doing before committing to
> the
> > > work
> > > > > > >> of
> > > > > > >>>>>>> fleshing
> > > > > > >>>>>>>>> out
> > > > > > >>>>>>>>>>> a proposal, prototype code is a natural outcome of
> that
> > > and
> > > > > > >> I'm
> > > > > > >>>>> not
> > > > > > >>>>>>>>> against
> > > > > > >>>>>>>>>>> someone providing such code for reference
> > > > > > >>>>>>>>>>> - I tend to view design/code as things that are often
> > > > > > >> developed
> > > > > > >>>>>>>>>>> simultaneously in an intertwined way
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Let's not be naive this is very rare that a
> > contributor
> > > > will
> > > > > > >>>>> accept
> > > > > > >>>>>>>>> that
> > > > > > >>>>>>>>>>> his work is to be thrown, usually devs takes coding
> as
> > > > > > >> personal
> > > > > > >>>>>>> creation
> > > > > > >>>>>>>>>>> and they get attached to it.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> If we have a clear review process that emphasizes the
> > > need
> > > > > > >> for
> > > > > > >>>>> early
> > > > > > >>>>>>>>>>> consensus building, with separate design and code
> > review,
> > > > > > >> then
> > > > > > >>> I
> > > > > > >>>>>> feel
> > > > > > >>>>>>>>> we've
> > > > > > >>>>>>>>>>> done enough and don't need a hard rule against having
> > > some
> > > > > > >> code
> > > > > > >>>>>> linked
> > > > > > >>>>>>>>> with
> > > > > > >>>>>>>>>>> the initial proposal. If a potential contributor then
> > > still
> > > > > > >>>>> wants to
> > > > > > >>>>>>> go
> > > > > > >>>>>>>>>>> ahead and write a lot of code that may be rejected or
> > > > change
> > > > > > >>>>>>>>> significantly,
> > > > > > >>>>>>>>>>> the risks were made clear.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Once code is written hard to think abstract.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I can see the validity of the concern, but I
> personally
> > > > don't
> > > > > > >>>>> see it
> > > > > > >>>>>>> as
> > > > > > >>>>>>>>> a
> > > > > > >>>>>>>>>>> huge risk. My impression from the Druid PR reviews
> I've
> > > > seen
> > > > > > >> is
> > > > > > >>>>> that
> > > > > > >>>>>>> our
> > > > > > >>>>>>>>>>> reviewers are able to keep abstract design vs.
> > > > implementation
> > > > > > >>>>>> details
> > > > > > >>>>>>>>>>> separate and consider alternate designs when
> reviewing.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> To summarize I think it's probably enough to have a
> > > policy
> > > > > > >>> along
> > > > > > >>>>> the
> > > > > > >>>>>>>>> lines
> > > > > > >>>>>>>>>>> of:
> > > > > > >>>>>>>>>>> - Create more formalized guidelines for proposals and
> > > what
> > > > > > >>>>> changes
> > > > > > >>>>>>>>> require
> > > > > > >>>>>>>>>>> proposals
> > > > > > >>>>>>>>>>> - Separate design and code review for major changes,
> > with
> > > > > > >>> design
> > > > > > >>>>>>> review
> > > > > > >>>>>>>>>>> first, code-level review after reaching consensus on
> > the
> > > > > > >>> design.
> > > > > > >>>>>>>>>>> - Code before the design review is completed is just
> > for
> > > > > > >>>>> reference,
> > > > > > >>>>>>> not
> > > > > > >>>>>>>>>>> regarded as a candidate for review/merging.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> - Jon
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > > > > > >>>>>>>>> slim.bouguerra@gmail.com>
> > > > > > >>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <
> > > > > > >>>>> clint.wylie@imply.io>
> > > > > > >>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> I am definitely biased in this matter as an owner
> of
> > > > > > >> another
> > > > > > >>>>> large
> > > > > > >>>>>>> PR
> > > > > > >>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>> wasn't preceded by a direct proposal or dev list
> > > > > > >> discussion,
> > > > > > >>>>> and
> > > > > > >>>>>> in
> > > > > > >>>>>>>>>>>> general
> > > > > > >>>>>>>>>>>>> I agree that proposal first is usually better, but
> I
> > > > think
> > > > > > >> in
> > > > > > >>>>> some
> > > > > > >>>>>>>>>>> rarer
> > > > > > >>>>>>>>>>>>> cases approaching a problem code first *is* the
> most
> > > > > > >>>>> appropriate
> > > > > > >>>>>> way
> > > > > > >>>>>>>>> to
> > > > > > >>>>>>>>>>>>> have a discussion.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I am wondering here what is the case where code
> first
> > is
> > > > > > >>> better?
> > > > > > >>>>>>>>>>>> In general when you are writing code you have an
> idea
> > > > about
> > > > > > >>> what
> > > > > > >>>>>> you
> > > > > > >>>>>>>>> want
> > > > > > >>>>>>>>>>>> to change, why you want to change and why you want
> to
> > > > change
> > > > > > >>> it.
> > > > > > >>>>>>>>>>>> I do not see what is wrong with sharing this
> primitive
> > > > ideas
> > > > > > >>> and
> > > > > > >>>>>>>>> thoughts
> > > > > > >>>>>>>>>>>> as an abstract proposal (at least to avoid
> > overlapping)
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I see nothing wrong with it so long as the author
> > > > > > >>>>>>>>>>>>> accepts that the PR is treated as a combined
> proposal
> > > and
> > > > > > >>>>> proof of
> > > > > > >>>>>>>>>>>> concept,
> > > > > > >>>>>>>>>>>>> and fair game to be radically changed via
> discussion
> > or
> > > > > > >> even
> > > > > > >>>>>>> rejected,
> > > > > > >>>>>>>>>>>>> which sounds like Gian's attitude on the matter and
> > is
> > > > mine
> > > > > > >>> as
> > > > > > >>>>>> well
> > > > > > >>>>>>>>>>> with
> > > > > > >>>>>>>>>>>> my
> > > > > > >>>>>>>>>>>>> compression stuff.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Let's not be naive this is very rare that a
> > contributor
> > > > will
> > > > > > >>>>> accept
> > > > > > >>>>>>>>> that
> > > > > > >>>>>>>>>>>> his work is to be thrown, usually devs takes coding
> as
> > > > > > >>> personal
> > > > > > >>>>>>>>> creation
> > > > > > >>>>>>>>>>>> and they get attached to it.
> > > > > > >>>>>>>>>>>> To my point you can take a look on some old issue in
> > the
> > > > > > >> Druid
> > > > > > >>>>>> forum
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> > > > > > >>>>>>>>>>>> and am sure other communities have similar problems.
> > > > > > >>>>>>>>>>>> So leaving the door open to some side cases is not a
> > > good
> > > > > > >> idea
> > > > > > >>>>> in
> > > > > > >>>>>> my
> > > > > > >>>>>>>>>>>> opinion and will lead to similar issue in the
> future.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> This seems to me especially likely to happen in
> cases
> > > > > > >>>>>>>>>>>>> where an approach still needs proven to be a viable
> > > idea
> > > > > > >> *to
> > > > > > >>>>> the
> > > > > > >>>>>>>>>>> author*,
> > > > > > >>>>>>>>>>>>> so that a much more productive discussion can be
> had
> > in
> > > > the
> > > > > > >>>>> first
> > > > > > >>>>>>>>>>> place.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> I think there is a trade off, I don't think we want
> > to
> > > > > > >>>>> discourage
> > > > > > >>>>>>>>>>>>> experimentation by walling it off behind mandatory
> > > > > > >>> discussions
> > > > > > >>>>>>> before
> > > > > > >>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>> can even start, but I do think formalizing the
> > process
> > > > for
> > > > > > >>>>> large
> > > > > > >>>>>>>>>>> changes
> > > > > > >>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>> a good thing, especially since we probably can't
> > expect
> > > > the
> > > > > > >>>>> wider
> > > > > > >>>>>>>>>>>> community
> > > > > > >>>>>>>>>>>>> to have the same attitude towards a large PR
> getting
> > > > > > >>> discarded
> > > > > > >>>>> as
> > > > > > >>>>>> a
> > > > > > >>>>>>>>>>>>> committer might. I think the Kafka approach is
> > > > reasonable,
> > > > > > >> a
> > > > > > >>>>> bit
> > > > > > >>>>>>> more
> > > > > > >>>>>>>>>>>>> formal than our design review process but not
> > > > overbearing.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Can you please explain what is overbearing ? what
> can
> > be
> > > > > > >>>>> changed to
> > > > > > >>>>>>>>> make
> > > > > > >>>>>>>>>>> it
> > > > > > >>>>>>>>>>>> easy ?
> > > > > > >>>>>>>>>>>> Most of the points are kind of the actual questions
> > that
> > > > you
> > > > > > >>>>> want
> > > > > > >>>>>> to
> > > > > > >>>>>>>>>>>> address before hand anyway isn't ?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Going code first
> > > > > > >>>>>>>>>>>>> should be in general discouraged, but when it does
> > > > happen,
> > > > > > >> it
> > > > > > >>>>>> seems
> > > > > > >>>>>>>>>>> like
> > > > > > >>>>>>>>>>>>> opening DIP/an issue/starting a mailing list thread
> > or
> > > > > > >>>>> whatever we
> > > > > > >>>>>>> go
> > > > > > >>>>>>>>>>>> with
> > > > > > >>>>>>>>>>>>> to have a more high level design discussion
> alongside
> > > the
> > > > > > >>>>>> reference
> > > > > > >>>>>>> PR
> > > > > > >>>>>>>>>>>>> could alleviate some of these complaints?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> What are the complaints ?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> +1 for "DIP" heh, I think making
> > > > > > >>>>>>>>>>>>> them in the form of github issues is probably
> > > > appropriate,
> > > > > > >>>>> with a
> > > > > > >>>>>>> dev
> > > > > > >>>>>>>>>>>> list
> > > > > > >>>>>>>>>>>>> thread to announce them perhaps?
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I think  github issue with [Proposal] header like
> > > > > > >>>>>>>>>>>>
> https://github.com/apache/incubator-druid/issues/4349
> > > is
> > > > > > >> good
> > > > > > >>>>> to
> > > > > > >>>>>> me,
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Thanks!
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <
> > > > > > >>>>> bslim@apache.org>
> > > > > > >>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Thanks everyone for interacting with this thread.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> The fact that i, Roman, Jihoon  and others in the
> > past
> > > > (FJ
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ
> > > > > > >>>>> )
> > > > > > >>>>>>>>>>>>>> raised this point indicates that PRs without a
> > > proposal
> > > > > > >> are
> > > > > > >>>>>> indeed
> > > > > > >>>>>>> an
> > > > > > >>>>>>>>>>>>> issue
> > > > > > >>>>>>>>>>>>>> and we need to solve it.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Something Similar to KIP maybe called DIPs is fine
> > > with
> > > > > > >> me.
> > > > > > >>>>>>>>>>>>>> What i strive to see is the following:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> [Step 1] formalize what is the kind of work that
> > > needs a
> > > > > > >>>>> formal
> > > > > > >>>>>>>>>>>>> Proposal, I
> > > > > > >>>>>>>>>>>>>> think Roman and Jihoon has already covered that
> > pretty
> > > > > > >> well.
> > > > > > >>>>> am
> > > > > > >>>>>> +1
> > > > > > >>>>>>> on
> > > > > > >>>>>>>>>>>>> that.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> I am strongly in favor of the separation of
> Proposal
> > > > > > >> Review
> > > > > > >>>>> and
> > > > > > >>>>>>>>>>> (later)
> > > > > > >>>>>>>>>>>>>> Code review PRs. My  main reasons:
> > > > > > >>>>>>>>>>>>>> Most importantly code reviewing will introduce lot
> > of
> > > > > > >> noise
> > > > > > >>>>> and
> > > > > > >>>>>>> will
> > > > > > >>>>>>>>>>>>>> ultimately make  the GitHub page unreadable.
> > > > > > >>>>>>>>>>>>>> Avoid overlapping of work.
> > > > > > >>>>>>>>>>>>>> Once code is written hard to think abstract.
> > > > > > >>>>>>>>>>>>>> Separate page for Design review later can always
> be
> > > used
> > > > > > >> it
> > > > > > >>>>> as a
> > > > > > >>>>>>>>>>> Design
> > > > > > >>>>>>>>>>>>>> document that is readable and code free-ish.
> > > > > > >>>>>>>>>>>>>> As i said the goal of this first round is to see
> if
> > > the
> > > > > > >>>>> community
> > > > > > >>>>>>>>>>> agree
> > > > > > >>>>>>>>>>>>>> about such change, then make the process of design
> > > more
> > > > > > >>>>> inclusive
> > > > > > >>>>>>>>>>> thus
> > > > > > >>>>>>>>>>>>>> other contributors can submit a counter proposals.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> [Step 2] IF everybody agree about that point Step
> 2
> > is
> > > > to
> > > > > > >>>>> define
> > > > > > >>>>>>>>>>> which
> > > > > > >>>>>>>>>>>>>> medium is used to Publish a primitive form of a
> CODE
> > > > FREE
> > > > > > >>>>>> Abstract
> > > > > > >>>>>>>>>>>>> Proposal
> > > > > > >>>>>>>>>>>>>> containing at least the following bullet points.
> > > > > > >>>>>>>>>>>>>> - The problem description and motivation
> > > > > > >>>>>>>>>>>>>> - Overview of the proposed change
> > > > > > >>>>>>>>>>>>>> - Operational impact (compatibility/ plans to
> > > upgrades)
> > > > > > >>> public
> > > > > > >>>>>> API
> > > > > > >>>>>>>>>>>>> changes,
> > > > > > >>>>>>>>>>>>>> configuration changes, algorithm, and so on
> > > > > > >>>>>>>>>>>>>> - Expected benefits and drawbacks
> > > > > > >>>>>>>>>>>>>> - Rationale and alternatives
> > > > > > >>>>>>>>>>>>>> - Estimate Time to Deliver if possible.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> The way i think this can be is a Github issue
> where
> > > > member
> > > > > > >>> of
> > > > > > >>>>> the
> > > > > > >>>>>>>>>>>>> community
> > > > > > >>>>>>>>>>>>>> will interact via comments and the author will be
> > > > updating
> > > > > > >>> the
> > > > > > >>>>>>>>>>>>> description
> > > > > > >>>>>>>>>>>>>> in the light of comments provided by the
> community.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> During and near the end of the design discussions
> > the
> > > > > > >>> author/s
> > > > > > >>>>>> can
> > > > > > >>>>>>>>>>>> start
> > > > > > >>>>>>>>>>>>>> writing POCs to help guide the review process this
> > > > > > >> naturally
> > > > > > >>>>> will
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>> Pull
> > > > > > >>>>>>>>>>>>>> request with actual code.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> *Now the most important thing is that we need to
> > agree
> > > > > > >> that
> > > > > > >>>>> any
> > > > > > >>>>>>> work
> > > > > > >>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>> does not align with this formal process will be
> > > ignored
> > > > > > >> and
> > > > > > >>>>> the
> > > > > > >>>>>>>>>>> author
> > > > > > >>>>>>>>>>>>> will
> > > > > > >>>>>>>>>>>>>> be asked to start with a DIP*
> > > > > > >>>>>>>>>>>>>> *That is what i meant with  “If it didn’t happen
> on
> > > the
> > > > > > >>>>> mailing
> > > > > > >>>>>>> list,
> > > > > > >>>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>>> didn’t happen.”*
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Thanks and happy coding!
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <
> > > > > > >>> gian@apache.org>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> One of the advantages I see with a more formal
> > > process
> > > > is
> > > > > > >>>>> (like
> > > > > > >>>>>>>>>>> Kafka
> > > > > > >>>>>>>>>>>>>> KIPs)
> > > > > > >>>>>>>>>>>>>>> is that it levels the playing field a bit and
> sets
> > > some
> > > > > > >>>>> ground
> > > > > > >>>>>>>>>>> rules
> > > > > > >>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>> working together. In a way it can help encourage
> > > > > > >>>>> contributions
> > > > > > >>>>>> by
> > > > > > >>>>>>>>>>>>> making
> > > > > > >>>>>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>>>> clear what is expected of potential contributors.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> We have a design review process today that is not
> > as
> > > > > > >> formal
> > > > > > >>>>> as
> > > > > > >>>>>>>>>>> KIPs,
> > > > > > >>>>>>>>>>>>> but
> > > > > > >>>>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>> somewhat heavier than the one you describe. Maybe
> > we
> > > > > > >> could
> > > > > > >>>>> tweak
> > > > > > >>>>>>>>>>> our
> > > > > > >>>>>>>>>>>>>>> current one by starting to do design reviews
> > > separately
> > > > > > >>> from
> > > > > > >>>>>> PRs.
> > > > > > >>>>>>>>>>>> i.e.,
> > > > > > >>>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>> anything that meets our 'design review' criteria,
> > do
> > > > that
> > > > > > >>> on
> > > > > > >>>>> the
> > > > > > >>>>>>>>>>> dev
> > > > > > >>>>>>>>>>>>> list
> > > > > > >>>>>>>>>>>>>>> or in a separate issue, and keep the PR focused
> on
> > > > > > >>> code-level
> > > > > > >>>>>>>>>>> stuff.
> > > > > > >>>>>>>>>>>>> That
> > > > > > >>>>>>>>>>>>>>> way we don't end up trying to do both at once.
> And
> > it
> > > > > > >> makes
> > > > > > >>>>> it
> > > > > > >>>>>>>>>>> easier
> > > > > > >>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>> start talking about design before the code is
> > ready,
> > > > > > >> which
> > > > > > >>>>> would
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>>>>>>> better.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <
> > > > > > >>> jhyde@apache.org
> > > > > > >>>>>>
> > > > > > >>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> It’s really hard to say no to a contribution
> when
> > > > > > >> someone
> > > > > > >>>>> has
> > > > > > >>>>>> put
> > > > > > >>>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>> significant amount of work.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> The following approach is simple and works
> really
> > > > well:
> > > > > > >>>>> Before
> > > > > > >>>>>>>>>>> you
> > > > > > >>>>>>>>>>>>>> start
> > > > > > >>>>>>>>>>>>>>>> work, log a case, describing the problem. When
> you
> > > > have
> > > > > > >>> some
> > > > > > >>>>>>>>>>> ideas
> > > > > > >>>>>>>>>>>>>> about
> > > > > > >>>>>>>>>>>>>>>> design, add those to the case. When you have a
> > code
> > > > > > >>> branch,
> > > > > > >>>>> add
> > > > > > >>>>>>>>>>> its
> > > > > > >>>>>>>>>>>>> URL
> > > > > > >>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>> the case. And so forth. At any point in the
> > > > proceedings,
> > > > > > >>>>> people
> > > > > > >>>>>>>>>>> can
> > > > > > >>>>>>>>>>>>>> chime
> > > > > > >>>>>>>>>>>>>>>> in with their opinions.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> In my opinion, a formal “design review” process
> is
> > > not
> > > > > > >>>>>> necessary.
> > > > > > >>>>>>>>>>>>> Just
> > > > > > >>>>>>>>>>>>>>>> build consensus iteratively, by starting the
> > > > > > >> conversation
> > > > > > >>>>> early
> > > > > > >>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>> process.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Julian
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <
> > > > > > >>> gian@apache.org
> > > > > > >>>>>>
> > > > > > >>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> In this particular case: please consider the PR
> > as
> > > a
> > > > > > >>>>> proposal.
> > > > > > >>>>>>>>>>>>> Don't
> > > > > > >>>>>>>>>>>>>>> feel
> > > > > > >>>>>>>>>>>>>>>>> like just because there is code there that
> takes
> > a
> > > > > > >>> certain
> > > > > > >>>>>>>>>>>>> approach,
> > > > > > >>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>> the approach is somehow sacred. I had to
> > implement
> > > > > > >>>>> something
> > > > > > >>>>>> to
> > > > > > >>>>>>>>>>>>>>>> crystallize
> > > > > > >>>>>>>>>>>>>>>>> my own thinking about how the problem could be
> > > > > > >>> approached.
> > > > > > >>>>> I
> > > > > > >>>>>>>>>>>> won't
> > > > > > >>>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>>>>>> disappointed if, as a community, we decide a
> > > > different
> > > > > > >>>>>>>>>>> direction
> > > > > > >>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>>> better
> > > > > > >>>>>>>>>>>>>>>>> and the code all gets thrown away. That's one
> of
> > > the
> > > > > > >>>>> reasons
> > > > > > >>>>>>>>>>>> that I
> > > > > > >>>>>>>>>>>>>>>> removed
> > > > > > >>>>>>>>>>>>>>>>> the 0.14.0 milestone that was added to the
> patch.
> > > (I
> > > > > > >>> don't
> > > > > > >>>>>> want
> > > > > > >>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>> rush
> > > > > > >>>>>>>>>>>>>>>> it,
> > > > > > >>>>>>>>>>>>>>>>> nor do I think that's a good idea.)
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> In general: Sounds like we could do with some
> > more
> > > > > > >>>>>>>>>>> formalization
> > > > > > >>>>>>>>>>>>>> around
> > > > > > >>>>>>>>>>>>>>>>> what a proposal looks like, which sorts of
> > changes
> > > > need
> > > > > > >>>>> one,
> > > > > > >>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>> when
> > > > > > >>>>>>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>>>>>> the dev cycle it is appropriate. FWIW I think
> > > Kafka's
> > > > > > >>>>> process
> > > > > > >>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>> more
> > > > > > >>>>>>>>>>>>>>> or
> > > > > > >>>>>>>>>>>>>>>>> less fine, and would be okay with adopting it
> for
> > > > Druid
> > > > > > >>> if
> > > > > > >>>>>>>>>>> people
> > > > > > >>>>>>>>>>>>>> like
> > > > > > >>>>>>>>>>>>>>>> it.
> > > > > > >>>>>>>>>>>>>>>>> Right now our standards for what requires a
> > "design
> > > > > > >>> review"
> > > > > > >>>>>> are
> > > > > > >>>>>>>>>>>>> very
> > > > > > >>>>>>>>>>>>>>>>> similar to the Kafka community standards for
> what
> > > > > > >>> requires
> > > > > > >>>>> a
> > > > > > >>>>>>>>>>> KIP,
> > > > > > >>>>>>>>>>>>> so
> > > > > > >>>>>>>>>>>>>> we
> > > > > > >>>>>>>>>>>>>>>>> have some familiarity with those concepts.
> > However
> > > we
> > > > > > >>> don't
> > > > > > >>>>>>>>>>>>> separate
> > > > > > >>>>>>>>>>>>>> PR
> > > > > > >>>>>>>>>>>>>>>>> review and proposal discussion as strictly as
> > they
> > > > do,
> > > > > > >>>>> which
> > > > > > >>>>>>>>>>>> seems
> > > > > > >>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>>>>>> the foundation for the feeling of exclusion
> that
> > is
> > > > > > >> being
> > > > > > >>>>> felt
> > > > > > >>>>>>>>>>>>> here.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Separately: I just redid the description on
> > > > > > >>>>>>>>>>>>>>>>>
> > > https://github.com/apache/incubator-druid/pull/6794
> > > > to
> > > > > > >>> be
> > > > > > >>>>>> more
> > > > > > >>>>>>>>>>>>>>>> proposal-y.
> > > > > > >>>>>>>>>>>>>>>>> I followed the KIP style:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > >>>>>>>>>>>>>>>> .
> > > > > > >>>>>>>>>>>>>>>>> Please refresh the page and see if it looks
> more
> > > > > > >> useful.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Gian
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <
> > > > > > >>>>> jhyde@apache.org
> > > > > > >>>>>>>
> > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Slim,
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> I agree with your points that offline
> > development
> > > is
> > > > > > >> bad
> > > > > > >>>>> for
> > > > > > >>>>>>>>>>>>>>> community.
> > > > > > >>>>>>>>>>>>>>>>>> But I don’t think you need much mentor help.
> You
> > > > have
> > > > > > >>>>> raised
> > > > > > >>>>>>>>>>>> valid
> > > > > > >>>>>>>>>>>>>>>> issues
> > > > > > >>>>>>>>>>>>>>>>>> and the Druid community needs to decide what
> its
> > > > > > >>>>> development
> > > > > > >>>>>>>>>>>>>> practices
> > > > > > >>>>>>>>>>>>>>>>>> should be.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Julian
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
> > > > > > >>>>>>>>>>> bslim@apache.org>
> > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Hello everyone and hope you all have very
> good
> > > > > > >>> holidays.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> First, this email is not directed on the
> author
> > > or
> > > > > > >> the
> > > > > > >>> PR
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > https://github.com/apache/incubator-druid/pull/6794
> > > > > > >>> it
> > > > > > >>>>>>>>>>> self,
> > > > > > >>>>>>>>>>>>> but
> > > > > > >>>>>>>>>>>>>> i
> > > > > > >>>>>>>>>>>>>>>> see
> > > > > > >>>>>>>>>>>>>>>>>>> this PR as a perfect example.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> One of the foundation of Apache Way or what i
> > > would
> > > > > > >>>>> simply
> > > > > > >>>>>>>>>>> call
> > > > > > >>>>>>>>>>>>>> open
> > > > > > >>>>>>>>>>>>>>>>>> source
> > > > > > >>>>>>>>>>>>>>>>>>> community driven development is that
> "Technical
> > > > > > >>> decisions
> > > > > > >>>>>> are
> > > > > > >>>>>>>>>>>>>>>> discussed,
> > > > > > >>>>>>>>>>>>>>>>>>> decided, and archived publicly.
> > > > > > >>>>>>>>>>>>>>>>>>> developpement"
> > > > > > >>>>>>>>>>>>>>>>>>> Which means that big technical  changes such
> as
> > > the
> > > > > > >> one
> > > > > > >>>>>>>>>>> brought
> > > > > > >>>>>>>>>>>>> by
> > > > > > >>>>>>>>>>>>>>>> #/6794
> > > > > > >>>>>>>>>>>>>>>>>>> should have started as a proposal and round
> of
> > > > > > >>>>> discussions
> > > > > > >>>>>>>>>>>> about
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>> major
> > > > > > >>>>>>>>>>>>>>>>>>> changes designs not as 11K line of code.
> > > > > > >>>>>>>>>>>>>>>>>>> I believe such openness will promote a lot of
> > > good
> > > > > > >>>>> benefits
> > > > > > >>>>>>>>>>>> such
> > > > > > >>>>>>>>>>>>>> as:
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> - ensures community health and growth.
> > > > > > >>>>>>>>>>>>>>>>>>> - ensures everyone can participate not only
> the
> > > > > > >> authors
> > > > > > >>>>> and
> > > > > > >>>>>>>>>>> his
> > > > > > >>>>>>>>>>>>>>>>>> co-workers.
> > > > > > >>>>>>>>>>>>>>>>>>> - ensures that the project is driven by the
> > > > community
> > > > > > >>> and
> > > > > > >>>>>>>>>>> not a
> > > > > > >>>>>>>>>>>>>> given
> > > > > > >>>>>>>>>>>>>>>>>>> company or an individual.
> > > > > > >>>>>>>>>>>>>>>>>>> - ensures that there is consensus (not saying
> > > 100%
> > > > > > >>>>>>>>>>> agreement;)
> > > > > > >>>>>>>>>>>>>>> however
> > > > > > >>>>>>>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>>>>>>>> means that all individuals will accept the
> > > current
> > > > > > >>>>> progress
> > > > > > >>>>>>>>>>> on
> > > > > > >>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>> project
> > > > > > >>>>>>>>>>>>>>>>>>> until some better proposal is put forth.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Personally such BIG offline PR makes me feel
> > > > excluded
> > > > > > >>> and
> > > > > > >>>>>>>>>>>> doesn't
> > > > > > >>>>>>>>>>>>>>> give
> > > > > > >>>>>>>>>>>>>>>>>> me a
> > > > > > >>>>>>>>>>>>>>>>>>> sense that i belong to  a community at all.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> To prevent such off list development i think
> > as a
> > > > > > >> Druid
> > > > > > >>>>>>>>>>>> Community
> > > > > > >>>>>>>>>>>>>> we
> > > > > > >>>>>>>>>>>>>>>> need
> > > > > > >>>>>>>>>>>>>>>>>>> to stick to the apache way “If it didn’t
> happen
> > > on
> > > > > > >> the
> > > > > > >>>>>>>>>>> mailing
> > > > > > >>>>>>>>>>>>>> list,
> > > > > > >>>>>>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>>>>>>>> didn’t happen.”
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> I would appreciate if some of the Apache
> mentor
> > > > help
> > > > > > >>> with
> > > > > > >>>>>>>>>>> this.
> > > > > > >>>>>>>>>>>>>>>>>>> Thanks
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>
> > > > ---------------------------------------------------------------------
> > > > > > >>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > > > > > >>> dev-unsubscribe@druid.apache.org
> > > > > > >>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > >>>>> dev-help@druid.apache.org
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>
> > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > > > > > >> dev-unsubscribe@druid.apache.org
> > > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > >>> dev-help@druid.apache.org
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> --
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> B-Slim
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>
> > > > > > >>>
> > > > > >
> > > >
> > _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>
> > > > >
> ---------------------------------------------------------------------
> > > > > > >>>>>>>>> To unsubscribe, e-mail:
> dev-unsubscribe@druid.apache.org
> > > > > > >>>>>>>>> For additional commands, e-mail:
> > dev-help@druid.apache.org
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>
> > > > ---------------------------------------------------------------------
> > > > > > >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > > > >>>>>>> For additional commands, e-mail:
> dev-help@druid.apache.org
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > > > For additional commands, e-mail: dev-help@druid.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Off list major development

[Jonathan Wei <w3...@gmail.com>]

The proposal template PR is here:
https://github.com/apache/incubator-druid/pull/7062

Thanks,
Jon

On Tue, Feb 12, 2019 at 3:20 PM Jonathan Wei <jo...@apache.org> wrote:

> Cool, I'll go ahead and make a template based on the previous discussion.
>
> Thanks,
> Jon
>
> On Tue, Feb 12, 2019 at 9:16 AM Slim Bouguerra <bs...@apache.org> wrote:
>
>> @Gian All the above looks good to me.
>> I think having a template will tremendously help current devs and new
>> contributors, and will have a tremendous effect on the project !
>> Thanks
>>
>> On Tue, Feb 12, 2019 at 9:00 AM Gian Merlino <gi...@apache.org> wrote:
>>
>> > Does anyone have thoughts on the above suggestions?
>> >
>> > On Fri, Feb 1, 2019 at 2:16 PM Gian Merlino <gi...@apache.org> wrote:
>> >
>> > > I think we should clarify the process too. Might I suggest,
>> > >
>> > > 1) Add a GitHub issue template with proposal headers and some
>> description
>> > > of what each section should be, so people can fill them in easily.
>> > > 2) Suggest that for any change that would need a design review per
>> > > http://druid.io/community/, the author also creates a proposal issue
>> > > following that template. It can be very short if the change is simple.
>> > The
>> > > design discussion should take place on the proposal issue, and the
>> code
>> > > review should take place on the PR. A +1 on either the issue or the PR
>> > > would be considered a +1 for the design, while only a +1 on the PR
>> would
>> > be
>> > > considered a +1 for the code itself.
>> > > 3) Update http://druid.io/community/ and our CONTRIBUTING.md with
>> > > guidance about (2) and encouraging that the proposal issues are
>> created
>> > > early in the dev cycle.
>> > >
>> > > I am thinking of "suggest" rather than "require" in (2) so we can
>> start
>> > > slow and see how we like this process before making it mandatory.
>> > >
>> > > On Fri, Feb 1, 2019 at 2:22 AM Clint Wylie <cl...@imply.io>
>> wrote:
>> > >
>> > >> +1 for proposal template.
>> > >>
>> > >> Do we also need to clarify the process that goes along with the
>> > proposals?
>> > >> (It seems clear to me we've reached consensus in wanting a proposal
>> > >> process, but less clear if we have a clear picture of or have reached
>> > >> consensus on the process itself). Things like when voting happens,
>> > >> appropriate PR timing, voting period, announcements to dev list,
>> > >> significance of silence (implicit +1 or -1?), etc. Even if just
>> adapting
>> > >> Kafka's I think it might be a good idea to lay it out in this thread.
>> > >>
>> > >> Beyond putting reference to this stuff in top level github readme
>> and on
>> > >> the website, is there anything more we should do to guide people that
>> > are
>> > >> thinking about contributing to use the proposal process?
>> > >>
>> > >> On Thu, Jan 31, 2019 at 2:47 PM Jonathan Wei <jo...@apache.org>
>> wrote:
>> > >>
>> > >> > That structure sounds good:
>> > >> > - expanding rejected alternatives to a broader rationale section
>> > sounds
>> > >> > good
>> > >> > - I like "operational impact" as suggested by Slim and Gian more
>> than
>> > >> the
>> > >> > corresponding KIP terminology
>> > >> > - Future work is a good addition
>> > >> >
>> > >> > For test plan, I don't have a very strong opinion on this, but I'm
>> > >> thinking
>> > >> > it could make sense as an optional section (if someone has one,
>> that's
>> > >> > cool, if not, that's cool too, but perhaps having it present in the
>> > >> > template would encourage ppl to think about testing strategies
>> early
>> > on
>> > >> if
>> > >> > they aren't already)
>> > >> >
>> > >> >
>> > >> > On Thu, Jan 31, 2019 at 2:17 PM Jihoon Son <ji...@apache.org>
>> > >> wrote:
>> > >> >
>> > >> > > Thanks Gian.
>> > >> > > The suggested template looks good to me.
>> > >> > >
>> > >> > > Jihoon
>> > >> > >
>> > >> > > On Thu, Jan 31, 2019 at 9:27 AM Gian Merlino <gi...@apache.org>
>> > wrote:
>> > >> > >
>> > >> > > > If it's not clear - I am agreeing with Jihoon and Slim that a
>> > >> separate
>> > >> > > > "Rationale" section makes sense in addition to a couple other
>> > >> suggested
>> > >> > > > tweaks.
>> > >> > > >
>> > >> > > > On Wed, Jan 30, 2019 at 3:46 PM Gian Merlino <gi...@apache.org>
>> > >> wrote:
>> > >> > > >
>> > >> > > > > I think it'd also be nice to tweak a couple parts of the KIP
>> > >> template
>> > >> > > > > (Motivation; Public Interfaces; Proposed Changes;
>> Compatibility,
>> > >> > > > > Deprecation, and Migration Plan; Test Plan; Rejected
>> > >> Alternatives). A
>> > >> > > > > couple people have suggested adding a "Rationale" section,
>> how
>> > >> about
>> > >> > > > adding
>> > >> > > > > that and removing "Rejected alternatives" -- rolling them in
>> > >> > together?
>> > >> > > > And
>> > >> > > > > dropping "test plan", since IMO that discussion can be
>> deferred
>> > to
>> > >> > the
>> > >> > > PR
>> > >> > > > > itself, when there is code ready. Finally, adding "future
>> work",
>> > >> > > > detailing
>> > >> > > > > where this change might lead us.
>> > >> > > > >
>> > >> > > > > So in particular the template I am suggesting would be
>> something
>> > >> like
>> > >> > > > this.
>> > >> > > > >
>> > >> > > > > 1) Motivation: A description of the problem.
>> > >> > > > > 2) Proposed changes: Should usually be the longest section.
>> > Should
>> > >> > > > include
>> > >> > > > > any changes that are proposed to user-facing interfaces
>> > >> > (configuration
>> > >> > > > > parameters, JSON query/ingest specs, SQL language, emitted
>> > >> metrics,
>> > >> > and
>> > >> > > > so
>> > >> > > > > on).
>> > >> > > > > 3) Rationale: A discussion of why this particular solution is
>> > the
>> > >> > best
>> > >> > > > > one. One good way to approach this is to discuss other
>> > alternative
>> > >> > > > > solutions that you considered and decided against. This
>> should
>> > >> also
>> > >> > > > include
>> > >> > > > > a discussion of any specific benefits or drawbacks you are
>> aware
>> > >> of.
>> > >> > > > > 4) Operational impact: Is anything going to be deprecated or
>> > >> removed
>> > >> > by
>> > >> > > > > this change? Is there a migration path that cluster operators
>> > >> need to
>> > >> > > be
>> > >> > > > > aware of? Will there be any effect on the ability to do a
>> > rolling
>> > >> > > > upgrade,
>> > >> > > > > or to do a rolling _downgrade_ if an operator wants to switch
>> > back
>> > >> > to a
>> > >> > > > > previous version?
>> > >> > > > > 5) Future work: A discussion of things that you believe are
>> out
>> > of
>> > >> > > scope
>> > >> > > > > for the particular proposal but would be nice follow-ups. It
>> > helps
>> > >> > show
>> > >> > > > > where a particular change could be leading us. There isn't
>> any
>> > >> > > commitment
>> > >> > > > > that the proposal author will actually work on this stuff.
>> It is
>> > >> okay
>> > >> > > if
>> > >> > > > > this section is empty.
>> > >> > > > >
>> > >> > > > > On Wed, Jan 30, 2019 at 3:14 PM Jihoon Son <
>> > jihoonson@apache.org>
>> > >> > > wrote:
>> > >> > > > >
>> > >> > > > >> Thanks Eyal and Jon for starting the discussion about
>> making a
>> > >> > > template!
>> > >> > > > >>
>> > >> > > > >> The KIP template looks good, but I would like to add one
>> more.
>> > >> > > > >> The current template is:
>> > >> > > > >>
>> > >> > > > >> - Motivation
>> > >> > > > >> - Public Interfaces
>> > >> > > > >> - Proposed Changes
>> > >> > > > >> - Compatibility, Deprecation, and Migration Plan
>> > >> > > > >> - Test Plan
>> > >> > > > >> - Rejected Alternatives
>> > >> > > > >>
>> > >> > > > >> It includes almost everything required for proposals, but I
>> > think
>> > >> > it's
>> > >> > > > >> missing why the author chose the proposed changes.
>> > >> > > > >> So, I think it would be great if we can add 'Rationale' or
>> > >> 'Expected
>> > >> > > > >> benefits and drawbacks'.
>> > >> > > > >> People might include it by themselves in 'Motivation' or
>> > >> 'Proposed
>> > >> > > > >> Changes', but it would be good if there's an explicit
>> section
>> > to
>> > >> > > > describe
>> > >> > > > >> it.
>> > >> > > > >>
>> > >> > > > >> Best,
>> > >> > > > >> Jihoon
>> > >> > > > >>
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>

Re: Off list major development

[Jonathan Wei <jo...@apache.org>]

Cool, I'll go ahead and make a template based on the previous discussion.

Thanks,
Jon

On Tue, Feb 12, 2019 at 9:16 AM Slim Bouguerra <bs...@apache.org> wrote:

> @Gian All the above looks good to me.
> I think having a template will tremendously help current devs and new
> contributors, and will have a tremendous effect on the project !
> Thanks
>
> On Tue, Feb 12, 2019 at 9:00 AM Gian Merlino <gi...@apache.org> wrote:
>
> > Does anyone have thoughts on the above suggestions?
> >
> > On Fri, Feb 1, 2019 at 2:16 PM Gian Merlino <gi...@apache.org> wrote:
> >
> > > I think we should clarify the process too. Might I suggest,
> > >
> > > 1) Add a GitHub issue template with proposal headers and some
> description
> > > of what each section should be, so people can fill them in easily.
> > > 2) Suggest that for any change that would need a design review per
> > > http://druid.io/community/, the author also creates a proposal issue
> > > following that template. It can be very short if the change is simple.
> > The
> > > design discussion should take place on the proposal issue, and the code
> > > review should take place on the PR. A +1 on either the issue or the PR
> > > would be considered a +1 for the design, while only a +1 on the PR
> would
> > be
> > > considered a +1 for the code itself.
> > > 3) Update http://druid.io/community/ and our CONTRIBUTING.md with
> > > guidance about (2) and encouraging that the proposal issues are created
> > > early in the dev cycle.
> > >
> > > I am thinking of "suggest" rather than "require" in (2) so we can start
> > > slow and see how we like this process before making it mandatory.
> > >
> > > On Fri, Feb 1, 2019 at 2:22 AM Clint Wylie <cl...@imply.io>
> wrote:
> > >
> > >> +1 for proposal template.
> > >>
> > >> Do we also need to clarify the process that goes along with the
> > proposals?
> > >> (It seems clear to me we've reached consensus in wanting a proposal
> > >> process, but less clear if we have a clear picture of or have reached
> > >> consensus on the process itself). Things like when voting happens,
> > >> appropriate PR timing, voting period, announcements to dev list,
> > >> significance of silence (implicit +1 or -1?), etc. Even if just
> adapting
> > >> Kafka's I think it might be a good idea to lay it out in this thread.
> > >>
> > >> Beyond putting reference to this stuff in top level github readme and
> on
> > >> the website, is there anything more we should do to guide people that
> > are
> > >> thinking about contributing to use the proposal process?
> > >>
> > >> On Thu, Jan 31, 2019 at 2:47 PM Jonathan Wei <jo...@apache.org>
> wrote:
> > >>
> > >> > That structure sounds good:
> > >> > - expanding rejected alternatives to a broader rationale section
> > sounds
> > >> > good
> > >> > - I like "operational impact" as suggested by Slim and Gian more
> than
> > >> the
> > >> > corresponding KIP terminology
> > >> > - Future work is a good addition
> > >> >
> > >> > For test plan, I don't have a very strong opinion on this, but I'm
> > >> thinking
> > >> > it could make sense as an optional section (if someone has one,
> that's
> > >> > cool, if not, that's cool too, but perhaps having it present in the
> > >> > template would encourage ppl to think about testing strategies early
> > on
> > >> if
> > >> > they aren't already)
> > >> >
> > >> >
> > >> > On Thu, Jan 31, 2019 at 2:17 PM Jihoon Son <ji...@apache.org>
> > >> wrote:
> > >> >
> > >> > > Thanks Gian.
> > >> > > The suggested template looks good to me.
> > >> > >
> > >> > > Jihoon
> > >> > >
> > >> > > On Thu, Jan 31, 2019 at 9:27 AM Gian Merlino <gi...@apache.org>
> > wrote:
> > >> > >
> > >> > > > If it's not clear - I am agreeing with Jihoon and Slim that a
> > >> separate
> > >> > > > "Rationale" section makes sense in addition to a couple other
> > >> suggested
> > >> > > > tweaks.
> > >> > > >
> > >> > > > On Wed, Jan 30, 2019 at 3:46 PM Gian Merlino <gi...@apache.org>
> > >> wrote:
> > >> > > >
> > >> > > > > I think it'd also be nice to tweak a couple parts of the KIP
> > >> template
> > >> > > > > (Motivation; Public Interfaces; Proposed Changes;
> Compatibility,
> > >> > > > > Deprecation, and Migration Plan; Test Plan; Rejected
> > >> Alternatives). A
> > >> > > > > couple people have suggested adding a "Rationale" section, how
> > >> about
> > >> > > > adding
> > >> > > > > that and removing "Rejected alternatives" -- rolling them in
> > >> > together?
> > >> > > > And
> > >> > > > > dropping "test plan", since IMO that discussion can be
> deferred
> > to
> > >> > the
> > >> > > PR
> > >> > > > > itself, when there is code ready. Finally, adding "future
> work",
> > >> > > > detailing
> > >> > > > > where this change might lead us.
> > >> > > > >
> > >> > > > > So in particular the template I am suggesting would be
> something
> > >> like
> > >> > > > this.
> > >> > > > >
> > >> > > > > 1) Motivation: A description of the problem.
> > >> > > > > 2) Proposed changes: Should usually be the longest section.
> > Should
> > >> > > > include
> > >> > > > > any changes that are proposed to user-facing interfaces
> > >> > (configuration
> > >> > > > > parameters, JSON query/ingest specs, SQL language, emitted
> > >> metrics,
> > >> > and
> > >> > > > so
> > >> > > > > on).
> > >> > > > > 3) Rationale: A discussion of why this particular solution is
> > the
> > >> > best
> > >> > > > > one. One good way to approach this is to discuss other
> > alternative
> > >> > > > > solutions that you considered and decided against. This should
> > >> also
> > >> > > > include
> > >> > > > > a discussion of any specific benefits or drawbacks you are
> aware
> > >> of.
> > >> > > > > 4) Operational impact: Is anything going to be deprecated or
> > >> removed
> > >> > by
> > >> > > > > this change? Is there a migration path that cluster operators
> > >> need to
> > >> > > be
> > >> > > > > aware of? Will there be any effect on the ability to do a
> > rolling
> > >> > > > upgrade,
> > >> > > > > or to do a rolling _downgrade_ if an operator wants to switch
> > back
> > >> > to a
> > >> > > > > previous version?
> > >> > > > > 5) Future work: A discussion of things that you believe are
> out
> > of
> > >> > > scope
> > >> > > > > for the particular proposal but would be nice follow-ups. It
> > helps
> > >> > show
> > >> > > > > where a particular change could be leading us. There isn't any
> > >> > > commitment
> > >> > > > > that the proposal author will actually work on this stuff. It
> is
> > >> okay
> > >> > > if
> > >> > > > > this section is empty.
> > >> > > > >
> > >> > > > > On Wed, Jan 30, 2019 at 3:14 PM Jihoon Son <
> > jihoonson@apache.org>
> > >> > > wrote:
> > >> > > > >
> > >> > > > >> Thanks Eyal and Jon for starting the discussion about making
> a
> > >> > > template!
> > >> > > > >>
> > >> > > > >> The KIP template looks good, but I would like to add one
> more.
> > >> > > > >> The current template is:
> > >> > > > >>
> > >> > > > >> - Motivation
> > >> > > > >> - Public Interfaces
> > >> > > > >> - Proposed Changes
> > >> > > > >> - Compatibility, Deprecation, and Migration Plan
> > >> > > > >> - Test Plan
> > >> > > > >> - Rejected Alternatives
> > >> > > > >>
> > >> > > > >> It includes almost everything required for proposals, but I
> > think
> > >> > it's
> > >> > > > >> missing why the author chose the proposed changes.
> > >> > > > >> So, I think it would be great if we can add 'Rationale' or
> > >> 'Expected
> > >> > > > >> benefits and drawbacks'.
> > >> > > > >> People might include it by themselves in 'Motivation' or
> > >> 'Proposed
> > >> > > > >> Changes', but it would be good if there's an explicit section
> > to
> > >> > > > describe
> > >> > > > >> it.
> > >> > > > >>
> > >> > > > >> Best,
> > >> > > > >> Jihoon
> > >> > > > >>
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: Off list major development

[Slim Bouguerra <bs...@apache.org>]

@Gian All the above looks good to me.
I think having a template will tremendously help current devs and new
contributors, and will have a tremendous effect on the project !
Thanks

On Tue, Feb 12, 2019 at 9:00 AM Gian Merlino <gi...@apache.org> wrote:

> Does anyone have thoughts on the above suggestions?
>
> On Fri, Feb 1, 2019 at 2:16 PM Gian Merlino <gi...@apache.org> wrote:
>
> > I think we should clarify the process too. Might I suggest,
> >
> > 1) Add a GitHub issue template with proposal headers and some description
> > of what each section should be, so people can fill them in easily.
> > 2) Suggest that for any change that would need a design review per
> > http://druid.io/community/, the author also creates a proposal issue
> > following that template. It can be very short if the change is simple.
> The
> > design discussion should take place on the proposal issue, and the code
> > review should take place on the PR. A +1 on either the issue or the PR
> > would be considered a +1 for the design, while only a +1 on the PR would
> be
> > considered a +1 for the code itself.
> > 3) Update http://druid.io/community/ and our CONTRIBUTING.md with
> > guidance about (2) and encouraging that the proposal issues are created
> > early in the dev cycle.
> >
> > I am thinking of "suggest" rather than "require" in (2) so we can start
> > slow and see how we like this process before making it mandatory.
> >
> > On Fri, Feb 1, 2019 at 2:22 AM Clint Wylie <cl...@imply.io> wrote:
> >
> >> +1 for proposal template.
> >>
> >> Do we also need to clarify the process that goes along with the
> proposals?
> >> (It seems clear to me we've reached consensus in wanting a proposal
> >> process, but less clear if we have a clear picture of or have reached
> >> consensus on the process itself). Things like when voting happens,
> >> appropriate PR timing, voting period, announcements to dev list,
> >> significance of silence (implicit +1 or -1?), etc. Even if just adapting
> >> Kafka's I think it might be a good idea to lay it out in this thread.
> >>
> >> Beyond putting reference to this stuff in top level github readme and on
> >> the website, is there anything more we should do to guide people that
> are
> >> thinking about contributing to use the proposal process?
> >>
> >> On Thu, Jan 31, 2019 at 2:47 PM Jonathan Wei <jo...@apache.org> wrote:
> >>
> >> > That structure sounds good:
> >> > - expanding rejected alternatives to a broader rationale section
> sounds
> >> > good
> >> > - I like "operational impact" as suggested by Slim and Gian more than
> >> the
> >> > corresponding KIP terminology
> >> > - Future work is a good addition
> >> >
> >> > For test plan, I don't have a very strong opinion on this, but I'm
> >> thinking
> >> > it could make sense as an optional section (if someone has one, that's
> >> > cool, if not, that's cool too, but perhaps having it present in the
> >> > template would encourage ppl to think about testing strategies early
> on
> >> if
> >> > they aren't already)
> >> >
> >> >
> >> > On Thu, Jan 31, 2019 at 2:17 PM Jihoon Son <ji...@apache.org>
> >> wrote:
> >> >
> >> > > Thanks Gian.
> >> > > The suggested template looks good to me.
> >> > >
> >> > > Jihoon
> >> > >
> >> > > On Thu, Jan 31, 2019 at 9:27 AM Gian Merlino <gi...@apache.org>
> wrote:
> >> > >
> >> > > > If it's not clear - I am agreeing with Jihoon and Slim that a
> >> separate
> >> > > > "Rationale" section makes sense in addition to a couple other
> >> suggested
> >> > > > tweaks.
> >> > > >
> >> > > > On Wed, Jan 30, 2019 at 3:46 PM Gian Merlino <gi...@apache.org>
> >> wrote:
> >> > > >
> >> > > > > I think it'd also be nice to tweak a couple parts of the KIP
> >> template
> >> > > > > (Motivation; Public Interfaces; Proposed Changes; Compatibility,
> >> > > > > Deprecation, and Migration Plan; Test Plan; Rejected
> >> Alternatives). A
> >> > > > > couple people have suggested adding a "Rationale" section, how
> >> about
> >> > > > adding
> >> > > > > that and removing "Rejected alternatives" -- rolling them in
> >> > together?
> >> > > > And
> >> > > > > dropping "test plan", since IMO that discussion can be deferred
> to
> >> > the
> >> > > PR
> >> > > > > itself, when there is code ready. Finally, adding "future work",
> >> > > > detailing
> >> > > > > where this change might lead us.
> >> > > > >
> >> > > > > So in particular the template I am suggesting would be something
> >> like
> >> > > > this.
> >> > > > >
> >> > > > > 1) Motivation: A description of the problem.
> >> > > > > 2) Proposed changes: Should usually be the longest section.
> Should
> >> > > > include
> >> > > > > any changes that are proposed to user-facing interfaces
> >> > (configuration
> >> > > > > parameters, JSON query/ingest specs, SQL language, emitted
> >> metrics,
> >> > and
> >> > > > so
> >> > > > > on).
> >> > > > > 3) Rationale: A discussion of why this particular solution is
> the
> >> > best
> >> > > > > one. One good way to approach this is to discuss other
> alternative
> >> > > > > solutions that you considered and decided against. This should
> >> also
> >> > > > include
> >> > > > > a discussion of any specific benefits or drawbacks you are aware
> >> of.
> >> > > > > 4) Operational impact: Is anything going to be deprecated or
> >> removed
> >> > by
> >> > > > > this change? Is there a migration path that cluster operators
> >> need to
> >> > > be
> >> > > > > aware of? Will there be any effect on the ability to do a
> rolling
> >> > > > upgrade,
> >> > > > > or to do a rolling _downgrade_ if an operator wants to switch
> back
> >> > to a
> >> > > > > previous version?
> >> > > > > 5) Future work: A discussion of things that you believe are out
> of
> >> > > scope
> >> > > > > for the particular proposal but would be nice follow-ups. It
> helps
> >> > show
> >> > > > > where a particular change could be leading us. There isn't any
> >> > > commitment
> >> > > > > that the proposal author will actually work on this stuff. It is
> >> okay
> >> > > if
> >> > > > > this section is empty.
> >> > > > >
> >> > > > > On Wed, Jan 30, 2019 at 3:14 PM Jihoon Son <
> jihoonson@apache.org>
> >> > > wrote:
> >> > > > >
> >> > > > >> Thanks Eyal and Jon for starting the discussion about making a
> >> > > template!
> >> > > > >>
> >> > > > >> The KIP template looks good, but I would like to add one more.
> >> > > > >> The current template is:
> >> > > > >>
> >> > > > >> - Motivation
> >> > > > >> - Public Interfaces
> >> > > > >> - Proposed Changes
> >> > > > >> - Compatibility, Deprecation, and Migration Plan
> >> > > > >> - Test Plan
> >> > > > >> - Rejected Alternatives
> >> > > > >>
> >> > > > >> It includes almost everything required for proposals, but I
> think
> >> > it's
> >> > > > >> missing why the author chose the proposed changes.
> >> > > > >> So, I think it would be great if we can add 'Rationale' or
> >> 'Expected
> >> > > > >> benefits and drawbacks'.
> >> > > > >> People might include it by themselves in 'Motivation' or
> >> 'Proposed
> >> > > > >> Changes', but it would be good if there's an explicit section
> to
> >> > > > describe
> >> > > > >> it.
> >> > > > >>
> >> > > > >> Best,
> >> > > > >> Jihoon
> >> > > > >>
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: Off list major development

[Gian Merlino <gi...@apache.org>]

Does anyone have thoughts on the above suggestions?

On Fri, Feb 1, 2019 at 2:16 PM Gian Merlino <gi...@apache.org> wrote:

> I think we should clarify the process too. Might I suggest,
>
> 1) Add a GitHub issue template with proposal headers and some description
> of what each section should be, so people can fill them in easily.
> 2) Suggest that for any change that would need a design review per
> http://druid.io/community/, the author also creates a proposal issue
> following that template. It can be very short if the change is simple. The
> design discussion should take place on the proposal issue, and the code
> review should take place on the PR. A +1 on either the issue or the PR
> would be considered a +1 for the design, while only a +1 on the PR would be
> considered a +1 for the code itself.
> 3) Update http://druid.io/community/ and our CONTRIBUTING.md with
> guidance about (2) and encouraging that the proposal issues are created
> early in the dev cycle.
>
> I am thinking of "suggest" rather than "require" in (2) so we can start
> slow and see how we like this process before making it mandatory.
>
> On Fri, Feb 1, 2019 at 2:22 AM Clint Wylie <cl...@imply.io> wrote:
>
>> +1 for proposal template.
>>
>> Do we also need to clarify the process that goes along with the proposals?
>> (It seems clear to me we've reached consensus in wanting a proposal
>> process, but less clear if we have a clear picture of or have reached
>> consensus on the process itself). Things like when voting happens,
>> appropriate PR timing, voting period, announcements to dev list,
>> significance of silence (implicit +1 or -1?), etc. Even if just adapting
>> Kafka's I think it might be a good idea to lay it out in this thread.
>>
>> Beyond putting reference to this stuff in top level github readme and on
>> the website, is there anything more we should do to guide people that are
>> thinking about contributing to use the proposal process?
>>
>> On Thu, Jan 31, 2019 at 2:47 PM Jonathan Wei <jo...@apache.org> wrote:
>>
>> > That structure sounds good:
>> > - expanding rejected alternatives to a broader rationale section sounds
>> > good
>> > - I like "operational impact" as suggested by Slim and Gian more than
>> the
>> > corresponding KIP terminology
>> > - Future work is a good addition
>> >
>> > For test plan, I don't have a very strong opinion on this, but I'm
>> thinking
>> > it could make sense as an optional section (if someone has one, that's
>> > cool, if not, that's cool too, but perhaps having it present in the
>> > template would encourage ppl to think about testing strategies early on
>> if
>> > they aren't already)
>> >
>> >
>> > On Thu, Jan 31, 2019 at 2:17 PM Jihoon Son <ji...@apache.org>
>> wrote:
>> >
>> > > Thanks Gian.
>> > > The suggested template looks good to me.
>> > >
>> > > Jihoon
>> > >
>> > > On Thu, Jan 31, 2019 at 9:27 AM Gian Merlino <gi...@apache.org> wrote:
>> > >
>> > > > If it's not clear - I am agreeing with Jihoon and Slim that a
>> separate
>> > > > "Rationale" section makes sense in addition to a couple other
>> suggested
>> > > > tweaks.
>> > > >
>> > > > On Wed, Jan 30, 2019 at 3:46 PM Gian Merlino <gi...@apache.org>
>> wrote:
>> > > >
>> > > > > I think it'd also be nice to tweak a couple parts of the KIP
>> template
>> > > > > (Motivation; Public Interfaces; Proposed Changes; Compatibility,
>> > > > > Deprecation, and Migration Plan; Test Plan; Rejected
>> Alternatives). A
>> > > > > couple people have suggested adding a "Rationale" section, how
>> about
>> > > > adding
>> > > > > that and removing "Rejected alternatives" -- rolling them in
>> > together?
>> > > > And
>> > > > > dropping "test plan", since IMO that discussion can be deferred to
>> > the
>> > > PR
>> > > > > itself, when there is code ready. Finally, adding "future work",
>> > > > detailing
>> > > > > where this change might lead us.
>> > > > >
>> > > > > So in particular the template I am suggesting would be something
>> like
>> > > > this.
>> > > > >
>> > > > > 1) Motivation: A description of the problem.
>> > > > > 2) Proposed changes: Should usually be the longest section. Should
>> > > > include
>> > > > > any changes that are proposed to user-facing interfaces
>> > (configuration
>> > > > > parameters, JSON query/ingest specs, SQL language, emitted
>> metrics,
>> > and
>> > > > so
>> > > > > on).
>> > > > > 3) Rationale: A discussion of why this particular solution is the
>> > best
>> > > > > one. One good way to approach this is to discuss other alternative
>> > > > > solutions that you considered and decided against. This should
>> also
>> > > > include
>> > > > > a discussion of any specific benefits or drawbacks you are aware
>> of.
>> > > > > 4) Operational impact: Is anything going to be deprecated or
>> removed
>> > by
>> > > > > this change? Is there a migration path that cluster operators
>> need to
>> > > be
>> > > > > aware of? Will there be any effect on the ability to do a rolling
>> > > > upgrade,
>> > > > > or to do a rolling _downgrade_ if an operator wants to switch back
>> > to a
>> > > > > previous version?
>> > > > > 5) Future work: A discussion of things that you believe are out of
>> > > scope
>> > > > > for the particular proposal but would be nice follow-ups. It helps
>> > show
>> > > > > where a particular change could be leading us. There isn't any
>> > > commitment
>> > > > > that the proposal author will actually work on this stuff. It is
>> okay
>> > > if
>> > > > > this section is empty.
>> > > > >
>> > > > > On Wed, Jan 30, 2019 at 3:14 PM Jihoon Son <ji...@apache.org>
>> > > wrote:
>> > > > >
>> > > > >> Thanks Eyal and Jon for starting the discussion about making a
>> > > template!
>> > > > >>
>> > > > >> The KIP template looks good, but I would like to add one more.
>> > > > >> The current template is:
>> > > > >>
>> > > > >> - Motivation
>> > > > >> - Public Interfaces
>> > > > >> - Proposed Changes
>> > > > >> - Compatibility, Deprecation, and Migration Plan
>> > > > >> - Test Plan
>> > > > >> - Rejected Alternatives
>> > > > >>
>> > > > >> It includes almost everything required for proposals, but I think
>> > it's
>> > > > >> missing why the author chose the proposed changes.
>> > > > >> So, I think it would be great if we can add 'Rationale' or
>> 'Expected
>> > > > >> benefits and drawbacks'.
>> > > > >> People might include it by themselves in 'Motivation' or
>> 'Proposed
>> > > > >> Changes', but it would be good if there's an explicit section to
>> > > > describe
>> > > > >> it.
>> > > > >>
>> > > > >> Best,
>> > > > >> Jihoon
>> > > > >>
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: Off list major development

[Gian Merlino <gi...@apache.org>]

I think we should clarify the process too. Might I suggest,

1) Add a GitHub issue template with proposal headers and some description
of what each section should be, so people can fill them in easily.
2) Suggest that for any change that would need a design review per
http://druid.io/community/, the author also creates a proposal issue
following that template. It can be very short if the change is simple. The
design discussion should take place on the proposal issue, and the code
review should take place on the PR. A +1 on either the issue or the PR
would be considered a +1 for the design, while only a +1 on the PR would be
considered a +1 for the code itself.
3) Update http://druid.io/community/ and our CONTRIBUTING.md with guidance
about (2) and encouraging that the proposal issues are created early in the
dev cycle.

I am thinking of "suggest" rather than "require" in (2) so we can start
slow and see how we like this process before making it mandatory.

On Fri, Feb 1, 2019 at 2:22 AM Clint Wylie <cl...@imply.io> wrote:

> +1 for proposal template.
>
> Do we also need to clarify the process that goes along with the proposals?
> (It seems clear to me we've reached consensus in wanting a proposal
> process, but less clear if we have a clear picture of or have reached
> consensus on the process itself). Things like when voting happens,
> appropriate PR timing, voting period, announcements to dev list,
> significance of silence (implicit +1 or -1?), etc. Even if just adapting
> Kafka's I think it might be a good idea to lay it out in this thread.
>
> Beyond putting reference to this stuff in top level github readme and on
> the website, is there anything more we should do to guide people that are
> thinking about contributing to use the proposal process?
>
> On Thu, Jan 31, 2019 at 2:47 PM Jonathan Wei <jo...@apache.org> wrote:
>
> > That structure sounds good:
> > - expanding rejected alternatives to a broader rationale section sounds
> > good
> > - I like "operational impact" as suggested by Slim and Gian more than the
> > corresponding KIP terminology
> > - Future work is a good addition
> >
> > For test plan, I don't have a very strong opinion on this, but I'm
> thinking
> > it could make sense as an optional section (if someone has one, that's
> > cool, if not, that's cool too, but perhaps having it present in the
> > template would encourage ppl to think about testing strategies early on
> if
> > they aren't already)
> >
> >
> > On Thu, Jan 31, 2019 at 2:17 PM Jihoon Son <ji...@apache.org> wrote:
> >
> > > Thanks Gian.
> > > The suggested template looks good to me.
> > >
> > > Jihoon
> > >
> > > On Thu, Jan 31, 2019 at 9:27 AM Gian Merlino <gi...@apache.org> wrote:
> > >
> > > > If it's not clear - I am agreeing with Jihoon and Slim that a
> separate
> > > > "Rationale" section makes sense in addition to a couple other
> suggested
> > > > tweaks.
> > > >
> > > > On Wed, Jan 30, 2019 at 3:46 PM Gian Merlino <gi...@apache.org>
> wrote:
> > > >
> > > > > I think it'd also be nice to tweak a couple parts of the KIP
> template
> > > > > (Motivation; Public Interfaces; Proposed Changes; Compatibility,
> > > > > Deprecation, and Migration Plan; Test Plan; Rejected
> Alternatives). A
> > > > > couple people have suggested adding a "Rationale" section, how
> about
> > > > adding
> > > > > that and removing "Rejected alternatives" -- rolling them in
> > together?
> > > > And
> > > > > dropping "test plan", since IMO that discussion can be deferred to
> > the
> > > PR
> > > > > itself, when there is code ready. Finally, adding "future work",
> > > > detailing
> > > > > where this change might lead us.
> > > > >
> > > > > So in particular the template I am suggesting would be something
> like
> > > > this.
> > > > >
> > > > > 1) Motivation: A description of the problem.
> > > > > 2) Proposed changes: Should usually be the longest section. Should
> > > > include
> > > > > any changes that are proposed to user-facing interfaces
> > (configuration
> > > > > parameters, JSON query/ingest specs, SQL language, emitted metrics,
> > and
> > > > so
> > > > > on).
> > > > > 3) Rationale: A discussion of why this particular solution is the
> > best
> > > > > one. One good way to approach this is to discuss other alternative
> > > > > solutions that you considered and decided against. This should also
> > > > include
> > > > > a discussion of any specific benefits or drawbacks you are aware
> of.
> > > > > 4) Operational impact: Is anything going to be deprecated or
> removed
> > by
> > > > > this change? Is there a migration path that cluster operators need
> to
> > > be
> > > > > aware of? Will there be any effect on the ability to do a rolling
> > > > upgrade,
> > > > > or to do a rolling _downgrade_ if an operator wants to switch back
> > to a
> > > > > previous version?
> > > > > 5) Future work: A discussion of things that you believe are out of
> > > scope
> > > > > for the particular proposal but would be nice follow-ups. It helps
> > show
> > > > > where a particular change could be leading us. There isn't any
> > > commitment
> > > > > that the proposal author will actually work on this stuff. It is
> okay
> > > if
> > > > > this section is empty.
> > > > >
> > > > > On Wed, Jan 30, 2019 at 3:14 PM Jihoon Son <ji...@apache.org>
> > > wrote:
> > > > >
> > > > >> Thanks Eyal and Jon for starting the discussion about making a
> > > template!
> > > > >>
> > > > >> The KIP template looks good, but I would like to add one more.
> > > > >> The current template is:
> > > > >>
> > > > >> - Motivation
> > > > >> - Public Interfaces
> > > > >> - Proposed Changes
> > > > >> - Compatibility, Deprecation, and Migration Plan
> > > > >> - Test Plan
> > > > >> - Rejected Alternatives
> > > > >>
> > > > >> It includes almost everything required for proposals, but I think
> > it's
> > > > >> missing why the author chose the proposed changes.
> > > > >> So, I think it would be great if we can add 'Rationale' or
> 'Expected
> > > > >> benefits and drawbacks'.
> > > > >> People might include it by themselves in 'Motivation' or 'Proposed
> > > > >> Changes', but it would be good if there's an explicit section to
> > > > describe
> > > > >> it.
> > > > >>
> > > > >> Best,
> > > > >> Jihoon
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Off list major development

[Clint Wylie <cl...@imply.io>]

+1 for proposal template.

Do we also need to clarify the process that goes along with the proposals?
(It seems clear to me we've reached consensus in wanting a proposal
process, but less clear if we have a clear picture of or have reached
consensus on the process itself). Things like when voting happens,
appropriate PR timing, voting period, announcements to dev list,
significance of silence (implicit +1 or -1?), etc. Even if just adapting
Kafka's I think it might be a good idea to lay it out in this thread.

Beyond putting reference to this stuff in top level github readme and on
the website, is there anything more we should do to guide people that are
thinking about contributing to use the proposal process?

On Thu, Jan 31, 2019 at 2:47 PM Jonathan Wei <jo...@apache.org> wrote:

> That structure sounds good:
> - expanding rejected alternatives to a broader rationale section sounds
> good
> - I like "operational impact" as suggested by Slim and Gian more than the
> corresponding KIP terminology
> - Future work is a good addition
>
> For test plan, I don't have a very strong opinion on this, but I'm thinking
> it could make sense as an optional section (if someone has one, that's
> cool, if not, that's cool too, but perhaps having it present in the
> template would encourage ppl to think about testing strategies early on if
> they aren't already)
>
>
> On Thu, Jan 31, 2019 at 2:17 PM Jihoon Son <ji...@apache.org> wrote:
>
> > Thanks Gian.
> > The suggested template looks good to me.
> >
> > Jihoon
> >
> > On Thu, Jan 31, 2019 at 9:27 AM Gian Merlino <gi...@apache.org> wrote:
> >
> > > If it's not clear - I am agreeing with Jihoon and Slim that a separate
> > > "Rationale" section makes sense in addition to a couple other suggested
> > > tweaks.
> > >
> > > On Wed, Jan 30, 2019 at 3:46 PM Gian Merlino <gi...@apache.org> wrote:
> > >
> > > > I think it'd also be nice to tweak a couple parts of the KIP template
> > > > (Motivation; Public Interfaces; Proposed Changes; Compatibility,
> > > > Deprecation, and Migration Plan; Test Plan; Rejected Alternatives). A
> > > > couple people have suggested adding a "Rationale" section, how about
> > > adding
> > > > that and removing "Rejected alternatives" -- rolling them in
> together?
> > > And
> > > > dropping "test plan", since IMO that discussion can be deferred to
> the
> > PR
> > > > itself, when there is code ready. Finally, adding "future work",
> > > detailing
> > > > where this change might lead us.
> > > >
> > > > So in particular the template I am suggesting would be something like
> > > this.
> > > >
> > > > 1) Motivation: A description of the problem.
> > > > 2) Proposed changes: Should usually be the longest section. Should
> > > include
> > > > any changes that are proposed to user-facing interfaces
> (configuration
> > > > parameters, JSON query/ingest specs, SQL language, emitted metrics,
> and
> > > so
> > > > on).
> > > > 3) Rationale: A discussion of why this particular solution is the
> best
> > > > one. One good way to approach this is to discuss other alternative
> > > > solutions that you considered and decided against. This should also
> > > include
> > > > a discussion of any specific benefits or drawbacks you are aware of.
> > > > 4) Operational impact: Is anything going to be deprecated or removed
> by
> > > > this change? Is there a migration path that cluster operators need to
> > be
> > > > aware of? Will there be any effect on the ability to do a rolling
> > > upgrade,
> > > > or to do a rolling _downgrade_ if an operator wants to switch back
> to a
> > > > previous version?
> > > > 5) Future work: A discussion of things that you believe are out of
> > scope
> > > > for the particular proposal but would be nice follow-ups. It helps
> show
> > > > where a particular change could be leading us. There isn't any
> > commitment
> > > > that the proposal author will actually work on this stuff. It is okay
> > if
> > > > this section is empty.
> > > >
> > > > On Wed, Jan 30, 2019 at 3:14 PM Jihoon Son <ji...@apache.org>
> > wrote:
> > > >
> > > >> Thanks Eyal and Jon for starting the discussion about making a
> > template!
> > > >>
> > > >> The KIP template looks good, but I would like to add one more.
> > > >> The current template is:
> > > >>
> > > >> - Motivation
> > > >> - Public Interfaces
> > > >> - Proposed Changes
> > > >> - Compatibility, Deprecation, and Migration Plan
> > > >> - Test Plan
> > > >> - Rejected Alternatives
> > > >>
> > > >> It includes almost everything required for proposals, but I think
> it's
> > > >> missing why the author chose the proposed changes.
> > > >> So, I think it would be great if we can add 'Rationale' or 'Expected
> > > >> benefits and drawbacks'.
> > > >> People might include it by themselves in 'Motivation' or 'Proposed
> > > >> Changes', but it would be good if there's an explicit section to
> > > describe
> > > >> it.
> > > >>
> > > >> Best,
> > > >> Jihoon
> > > >>
> > > >
> > >
> >
>

Re: Off list major development

[Gian Merlino <gi...@apache.org>]

Sure, I can see the value of putting it in the template to encourage
thinking about it. Sounds good to me.

On Thu, Jan 31, 2019 at 2:47 PM Jonathan Wei <jo...@apache.org> wrote:

> That structure sounds good:
> - expanding rejected alternatives to a broader rationale section sounds
> good
> - I like "operational impact" as suggested by Slim and Gian more than the
> corresponding KIP terminology
> - Future work is a good addition
>
> For test plan, I don't have a very strong opinion on this, but I'm thinking
> it could make sense as an optional section (if someone has one, that's
> cool, if not, that's cool too, but perhaps having it present in the
> template would encourage ppl to think about testing strategies early on if
> they aren't already)
>
>
> On Thu, Jan 31, 2019 at 2:17 PM Jihoon Son <ji...@apache.org> wrote:
>
> > Thanks Gian.
> > The suggested template looks good to me.
> >
> > Jihoon
> >
> > On Thu, Jan 31, 2019 at 9:27 AM Gian Merlino <gi...@apache.org> wrote:
> >
> > > If it's not clear - I am agreeing with Jihoon and Slim that a separate
> > > "Rationale" section makes sense in addition to a couple other suggested
> > > tweaks.
> > >
> > > On Wed, Jan 30, 2019 at 3:46 PM Gian Merlino <gi...@apache.org> wrote:
> > >
> > > > I think it'd also be nice to tweak a couple parts of the KIP template
> > > > (Motivation; Public Interfaces; Proposed Changes; Compatibility,
> > > > Deprecation, and Migration Plan; Test Plan; Rejected Alternatives). A
> > > > couple people have suggested adding a "Rationale" section, how about
> > > adding
> > > > that and removing "Rejected alternatives" -- rolling them in
> together?
> > > And
> > > > dropping "test plan", since IMO that discussion can be deferred to
> the
> > PR
> > > > itself, when there is code ready. Finally, adding "future work",
> > > detailing
> > > > where this change might lead us.
> > > >
> > > > So in particular the template I am suggesting would be something like
> > > this.
> > > >
> > > > 1) Motivation: A description of the problem.
> > > > 2) Proposed changes: Should usually be the longest section. Should
> > > include
> > > > any changes that are proposed to user-facing interfaces
> (configuration
> > > > parameters, JSON query/ingest specs, SQL language, emitted metrics,
> and
> > > so
> > > > on).
> > > > 3) Rationale: A discussion of why this particular solution is the
> best
> > > > one. One good way to approach this is to discuss other alternative
> > > > solutions that you considered and decided against. This should also
> > > include
> > > > a discussion of any specific benefits or drawbacks you are aware of.
> > > > 4) Operational impact: Is anything going to be deprecated or removed
> by
> > > > this change? Is there a migration path that cluster operators need to
> > be
> > > > aware of? Will there be any effect on the ability to do a rolling
> > > upgrade,
> > > > or to do a rolling _downgrade_ if an operator wants to switch back
> to a
> > > > previous version?
> > > > 5) Future work: A discussion of things that you believe are out of
> > scope
> > > > for the particular proposal but would be nice follow-ups. It helps
> show
> > > > where a particular change could be leading us. There isn't any
> > commitment
> > > > that the proposal author will actually work on this stuff. It is okay
> > if
> > > > this section is empty.
> > > >
> > > > On Wed, Jan 30, 2019 at 3:14 PM Jihoon Son <ji...@apache.org>
> > wrote:
> > > >
> > > >> Thanks Eyal and Jon for starting the discussion about making a
> > template!
> > > >>
> > > >> The KIP template looks good, but I would like to add one more.
> > > >> The current template is:
> > > >>
> > > >> - Motivation
> > > >> - Public Interfaces
> > > >> - Proposed Changes
> > > >> - Compatibility, Deprecation, and Migration Plan
> > > >> - Test Plan
> > > >> - Rejected Alternatives
> > > >>
> > > >> It includes almost everything required for proposals, but I think
> it's
> > > >> missing why the author chose the proposed changes.
> > > >> So, I think it would be great if we can add 'Rationale' or 'Expected
> > > >> benefits and drawbacks'.
> > > >> People might include it by themselves in 'Motivation' or 'Proposed
> > > >> Changes', but it would be good if there's an explicit section to
> > > describe
> > > >> it.
> > > >>
> > > >> Best,
> > > >> Jihoon
> > > >>
> > > >
> > >
> >
>

Re: Off list major development

[Jonathan Wei <jo...@apache.org>]

That structure sounds good:
- expanding rejected alternatives to a broader rationale section sounds good
- I like "operational impact" as suggested by Slim and Gian more than the
corresponding KIP terminology
- Future work is a good addition

For test plan, I don't have a very strong opinion on this, but I'm thinking
it could make sense as an optional section (if someone has one, that's
cool, if not, that's cool too, but perhaps having it present in the
template would encourage ppl to think about testing strategies early on if
they aren't already)


On Thu, Jan 31, 2019 at 2:17 PM Jihoon Son <ji...@apache.org> wrote:

> Thanks Gian.
> The suggested template looks good to me.
>
> Jihoon
>
> On Thu, Jan 31, 2019 at 9:27 AM Gian Merlino <gi...@apache.org> wrote:
>
> > If it's not clear - I am agreeing with Jihoon and Slim that a separate
> > "Rationale" section makes sense in addition to a couple other suggested
> > tweaks.
> >
> > On Wed, Jan 30, 2019 at 3:46 PM Gian Merlino <gi...@apache.org> wrote:
> >
> > > I think it'd also be nice to tweak a couple parts of the KIP template
> > > (Motivation; Public Interfaces; Proposed Changes; Compatibility,
> > > Deprecation, and Migration Plan; Test Plan; Rejected Alternatives). A
> > > couple people have suggested adding a "Rationale" section, how about
> > adding
> > > that and removing "Rejected alternatives" -- rolling them in together?
> > And
> > > dropping "test plan", since IMO that discussion can be deferred to the
> PR
> > > itself, when there is code ready. Finally, adding "future work",
> > detailing
> > > where this change might lead us.
> > >
> > > So in particular the template I am suggesting would be something like
> > this.
> > >
> > > 1) Motivation: A description of the problem.
> > > 2) Proposed changes: Should usually be the longest section. Should
> > include
> > > any changes that are proposed to user-facing interfaces (configuration
> > > parameters, JSON query/ingest specs, SQL language, emitted metrics, and
> > so
> > > on).
> > > 3) Rationale: A discussion of why this particular solution is the best
> > > one. One good way to approach this is to discuss other alternative
> > > solutions that you considered and decided against. This should also
> > include
> > > a discussion of any specific benefits or drawbacks you are aware of.
> > > 4) Operational impact: Is anything going to be deprecated or removed by
> > > this change? Is there a migration path that cluster operators need to
> be
> > > aware of? Will there be any effect on the ability to do a rolling
> > upgrade,
> > > or to do a rolling _downgrade_ if an operator wants to switch back to a
> > > previous version?
> > > 5) Future work: A discussion of things that you believe are out of
> scope
> > > for the particular proposal but would be nice follow-ups. It helps show
> > > where a particular change could be leading us. There isn't any
> commitment
> > > that the proposal author will actually work on this stuff. It is okay
> if
> > > this section is empty.
> > >
> > > On Wed, Jan 30, 2019 at 3:14 PM Jihoon Son <ji...@apache.org>
> wrote:
> > >
> > >> Thanks Eyal and Jon for starting the discussion about making a
> template!
> > >>
> > >> The KIP template looks good, but I would like to add one more.
> > >> The current template is:
> > >>
> > >> - Motivation
> > >> - Public Interfaces
> > >> - Proposed Changes
> > >> - Compatibility, Deprecation, and Migration Plan
> > >> - Test Plan
> > >> - Rejected Alternatives
> > >>
> > >> It includes almost everything required for proposals, but I think it's
> > >> missing why the author chose the proposed changes.
> > >> So, I think it would be great if we can add 'Rationale' or 'Expected
> > >> benefits and drawbacks'.
> > >> People might include it by themselves in 'Motivation' or 'Proposed
> > >> Changes', but it would be good if there's an explicit section to
> > describe
> > >> it.
> > >>
> > >> Best,
> > >> Jihoon
> > >>
> > >
> >
>

Re: Off list major development

[Jihoon Son <ji...@apache.org>]

Thanks Gian.
The suggested template looks good to me.

Jihoon

On Thu, Jan 31, 2019 at 9:27 AM Gian Merlino <gi...@apache.org> wrote:

> If it's not clear - I am agreeing with Jihoon and Slim that a separate
> "Rationale" section makes sense in addition to a couple other suggested
> tweaks.
>
> On Wed, Jan 30, 2019 at 3:46 PM Gian Merlino <gi...@apache.org> wrote:
>
> > I think it'd also be nice to tweak a couple parts of the KIP template
> > (Motivation; Public Interfaces; Proposed Changes; Compatibility,
> > Deprecation, and Migration Plan; Test Plan; Rejected Alternatives). A
> > couple people have suggested adding a "Rationale" section, how about
> adding
> > that and removing "Rejected alternatives" -- rolling them in together?
> And
> > dropping "test plan", since IMO that discussion can be deferred to the PR
> > itself, when there is code ready. Finally, adding "future work",
> detailing
> > where this change might lead us.
> >
> > So in particular the template I am suggesting would be something like
> this.
> >
> > 1) Motivation: A description of the problem.
> > 2) Proposed changes: Should usually be the longest section. Should
> include
> > any changes that are proposed to user-facing interfaces (configuration
> > parameters, JSON query/ingest specs, SQL language, emitted metrics, and
> so
> > on).
> > 3) Rationale: A discussion of why this particular solution is the best
> > one. One good way to approach this is to discuss other alternative
> > solutions that you considered and decided against. This should also
> include
> > a discussion of any specific benefits or drawbacks you are aware of.
> > 4) Operational impact: Is anything going to be deprecated or removed by
> > this change? Is there a migration path that cluster operators need to be
> > aware of? Will there be any effect on the ability to do a rolling
> upgrade,
> > or to do a rolling _downgrade_ if an operator wants to switch back to a
> > previous version?
> > 5) Future work: A discussion of things that you believe are out of scope
> > for the particular proposal but would be nice follow-ups. It helps show
> > where a particular change could be leading us. There isn't any commitment
> > that the proposal author will actually work on this stuff. It is okay if
> > this section is empty.
> >
> > On Wed, Jan 30, 2019 at 3:14 PM Jihoon Son <ji...@apache.org> wrote:
> >
> >> Thanks Eyal and Jon for starting the discussion about making a template!
> >>
> >> The KIP template looks good, but I would like to add one more.
> >> The current template is:
> >>
> >> - Motivation
> >> - Public Interfaces
> >> - Proposed Changes
> >> - Compatibility, Deprecation, and Migration Plan
> >> - Test Plan
> >> - Rejected Alternatives
> >>
> >> It includes almost everything required for proposals, but I think it's
> >> missing why the author chose the proposed changes.
> >> So, I think it would be great if we can add 'Rationale' or 'Expected
> >> benefits and drawbacks'.
> >> People might include it by themselves in 'Motivation' or 'Proposed
> >> Changes', but it would be good if there's an explicit section to
> describe
> >> it.
> >>
> >> Best,
> >> Jihoon
> >>
> >
>

Re: Off list major development

[Gian Merlino <gi...@apache.org>]

If it's not clear - I am agreeing with Jihoon and Slim that a separate
"Rationale" section makes sense in addition to a couple other suggested
tweaks.

On Wed, Jan 30, 2019 at 3:46 PM Gian Merlino <gi...@apache.org> wrote:

> I think it'd also be nice to tweak a couple parts of the KIP template
> (Motivation; Public Interfaces; Proposed Changes; Compatibility,
> Deprecation, and Migration Plan; Test Plan; Rejected Alternatives). A
> couple people have suggested adding a "Rationale" section, how about adding
> that and removing "Rejected alternatives" -- rolling them in together? And
> dropping "test plan", since IMO that discussion can be deferred to the PR
> itself, when there is code ready. Finally, adding "future work", detailing
> where this change might lead us.
>
> So in particular the template I am suggesting would be something like this.
>
> 1) Motivation: A description of the problem.
> 2) Proposed changes: Should usually be the longest section. Should include
> any changes that are proposed to user-facing interfaces (configuration
> parameters, JSON query/ingest specs, SQL language, emitted metrics, and so
> on).
> 3) Rationale: A discussion of why this particular solution is the best
> one. One good way to approach this is to discuss other alternative
> solutions that you considered and decided against. This should also include
> a discussion of any specific benefits or drawbacks you are aware of.
> 4) Operational impact: Is anything going to be deprecated or removed by
> this change? Is there a migration path that cluster operators need to be
> aware of? Will there be any effect on the ability to do a rolling upgrade,
> or to do a rolling _downgrade_ if an operator wants to switch back to a
> previous version?
> 5) Future work: A discussion of things that you believe are out of scope
> for the particular proposal but would be nice follow-ups. It helps show
> where a particular change could be leading us. There isn't any commitment
> that the proposal author will actually work on this stuff. It is okay if
> this section is empty.
>
> On Wed, Jan 30, 2019 at 3:14 PM Jihoon Son <ji...@apache.org> wrote:
>
>> Thanks Eyal and Jon for starting the discussion about making a template!
>>
>> The KIP template looks good, but I would like to add one more.
>> The current template is:
>>
>> - Motivation
>> - Public Interfaces
>> - Proposed Changes
>> - Compatibility, Deprecation, and Migration Plan
>> - Test Plan
>> - Rejected Alternatives
>>
>> It includes almost everything required for proposals, but I think it's
>> missing why the author chose the proposed changes.
>> So, I think it would be great if we can add 'Rationale' or 'Expected
>> benefits and drawbacks'.
>> People might include it by themselves in 'Motivation' or 'Proposed
>> Changes', but it would be good if there's an explicit section to describe
>> it.
>>
>> Best,
>> Jihoon
>>
>

Re: Off list major development

[Gian Merlino <gi...@apache.org>]

I think it'd also be nice to tweak a couple parts of the KIP template
(Motivation; Public Interfaces; Proposed Changes; Compatibility,
Deprecation, and Migration Plan; Test Plan; Rejected Alternatives). A
couple people have suggested adding a "Rationale" section, how about adding
that and removing "Rejected alternatives" -- rolling them in together? And
dropping "test plan", since IMO that discussion can be deferred to the PR
itself, when there is code ready. Finally, adding "future work", detailing
where this change might lead us.

So in particular the template I am suggesting would be something like this.

1) Motivation: A description of the problem.
2) Proposed changes: Should usually be the longest section. Should include
any changes that are proposed to user-facing interfaces (configuration
parameters, JSON query/ingest specs, SQL language, emitted metrics, and so
on).
3) Rationale: A discussion of why this particular solution is the best one.
One good way to approach this is to discuss other alternative solutions
that you considered and decided against. This should also include a
discussion of any specific benefits or drawbacks you are aware of.
4) Operational impact: Is anything going to be deprecated or removed by
this change? Is there a migration path that cluster operators need to be
aware of? Will there be any effect on the ability to do a rolling upgrade,
or to do a rolling _downgrade_ if an operator wants to switch back to a
previous version?
5) Future work: A discussion of things that you believe are out of scope
for the particular proposal but would be nice follow-ups. It helps show
where a particular change could be leading us. There isn't any commitment
that the proposal author will actually work on this stuff. It is okay if
this section is empty.

On Wed, Jan 30, 2019 at 3:14 PM Jihoon Son <ji...@apache.org> wrote:

> Thanks Eyal and Jon for starting the discussion about making a template!
>
> The KIP template looks good, but I would like to add one more.
> The current template is:
>
> - Motivation
> - Public Interfaces
> - Proposed Changes
> - Compatibility, Deprecation, and Migration Plan
> - Test Plan
> - Rejected Alternatives
>
> It includes almost everything required for proposals, but I think it's
> missing why the author chose the proposed changes.
> So, I think it would be great if we can add 'Rationale' or 'Expected
> benefits and drawbacks'.
> People might include it by themselves in 'Motivation' or 'Proposed
> Changes', but it would be good if there's an explicit section to describe
> it.
>
> Best,
> Jihoon
>

Re: Off list major development

[Jihoon Son <ji...@apache.org>]

Thanks Eyal and Jon for starting the discussion about making a template!

The KIP template looks good, but I would like to add one more.
The current template is:

- Motivation
- Public Interfaces
- Proposed Changes
- Compatibility, Deprecation, and Migration Plan
- Test Plan
- Rejected Alternatives

It includes almost everything required for proposals, but I think it's
missing why the author chose the proposed changes.
So, I think it would be great if we can add 'Rationale' or 'Expected
benefits and drawbacks'.
People might include it by themselves in 'Motivation' or 'Proposed
Changes', but it would be good if there's an explicit section to describe
it.

Best,
Jihoon

On Wed, Jan 30, 2019 at 11:22 AM Jonathan Wei <jo...@apache.org> wrote:

> Hi all,
>
> An issue has been opened by a community member suggesting that we create a
> template for proposals:
> https://github.com/apache/incubator-druid/issues/6949
>
> Having a template sounds convenient, and based on the discussion in this
> thread, I'm suggesting we adopt something based on the Kafka proposal
> format.
>
> I'm planning on creating such a template if there are no objections or
> alternative suggestions, so please take a look if you have thoughts on
> this.
>
> Thanks,
> Jon
>
> On Tue, Jan 15, 2019 at 12:07 PM Jihoon Son <gh...@gmail.com> wrote:
>
> > Good point.
> > If some authors raise PRs without noticing the need for a proposal, we
> > shouldn't ask them to close their PRs only because of the absence of the
> > proposal.
> >
> > "Design review" without a proposal for simple PRs would be good if we can
> > determine well what PRs need and what don't.
> > But, how do we know? Even for the same PR, someone may think it needs a
> > proposal but another may not.
> >
> > If someone don't notice the need for a proposal and raise a PR without
> it,
> > I'm fine with that.
> > However, we should still encourage writing a proposal before writing code
> > because we can avoid unnecessary effort.
> >
> > I think this kind of issue usually happens for first time contributors
> and
> > they will be better once they get used to Druid development.
> > And I believe someday even first contributors would follow this policy
> once
> > it gets settled down well in the community as Kafka community does.
> >
> > Jihoon
> >
> > On Tue, Jan 15, 2019 at 4:31 AM Roman Leventov <le...@gmail.com>
> > wrote:
> >
> > > In such small PRs, authors likely won't be aware that they need to
> > create a
> > > proposal in the first place. The first reviewer just adds the "Design
> > > Review" tag. It's also absolutely not about considering designs and
> > gauging
> > > the proposal, it's just verifying that a configuration / parameter /
> HTTP
> > > endpoint name is reasonable and aligned with the rest of Druid. So I
> > think
> > > that a separate proposal issue for such PRs is unnecessary bureaucracy.
> > >
> > > On Tue, 15 Jan 2019 at 07:45, Jihoon Son <gh...@gmail.com> wrote:
> > >
> > > > Roman,
> > > >
> > > > > Jihoon in
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> > > > and later Gian in this thread suggested that _every_ piece of work
> that
> > > > should be labelled as "Design Review" according to the current rules
> > > should
> > > > be accompanied by an issue. I don't agree with this, there are some
> PRs
> > > as
> > > > small as a few dozens of lines of code, that add some configuration
> > > > parameter and therefore should be labelled "Design Review". I don't
> > > thing a
> > > > separate proposal issue is needed for them, and even for a little
> > larger
> > > > PRs too.
> > > >
> > > > What I'm concerned with is how people feel if their design is not
> > > accepted
> > > > even though they wrote code. Of course, as Clint said, sometimes code
> > > helps
> > > > better understanding of the proposal. But, I believe this is the case
> > > when
> > > > the proposal is quite complicated and not easy to understand without
> > > code.
> > > > Also the authors should be aware of that they might rewrite the
> entire
> > > code
> > > > if the design should be changed.
> > > >
> > > > If writing code is simple, I don't see why the authors don't wait
> until
> > > the
> > > > review for their proposal is finished.
> > > >
> > > > Jihoon
> > > >
> > > > On Fri, Jan 11, 2019 at 9:51 AM Fangjin Yang <fa...@imply.io>
> wrote:
> > > >
> > > > > I agree with Gian, as an Apache committer, your responsibility is
> for
> > > the
> > > > > betterment of the project. I agree it is in the best interest of
> the
> > > > > project to stop thinking about what orgs people belong to. We are
> > all a
> > > > > part of the Apache software foundation, regardless of what our
> roles
> > > and
> > > > > titles are outside of it.
> > > > >
> > > > > On Fri, Jan 11, 2019 at 2:22 AM Roman Leventov <
> > leventov.ru@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > It's not that people from one org could abuse the project and
> push
> > > some
> > > > > > change, but that they have similar perspective (bubble effect)
> and
> > > some
> > > > > > important aspects of a large feature could escape their
> attention.
> > > > > >
> > > > > > I suggest it to be not a rigid rule, but a recommendation for
> > authors
> > > > of
> > > > > > large proposals to try to attract reviewers from other orgs.
> > > > > >
> > > > > > On Fri, 11 Jan 2019 at 02:51, Julian Hyde <jh...@apache.org>
> > wrote:
> > > > > >
> > > > > > > I agree with Gian.
> > > > > > >
> > > > > > > As an Apache committer, you only have one affiliation: you are
> > > > working
> > > > > in
> > > > > > > the best interests of the project.
> > > > > > >
> > > > > > > Obviously, in the real world there are other pressures. But we
> do
> > > our
> > > > > > best
> > > > > > > to compensate for them.
> > > > > > >
> > > > > > > Also, as a a community we try to design our process so as to
> > avoid
> > > > > undue
> > > > > > > influences. For instance, when I advocate for logging cases
> > early,
> > > I
> > > > am
> > > > > > > trying to mitigate the effect of product managers and VPs of
> > > > > engineering,
> > > > > > > who like to have their say in meeting rooms rather than on
> public
> > > > > mailing
> > > > > > > lists. That’s just one example; if we see other influences at
> > play,
> > > > > let’s
> > > > > > > evolve our process to try to level the playing field.
> > > > > > >
> > > > > > > Julian
> > > > > > >
> > > > > > >
> > > > > > > > On Jan 10, 2019, at 10:40 AM, Gian Merlino <gi...@apache.org>
> > > > wrote:
> > > > > > > >
> > > > > > > >>> What do community members think about also making a
> > requirement
> > > > > that
> > > > > > > >>> "Design Review" PRs and proposals are reviewed by at least
> > two
> > > > > people
> > > > > > > > with
> > > > > > > >>> different affiliation?
> > > > > > > >> This seems pretty reasonable to me. I haven't found anything
> > in
> > > > > Apache
> > > > > > > >> voting procedure docs (
> > > > > https://www.apache.org/foundation/voting.html)
> > > > > > > that
> > > > > > > >> seems to explicitly forbid something like this yet at least.
> > > > > > > >
> > > > > > > > On the other hand, my understanding of the Apache Way would
> > mean
> > > > that
> > > > > > > this
> > > > > > > > kind of rule doesn't make sense. In particular from
> > > > > > > > https://www.apache.org/foundation/how-it-works.html: "We
> > firmly
> > > > > > believe
> > > > > > > in
> > > > > > > > hats. Your role at the ASF is one assigned to you personally,
> > and
> > > > is
> > > > > > > > bestowed on you by your peers. It is not tied to your job or
> > > > current
> > > > > > > > employer or company." That sentiment seems incompatible with
> > > making
> > > > > > > > explicit rules about organizational diversity in voting. I
> have
> > > > also
> > > > > > > heard
> > > > > > > > a few people say things like: people are supposed to
> represent
> > > > > > > themselves,
> > > > > > > > not their employers.
> > > > > > > >
> > > > > > > > Obviously, though, people's actions and opinions are
> influenced
> > > by
> > > > > > their
> > > > > > > > employer. IMO, a better way to approach a potential problem
> > there
> > > > is
> > > > > > that
> > > > > > > > if people from a particular organization end up behaving
> > > abusively,
> > > > > > then
> > > > > > > > PMC members from other organizations (or in extreme
> situations
> > > the
> > > > > > Apache
> > > > > > > > board itself) should tell them to knock it off. And then do
> > more
> > > > > > serious
> > > > > > > > things if the inappropriate behavior continues. This kind
> > > > > > > >
> > > > > > > > The other thing you brought up, promoting more shared
> ownership
> > > and
> > > > > > > > awareness, I am hopeful that separating proposals from PRs
> will
> > > > help
> > > > > > with
> > > > > > > > that. One reason is that it takes much less time to
> understand
> > a
> > > > well
> > > > > > > > written proposal than it takes to understand a PR. Code is
> very
> > > > clear
> > > > > > but
> > > > > > > > it is also very verbose and takes a while to read and
> > understand.
> > > > So
> > > > > in
> > > > > > > > theory changing how we operate, in this way, should promote
> > more
> > > > > > > > understanding of more people across the code base.
> > > > > > > >
> > > > > > > > On Thu, Jan 10, 2019 at 1:34 AM Clint Wylie <
> > > clint.wylie@imply.io>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >>>
> > > > > > > >>> What do community members think about also making a
> > requirement
> > > > > that
> > > > > > > >>> "Design Review" PRs and proposals are reviewed by at least
> > two
> > > > > people
> > > > > > > >> with
> > > > > > > >>> different affiliation?
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> This seems pretty reasonable to me. I haven't found anything
> > in
> > > > > Apache
> > > > > > > >> voting procedure docs (
> > > > > https://www.apache.org/foundation/voting.html)
> > > > > > > that
> > > > > > > >> seems to explicitly forbid something like this yet at least.
> > > > > > > >>
> > > > > > > >> On Wed, Jan 9, 2019 at 8:52 PM Roman Leventov <
> > > > > leventov.ru@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> What do community members think about also making a
> > requirement
> > > > > that
> > > > > > > >>> "Design Review" PRs and proposals are reviewed by at least
> > two
> > > > > people
> > > > > > > >> with
> > > > > > > >>> different affiliation? IMO it's a good idea, because it
> > ensures
> > > > > that
> > > > > > > >>> different interests are taken into account. Also it pushes
> > > people
> > > > > to
> > > > > > > >> engage
> > > > > > > >>> with work done in other parts of Druid, improving shared
> code
> > > > > owning
> > > > > > > and
> > > > > > > >>> awareness.
> > > > > > > >>>
> > > > > > > >>> Or it's against the Apache Way because it's assumed that
> > there
> > > > are
> > > > > no
> > > > > > > >>> company boundaries within the community?
> > > > > > > >>>
> > > > > > > >>> On Thu, 10 Jan 2019 at 11:45, Roman Leventov <
> > > > > leventov.ru@gmail.com>
> > > > > > > >>> wrote:
> > > > > > > >>>
> > > > > > > >>>> I see two important reasons why it makes sense to file an
> > > issue
> > > > > and
> > > > > > > >>>> probably announce it in the mailing list, before writing a
> > lot
> > > > of
> > > > > > > code,
> > > > > > > >>>> despite not having a clear picture of what it will be and
> > any
> > > > > > > >> performance
> > > > > > > >>>> data:
> > > > > > > >>>> 1) somebody could already work on this problem privately
> in
> > > > > > parallel,
> > > > > > > >> it
> > > > > > > >>>> allows to avoid clash in people's works
> > > > > > > >>>> 2) some people could quickly think about the problem field
> > and
> > > > > share
> > > > > > > >>>> high-level ideas that could wildly change the direction in
> > > which
> > > > > the
> > > > > > > >>> author
> > > > > > > >>>> (the person who is going to write code) will move in his
> > work
> > > > from
> > > > > > > >> early
> > > > > > > >>> on.
> > > > > > > >>>>
> > > > > > > >>>> Jihoon in
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> > > > > > > >>> and
> > > > > > > >>>> later Gian in this thread suggested that _every_ piece of
> > work
> > > > > that
> > > > > > > >>> should
> > > > > > > >>>> be labelled as "Design Review" according to the current
> > rules
> > > > > should
> > > > > > > be
> > > > > > > >>>> accompanied by an issue. I don't agree with this, there
> are
> > > some
> > > > > PRs
> > > > > > > as
> > > > > > > >>>> small as a few dozens of lines of code, that add some
> > > > > configuration
> > > > > > > >>>> parameter and therefore should be labelled "Design
> Review".
> > I
> > > > > don't
> > > > > > > >>> thing a
> > > > > > > >>>> separate proposal issue is needed for them, and even for a
> > > > little
> > > > > > > >> larger
> > > > > > > >>>> PRs too.
> > > > > > > >>>>
> > > > > > > >>>> For the same reason, I also don't see the point of
> renaming
> > > > > "Design
> > > > > > > >>>> Review" into "Proposal", as well as separating "Design
> > Review"
> > > > > into
> > > > > > > >>>> "Proposal" and something like "API Review". I think a
> single
> > > > > "Design
> > > > > > > >>>> Review" tag handles it well.
> > > > > > > >>>>
> > > > > > > >>>> Gian mentioned an idea that PRs that follow a "Design
> > Review"
> > > > > > proposal
> > > > > > > >>>> issue shouldn't be "Design Review" themselves. I don't
> agree
> > > > with
> > > > > > > >> this, I
> > > > > > > >>>> think that actual code and performance data are important
> > > inputs
> > > > > > that
> > > > > > > >>>> should be re-evaluated at least by two people. I even
> think
> > > that
> > > > > > it's
> > > > > > > >>> very
> > > > > > > >>>> desirable that at least two people read _each line of
> > > production
> > > > > > code_
> > > > > > > >> in
> > > > > > > >>>> large PRs, although it's not what was done historically in
> > > > Druid,
> > > > > > > >> because
> > > > > > > >>>> large bodies of newly added code, with whole new classes
> and
> > > > > > > subsystems
> > > > > > > >>>> added, are also coincidentally tested worse than already
> > > > existing
> > > > > > > >> classes
> > > > > > > >>>> and subsystems, including in production. It seems to me
> that
> > > > those
> > > > > > > huge
> > > > > > > >>>> code influxes is a major source of bugs, that could later
> > take
> > > > > years
> > > > > > > to
> > > > > > > >>>> squeeze out from the codebase.
> > > > > > > >>>>
> > > > > > > >>>> On Wed, 9 Jan 2019 at 08:24, Clint Wylie <
> > > clint.wylie@imply.io>
> > > > > > > wrote:
> > > > > > > >>>>
> > > > > > > >>>>> Apologies for the delayed response.
> > > > > > > >>>>>
> > > > > > > >>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > > > > > > >>> slim.bouguerra@gmail.com>
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> I am wondering here what is the case where code first is
> > > > better?
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>> I don't think it's wrong to share ideas as early as
> > possible,
> > > > and
> > > > > > > >> after
> > > > > > > >>>>> this discussion I think I am in favor of it too. I just
> > meant
> > > > > that
> > > > > > I
> > > > > > > >>> don't
> > > > > > > >>>>> think it's always necessarily the most productive
> > discussion
> > > > > until
> > > > > > > >> code
> > > > > > > >>>>> exists sometimes, with the types of thing I am thinking
> of
> > > are
> > > > > > almost
> > > > > > > >>>>> entirely limited to cases where things might sound good
> to
> > > > anyone
> > > > > > on
> > > > > > > >>> paper
> > > > > > > >>>>> but in reality need a large amount of experiments
> conducted
> > > and
> > > > > > > >>>>> observations collected to determine that something is
> > > actually
> > > > > > worth
> > > > > > > >>>>> doing,
> > > > > > > >>>>> which I imagine is mainly things like reworking internals
> > for
> > > > > > > >>> performance
> > > > > > > >>>>> improvements.
> > > > > > > >>>>>
> > > > > > > >>>>> In the case of my combined proposal PR, I needed to prove
> > > that
> > > > > the
> > > > > > > >>> thing I
> > > > > > > >>>>> was working on was a good idea... and it wasn't directly.
> > > But I
> > > > > > came
> > > > > > > >> up
> > > > > > > >>>>> with another idea during the course of experiment turned
> > into
> > > > > > > >> something
> > > > > > > >>>>> compelling, so an initial proposal would have looked
> quite
> > a
> > > > lot
> > > > > > > >>> different
> > > > > > > >>>>> than what I ended up with. Once I had proven to myself
> that
> > > it
> > > > > was
> > > > > > a
> > > > > > > >>> good
> > > > > > > >>>>> idea, then I was comfortable sharing with the wider
> > > community.
> > > > > I'm
> > > > > > > not
> > > > > > > >>>>> certain how this would play out in an always proposal
> first
> > > > > model,
> > > > > > > >> maybe
> > > > > > > >>>>> the first proposal exists, I personally reject it after
> > > > updating
> > > > > > with
> > > > > > > >>>>> experiment results show it's a bad idea, continue
> > > experimenting
> > > > > and
> > > > > > > >>> raise
> > > > > > > >>>>> a
> > > > > > > >>>>> new one after the experiments start looking promising?
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>>> Let's not be naive this is very rare that a contributor
> > will
> > > > > > accept
> > > > > > > >>> that
> > > > > > > >>>>>> his work is to be thrown, usually devs takes coding as
> > > > personal
> > > > > > > >>> creation
> > > > > > > >>>>>> and they get attached to it.
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>> I agree, just because a handful of the committers have
> this
> > > > > > attitude,
> > > > > > > >> it
> > > > > > > >>>>> isn't fair to expect the wider community to also, that's
> > why
> > > I
> > > > am
> > > > > > in
> > > > > > > >>> favor
> > > > > > > >>>>> of formalizing the process.
> > > > > > > >>>>>
> > > > > > > >>>>> Can you please explain what is overbearing ? what can be
> > > > changed
> > > > > to
> > > > > > > >> make
> > > > > > > >>>>> it
> > > > > > > >>>>>> easy ?
> > > > > > > >>>>>> Most of the points are kind of the actual questions that
> > you
> > > > > want
> > > > > > to
> > > > > > > >>>>>> address before hand anyway isn't ?
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>> Sorry for the confusion, I said it's "not overbearing", I
> > > think
> > > > > > it's
> > > > > > > >>> fine.
> > > > > > > >>>>>
> > > > > > > >>>>> What are the complaints ?
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>> Is this and other previous threads not a complaint about
> > > > opening
> > > > > a
> > > > > > > >> large
> > > > > > > >>>>> PR
> > > > > > > >>>>> without a proposal? :) I just mean that formalizing the
> > > > process,
> > > > > > even
> > > > > > > >>> if a
> > > > > > > >>>>> proposal has a reference PR opened with it near
> > concurrently,
> > > > > could
> > > > > > > >>>>> prevent
> > > > > > > >>>>> these discussions from happening in the future because
> > ground
> > > > > rules
> > > > > > > >> have
> > > > > > > >>>>> been set and we are all on the same page I guess.
> > > > > > > >>>>>
> > > > > > > >>>>> I think we should also probably consider renaming the
> > "design
> > > > > > review"
> > > > > > > >>>>> label
> > > > > > > >>>>> to "proposal" or something to make it more clear that a
> PR
> > is
> > > > > > > >> associated
> > > > > > > >>>>> with a proposal. It might also be worth considering using
> > > > github
> > > > > > > >>> projects
> > > > > > > >>>>> for particularly large things that involve multiple
> follow
> > up
> > > > > PRs,
> > > > > > > >> but I
> > > > > > > >>>>> haven't used them in much depth to know if they add
> > anything.
> > > > > > > >>>>>
> > > > > > > >>>>> It seems like we are all converging on agreement to do a
> > > github
> > > > > > issue
> > > > > > > >>>>> proposal for larger changes (I would vote for announcing
> > them
> > > > on
> > > > > > the
> > > > > > > >> dev
> > > > > > > >>>>> list too for more visibility), so that design review is
> > > > separated
> > > > > > > from
> > > > > > > >>>>> code
> > > > > > > >>>>> review. I guess my main concern was not wanting to
> > discourage
> > > > > > > >>>>> experimentation by walling it off behind mandatory
> > > discussions,
> > > > > but
> > > > > > > >> the
> > > > > > > >>>>> more I think about it,  it doesn't technically change
> much
> > > > about
> > > > > > this
> > > > > > > >>>>> process, it just requires a more formal proposal to
> > accompany
> > > > any
> > > > > > > >>>>> experiments that are shared as a PR.
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>> On Tue, Jan 8, 2019 at 12:28 PM Gian Merlino <
> > > gian@apache.org>
> > > > > > > wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> I think for us, choosing to use GitHub issues as
> > discussion
> > > > > > threads
> > > > > > > >>> for
> > > > > > > >>>>>> potential 'major' contributions would be a good idea,
> > > > especially
> > > > > > if
> > > > > > > >> we
> > > > > > > >>>>>> encourage people to start them before PRs show up.
> > > Definitely
> > > > > > agree
> > > > > > > >>> that
> > > > > > > >>>>>> all contributors should go through the same process -- I
> > > > > couldn't
> > > > > > > >> see
> > > > > > > >>> it
> > > > > > > >>>>>> working well any other way.
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <
> > > jhyde@apache.org
> > > > >
> > > > > > > >> wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> Statically, yes, GitHub PRs are the same as GitHub
> cases.
> > > But
> > > > > > > >>>>>> dynamically,
> > > > > > > >>>>>>> they are different, because you can only log a PR when
> > you
> > > > have
> > > > > > > >>>>> finished
> > > > > > > >>>>>>> work.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> A lot of other Apache projects use JIRA, so there is a
> > > clear
> > > > > > > >>>>> distinction
> > > > > > > >>>>>>> between cases and contributions. JIRA cases, especially
> > > when
> > > > > > > >> logged
> > > > > > > >>>>> early
> > > > > > > >>>>>>> in the lifecycle of a contribution, become long-running
> > > > > > > >> conversation
> > > > > > > >>>>>>> threads with a lot of community participation. If the
> > Druid
> > > > > chose
> > > > > > > >> to
> > > > > > > >>>>> do
> > > > > > > >>>>>> so,
> > > > > > > >>>>>>> GitHub cases could be the same.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Be careful that you do not treat “potential
> contributors”
> > > (by
> > > > > > > >> which
> > > > > > > >>> I
> > > > > > > >>>>>>> presume you mean non-committers) differently from
> > > committers
> > > > > and
> > > > > > > >> PMC
> > > > > > > >>>>>>> members. Anyone starting a major piece of work should
> > > follow
> > > > > the
> > > > > > > >>> same
> > > > > > > >>>>>>> process. (Experienced committers probably have a
> somewhat
> > > > > better
> > > > > > > >>> idea
> > > > > > > >>>>>> what
> > > > > > > >>>>>>> work will turn out to be “major”, so they get a little
> > more
> > > > > > > >> leeway.)
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Julian
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> On Jan 7, 2019, at 12:10 PM, Gian Merlino <
> > > gian@apache.org>
> > > > > > > >>> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> I don't think there's a need to raise issues for every
> > > > > change: a
> > > > > > > >>>>> small
> > > > > > > >>>>>>> bug
> > > > > > > >>>>>>>> fix or doc fix should just go straight to PR. (GitHub
> > PRs
> > > > show
> > > > > > > >> up
> > > > > > > >>> as
> > > > > > > >>>>>>> issues
> > > > > > > >>>>>>>> in the issue-search UI/API, so it's not like this
> means
> > > the
> > > > > > > >> patch
> > > > > > > >>>>> has
> > > > > > > >>>>>> no
> > > > > > > >>>>>>>> corresponding issue -- in a sense the PR _is_ the
> > issue.)
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> I do think it makes sense to encourage potential
> > > > contributors
> > > > > to
> > > > > > > >>>>> write
> > > > > > > >>>>>> to
> > > > > > > >>>>>>>> the dev list or raise an issue if they aren't sure if
> > > > > something
> > > > > > > >>>>> would
> > > > > > > >>>>>>> need
> > > > > > > >>>>>>>> to go through a more heavy weight process.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Fwiw we do have a set of 'design review' criteria
> > already
> > > > (we
> > > > > > > >> had
> > > > > > > >>> a
> > > > > > > >>>>>>>> discussion about this a couple years ago) at:
> > > > > > > >>>>>>>>
> > http://druid.io/community/#getting-your-changes-accepted.
> > > > So
> > > > > we
> > > > > > > >>>>>>> wouldn't be
> > > > > > > >>>>>>>> starting from zero on defining that. We set it up back
> > > when
> > > > we
> > > > > > > >>> were
> > > > > > > >>>>>>> trying
> > > > > > > >>>>>>>> to _streamline_ our process -- we used to require two
> > > > > non-author
> > > > > > > >>> +1s
> > > > > > > >>>>>> for
> > > > > > > >>>>>>>> _every_ change, even minor ones. The introduction of
> > > design
> > > > > > > >> review
> > > > > > > >>>>>>> criteria
> > > > > > > >>>>>>>> was meant to classify which PRs need that level of
> > review
> > > > and
> > > > > > > >>> which
> > > > > > > >>>>>> ones
> > > > > > > >>>>>>>> are minor and can be merged with less review. I do
> think
> > > it
> > > > > > > >> helped
> > > > > > > >>>>> with
> > > > > > > >>>>>>>> getting minor PRs merged more quickly. The list of
> > > criteria
> > > > > is,
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> - Major architectural changes or API changes
> > > > > > > >>>>>>>> - HTTP requests and responses (e. g. a new HTTP
> > endpoint)
> > > > > > > >>>>>>>> - Interfaces for extensions
> > > > > > > >>>>>>>> - Server configuration (e. g. altering the behavior
> of a
> > > > > config
> > > > > > > >>>>>> property)
> > > > > > > >>>>>>>> - Emitted metrics
> > > > > > > >>>>>>>> - Other major changes, judged by the discretion of
> Druid
> > > > > > > >>> committers
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Some of it is subjective, but it has been in place
> for a
> > > > > while,
> > > > > > > >> so
> > > > > > > >>>>> it's
> > > > > > > >>>>>>> at
> > > > > > > >>>>>>>> least something we are relatively familiar with.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <
> > > > jhyde@apache.org
> > > > > >
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> Small contributions don’t need any design review,
> > whereas
> > > > > large
> > > > > > > >>>>>>>>> contributions need significant review. I don’t think
> we
> > > > > should
> > > > > > > >>>>> require
> > > > > > > >>>>>>> an
> > > > > > > >>>>>>>>> additional step for those (many) small contributions.
> > But
> > > > who
> > > > > > > >>>>> decides
> > > > > > > >>>>>>>>> whether a contribution fits into the small or large
> > > > category?
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> I think the solution is for authors to log a case (or
> > > send
> > > > an
> > > > > > > >>>>> email to
> > > > > > > >>>>>>>>> dev) before they start work on any contribution. Then
> > > > > > > >> committers
> > > > > > > >>>>> can
> > > > > > > >>>>>>>>> request a more heavy-weight process if they think it
> is
> > > > > needed.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Julian
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <
> > > > gian@apache.org>
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> It sounds like splitting design from code review is
> a
> > > > common
> > > > > > > >>> theme
> > > > > > > >>>>>> in a
> > > > > > > >>>>>>>>> few
> > > > > > > >>>>>>>>>> of the posts here. How does everyone feel about
> > making a
> > > > > point
> > > > > > > >>> of
> > > > > > > >>>>>>>>>> encouraging design reviews to be done as issues,
> > > separate
> > > > > from
> > > > > > > >>> the
> > > > > > > >>>>>> pull
> > > > > > > >>>>>>>>>> request, with the expectations that (1) the design
> > > review
> > > > > > > >> issue
> > > > > > > >>>>>>>>>> ("proposal") should generally appear somewhat
> _before_
> > > the
> > > > > > > >> pull
> > > > > > > >>>>>>> request;
> > > > > > > >>>>>>>>>> (2) pull requests should _not_ have design review
> > happen
> > > > on
> > > > > > > >>> them,
> > > > > > > >>>>>>> meaning
> > > > > > > >>>>>>>>>> there should no longer be PRs with design review
> tags,
> > > and
> > > > > we
> > > > > > > >>>>> should
> > > > > > > >>>>>>> move
> > > > > > > >>>>>>>>>> the design review approval process to the issue
> rather
> > > > than
> > > > > > > >> the
> > > > > > > >>>>> PR.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> For (1), even if we encourage design review
> > discussions
> > > to
> > > > > > > >> start
> > > > > > > >>>>>>> before a
> > > > > > > >>>>>>>>>> pull request appears, I don't see an issue with them
> > > > running
> > > > > > > >>>>>>> concurrently
> > > > > > > >>>>>>>>>> for a while at some point.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <
> > > > > > > >> jonwei@apache.org>
> > > > > > > >>>>>> wrote:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> Thanks for raising these concerns!
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> My initial thoughts:
> > > > > > > >>>>>>>>>>> - I agree that separation of design review and
> > > code-level
> > > > > > > >>> review
> > > > > > > >>>>> for
> > > > > > > >>>>>>>>> major
> > > > > > > >>>>>>>>>>> changes would be more efficient
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> - I agree that a clear, more formalized process for
> > > > > handling
> > > > > > > >>>>> major
> > > > > > > >>>>>>>>> changes
> > > > > > > >>>>>>>>>>> would be helpful for contributors:
> > > > > > > >>>>>>>>>>> - Define what is considered a major change
> > > > > > > >>>>>>>>>>> - Define a standard proposal structure, KIP-style
> > > > proposal
> > > > > > > >>> format
> > > > > > > >>>>>>>>> sounds
> > > > > > > >>>>>>>>>>> good to me
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> - I think it's too rigid to have a policy of "no
> code
> > > at
> > > > > all
> > > > > > > >>> with
> > > > > > > >>>>>> the
> > > > > > > >>>>>>>>>>> initial proposal"
> > > > > > > >>>>>>>>>>> - Code samples can be useful references for
> > > understanding
> > > > > > > >>> aspects
> > > > > > > >>>>>> of a
> > > > > > > >>>>>>>>>>> design
> > > > > > > >>>>>>>>>>> - In some cases it's necessary to run experiments
> to
> > > > fully
> > > > > > > >>>>>> understand
> > > > > > > >>>>>>> a
> > > > > > > >>>>>>>>>>> problem and determine an appropriate design, or to
> > > > > determine
> > > > > > > >>>>> whether
> > > > > > > >>>>>>>>>>> something is even worth doing before committing to
> > the
> > > > work
> > > > > > > >> of
> > > > > > > >>>>>>> fleshing
> > > > > > > >>>>>>>>> out
> > > > > > > >>>>>>>>>>> a proposal, prototype code is a natural outcome of
> > that
> > > > and
> > > > > > > >> I'm
> > > > > > > >>>>> not
> > > > > > > >>>>>>>>> against
> > > > > > > >>>>>>>>>>> someone providing such code for reference
> > > > > > > >>>>>>>>>>> - I tend to view design/code as things that are
> often
> > > > > > > >> developed
> > > > > > > >>>>>>>>>>> simultaneously in an intertwined way
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Let's not be naive this is very rare that a
> > > contributor
> > > > > will
> > > > > > > >>>>> accept
> > > > > > > >>>>>>>>> that
> > > > > > > >>>>>>>>>>> his work is to be thrown, usually devs takes coding
> > as
> > > > > > > >> personal
> > > > > > > >>>>>>> creation
> > > > > > > >>>>>>>>>>> and they get attached to it.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> If we have a clear review process that emphasizes
> the
> > > > need
> > > > > > > >> for
> > > > > > > >>>>> early
> > > > > > > >>>>>>>>>>> consensus building, with separate design and code
> > > review,
> > > > > > > >> then
> > > > > > > >>> I
> > > > > > > >>>>>> feel
> > > > > > > >>>>>>>>> we've
> > > > > > > >>>>>>>>>>> done enough and don't need a hard rule against
> having
> > > > some
> > > > > > > >> code
> > > > > > > >>>>>> linked
> > > > > > > >>>>>>>>> with
> > > > > > > >>>>>>>>>>> the initial proposal. If a potential contributor
> then
> > > > still
> > > > > > > >>>>> wants to
> > > > > > > >>>>>>> go
> > > > > > > >>>>>>>>>>> ahead and write a lot of code that may be rejected
> or
> > > > > change
> > > > > > > >>>>>>>>> significantly,
> > > > > > > >>>>>>>>>>> the risks were made clear.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Once code is written hard to think abstract.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I can see the validity of the concern, but I
> > personally
> > > > > don't
> > > > > > > >>>>> see it
> > > > > > > >>>>>>> as
> > > > > > > >>>>>>>>> a
> > > > > > > >>>>>>>>>>> huge risk. My impression from the Druid PR reviews
> > I've
> > > > > seen
> > > > > > > >> is
> > > > > > > >>>>> that
> > > > > > > >>>>>>> our
> > > > > > > >>>>>>>>>>> reviewers are able to keep abstract design vs.
> > > > > implementation
> > > > > > > >>>>>> details
> > > > > > > >>>>>>>>>>> separate and consider alternate designs when
> > reviewing.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> To summarize I think it's probably enough to have a
> > > > policy
> > > > > > > >>> along
> > > > > > > >>>>> the
> > > > > > > >>>>>>>>> lines
> > > > > > > >>>>>>>>>>> of:
> > > > > > > >>>>>>>>>>> - Create more formalized guidelines for proposals
> and
> > > > what
> > > > > > > >>>>> changes
> > > > > > > >>>>>>>>> require
> > > > > > > >>>>>>>>>>> proposals
> > > > > > > >>>>>>>>>>> - Separate design and code review for major
> changes,
> > > with
> > > > > > > >>> design
> > > > > > > >>>>>>> review
> > > > > > > >>>>>>>>>>> first, code-level review after reaching consensus
> on
> > > the
> > > > > > > >>> design.
> > > > > > > >>>>>>>>>>> - Code before the design review is completed is
> just
> > > for
> > > > > > > >>>>> reference,
> > > > > > > >>>>>>> not
> > > > > > > >>>>>>>>>>> regarded as a candidate for review/merging.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> - Jon
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > > > > > > >>>>>>>>> slim.bouguerra@gmail.com>
> > > > > > > >>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <
> > > > > > > >>>>> clint.wylie@imply.io>
> > > > > > > >>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> I am definitely biased in this matter as an owner
> > of
> > > > > > > >> another
> > > > > > > >>>>> large
> > > > > > > >>>>>>> PR
> > > > > > > >>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>> wasn't preceded by a direct proposal or dev list
> > > > > > > >> discussion,
> > > > > > > >>>>> and
> > > > > > > >>>>>> in
> > > > > > > >>>>>>>>>>>> general
> > > > > > > >>>>>>>>>>>>> I agree that proposal first is usually better,
> but
> > I
> > > > > think
> > > > > > > >> in
> > > > > > > >>>>> some
> > > > > > > >>>>>>>>>>> rarer
> > > > > > > >>>>>>>>>>>>> cases approaching a problem code first *is* the
> > most
> > > > > > > >>>>> appropriate
> > > > > > > >>>>>> way
> > > > > > > >>>>>>>>> to
> > > > > > > >>>>>>>>>>>>> have a discussion.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> I am wondering here what is the case where code
> > first
> > > is
> > > > > > > >>> better?
> > > > > > > >>>>>>>>>>>> In general when you are writing code you have an
> > idea
> > > > > about
> > > > > > > >>> what
> > > > > > > >>>>>> you
> > > > > > > >>>>>>>>> want
> > > > > > > >>>>>>>>>>>> to change, why you want to change and why you want
> > to
> > > > > change
> > > > > > > >>> it.
> > > > > > > >>>>>>>>>>>> I do not see what is wrong with sharing this
> > primitive
> > > > > ideas
> > > > > > > >>> and
> > > > > > > >>>>>>>>> thoughts
> > > > > > > >>>>>>>>>>>> as an abstract proposal (at least to avoid
> > > overlapping)
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> I see nothing wrong with it so long as the author
> > > > > > > >>>>>>>>>>>>> accepts that the PR is treated as a combined
> > proposal
> > > > and
> > > > > > > >>>>> proof of
> > > > > > > >>>>>>>>>>>> concept,
> > > > > > > >>>>>>>>>>>>> and fair game to be radically changed via
> > discussion
> > > or
> > > > > > > >> even
> > > > > > > >>>>>>> rejected,
> > > > > > > >>>>>>>>>>>>> which sounds like Gian's attitude on the matter
> and
> > > is
> > > > > mine
> > > > > > > >>> as
> > > > > > > >>>>>> well
> > > > > > > >>>>>>>>>>> with
> > > > > > > >>>>>>>>>>>> my
> > > > > > > >>>>>>>>>>>>> compression stuff.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Let's not be naive this is very rare that a
> > > contributor
> > > > > will
> > > > > > > >>>>> accept
> > > > > > > >>>>>>>>> that
> > > > > > > >>>>>>>>>>>> his work is to be thrown, usually devs takes
> coding
> > as
> > > > > > > >>> personal
> > > > > > > >>>>>>>>> creation
> > > > > > > >>>>>>>>>>>> and they get attached to it.
> > > > > > > >>>>>>>>>>>> To my point you can take a look on some old issue
> in
> > > the
> > > > > > > >> Druid
> > > > > > > >>>>>> forum
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> > > > > > > >>>>>>>>>>>> and am sure other communities have similar
> problems.
> > > > > > > >>>>>>>>>>>> So leaving the door open to some side cases is
> not a
> > > > good
> > > > > > > >> idea
> > > > > > > >>>>> in
> > > > > > > >>>>>> my
> > > > > > > >>>>>>>>>>>> opinion and will lead to similar issue in the
> > future.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> This seems to me especially likely to happen in
> > cases
> > > > > > > >>>>>>>>>>>>> where an approach still needs proven to be a
> viable
> > > > idea
> > > > > > > >> *to
> > > > > > > >>>>> the
> > > > > > > >>>>>>>>>>> author*,
> > > > > > > >>>>>>>>>>>>> so that a much more productive discussion can be
> > had
> > > in
> > > > > the
> > > > > > > >>>>> first
> > > > > > > >>>>>>>>>>> place.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> I think there is a trade off, I don't think we
> want
> > > to
> > > > > > > >>>>> discourage
> > > > > > > >>>>>>>>>>>>> experimentation by walling it off behind
> mandatory
> > > > > > > >>> discussions
> > > > > > > >>>>>>> before
> > > > > > > >>>>>>>>>>> it
> > > > > > > >>>>>>>>>>>>> can even start, but I do think formalizing the
> > > process
> > > > > for
> > > > > > > >>>>> large
> > > > > > > >>>>>>>>>>> changes
> > > > > > > >>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>> a good thing, especially since we probably can't
> > > expect
> > > > > the
> > > > > > > >>>>> wider
> > > > > > > >>>>>>>>>>>> community
> > > > > > > >>>>>>>>>>>>> to have the same attitude towards a large PR
> > getting
> > > > > > > >>> discarded
> > > > > > > >>>>> as
> > > > > > > >>>>>> a
> > > > > > > >>>>>>>>>>>>> committer might. I think the Kafka approach is
> > > > > reasonable,
> > > > > > > >> a
> > > > > > > >>>>> bit
> > > > > > > >>>>>>> more
> > > > > > > >>>>>>>>>>>>> formal than our design review process but not
> > > > > overbearing.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Can you please explain what is overbearing ? what
> > can
> > > be
> > > > > > > >>>>> changed to
> > > > > > > >>>>>>>>> make
> > > > > > > >>>>>>>>>>> it
> > > > > > > >>>>>>>>>>>> easy ?
> > > > > > > >>>>>>>>>>>> Most of the points are kind of the actual
> questions
> > > that
> > > > > you
> > > > > > > >>>>> want
> > > > > > > >>>>>> to
> > > > > > > >>>>>>>>>>>> address before hand anyway isn't ?
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Going code first
> > > > > > > >>>>>>>>>>>>> should be in general discouraged, but when it
> does
> > > > > happen,
> > > > > > > >> it
> > > > > > > >>>>>> seems
> > > > > > > >>>>>>>>>>> like
> > > > > > > >>>>>>>>>>>>> opening DIP/an issue/starting a mailing list
> thread
> > > or
> > > > > > > >>>>> whatever we
> > > > > > > >>>>>>> go
> > > > > > > >>>>>>>>>>>> with
> > > > > > > >>>>>>>>>>>>> to have a more high level design discussion
> > alongside
> > > > the
> > > > > > > >>>>>> reference
> > > > > > > >>>>>>> PR
> > > > > > > >>>>>>>>>>>>> could alleviate some of these complaints?
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> What are the complaints ?
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> +1 for "DIP" heh, I think making
> > > > > > > >>>>>>>>>>>>> them in the form of github issues is probably
> > > > > appropriate,
> > > > > > > >>>>> with a
> > > > > > > >>>>>>> dev
> > > > > > > >>>>>>>>>>>> list
> > > > > > > >>>>>>>>>>>>> thread to announce them perhaps?
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> I think  github issue with [Proposal] header like
> > > > > > > >>>>>>>>>>>>
> > https://github.com/apache/incubator-druid/issues/4349
> > > > is
> > > > > > > >> good
> > > > > > > >>>>> to
> > > > > > > >>>>>> me,
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Thanks!
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <
> > > > > > > >>>>> bslim@apache.org>
> > > > > > > >>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Thanks everyone for interacting with this
> thread.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> The fact that i, Roman, Jihoon  and others in
> the
> > > past
> > > > > (FJ
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ
> > > > > > > >>>>> )
> > > > > > > >>>>>>>>>>>>>> raised this point indicates that PRs without a
> > > > proposal
> > > > > > > >> are
> > > > > > > >>>>>> indeed
> > > > > > > >>>>>>> an
> > > > > > > >>>>>>>>>>>>> issue
> > > > > > > >>>>>>>>>>>>>> and we need to solve it.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Something Similar to KIP maybe called DIPs is
> fine
> > > > with
> > > > > > > >> me.
> > > > > > > >>>>>>>>>>>>>> What i strive to see is the following:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> [Step 1] formalize what is the kind of work that
> > > > needs a
> > > > > > > >>>>> formal
> > > > > > > >>>>>>>>>>>>> Proposal, I
> > > > > > > >>>>>>>>>>>>>> think Roman and Jihoon has already covered that
> > > pretty
> > > > > > > >> well.
> > > > > > > >>>>> am
> > > > > > > >>>>>> +1
> > > > > > > >>>>>>> on
> > > > > > > >>>>>>>>>>>>> that.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> I am strongly in favor of the separation of
> > Proposal
> > > > > > > >> Review
> > > > > > > >>>>> and
> > > > > > > >>>>>>>>>>> (later)
> > > > > > > >>>>>>>>>>>>>> Code review PRs. My  main reasons:
> > > > > > > >>>>>>>>>>>>>> Most importantly code reviewing will introduce
> lot
> > > of
> > > > > > > >> noise
> > > > > > > >>>>> and
> > > > > > > >>>>>>> will
> > > > > > > >>>>>>>>>>>>>> ultimately make  the GitHub page unreadable.
> > > > > > > >>>>>>>>>>>>>> Avoid overlapping of work.
> > > > > > > >>>>>>>>>>>>>> Once code is written hard to think abstract.
> > > > > > > >>>>>>>>>>>>>> Separate page for Design review later can always
> > be
> > > > used
> > > > > > > >> it
> > > > > > > >>>>> as a
> > > > > > > >>>>>>>>>>> Design
> > > > > > > >>>>>>>>>>>>>> document that is readable and code free-ish.
> > > > > > > >>>>>>>>>>>>>> As i said the goal of this first round is to see
> > if
> > > > the
> > > > > > > >>>>> community
> > > > > > > >>>>>>>>>>> agree
> > > > > > > >>>>>>>>>>>>>> about such change, then make the process of
> design
> > > > more
> > > > > > > >>>>> inclusive
> > > > > > > >>>>>>>>>>> thus
> > > > > > > >>>>>>>>>>>>>> other contributors can submit a counter
> proposals.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> [Step 2] IF everybody agree about that point
> Step
> > 2
> > > is
> > > > > to
> > > > > > > >>>>> define
> > > > > > > >>>>>>>>>>> which
> > > > > > > >>>>>>>>>>>>>> medium is used to Publish a primitive form of a
> > CODE
> > > > > FREE
> > > > > > > >>>>>> Abstract
> > > > > > > >>>>>>>>>>>>> Proposal
> > > > > > > >>>>>>>>>>>>>> containing at least the following bullet points.
> > > > > > > >>>>>>>>>>>>>> - The problem description and motivation
> > > > > > > >>>>>>>>>>>>>> - Overview of the proposed change
> > > > > > > >>>>>>>>>>>>>> - Operational impact (compatibility/ plans to
> > > > upgrades)
> > > > > > > >>> public
> > > > > > > >>>>>> API
> > > > > > > >>>>>>>>>>>>> changes,
> > > > > > > >>>>>>>>>>>>>> configuration changes, algorithm, and so on
> > > > > > > >>>>>>>>>>>>>> - Expected benefits and drawbacks
> > > > > > > >>>>>>>>>>>>>> - Rationale and alternatives
> > > > > > > >>>>>>>>>>>>>> - Estimate Time to Deliver if possible.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> The way i think this can be is a Github issue
> > where
> > > > > member
> > > > > > > >>> of
> > > > > > > >>>>> the
> > > > > > > >>>>>>>>>>>>> community
> > > > > > > >>>>>>>>>>>>>> will interact via comments and the author will
> be
> > > > > updating
> > > > > > > >>> the
> > > > > > > >>>>>>>>>>>>> description
> > > > > > > >>>>>>>>>>>>>> in the light of comments provided by the
> > community.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> During and near the end of the design
> discussions
> > > the
> > > > > > > >>> author/s
> > > > > > > >>>>>> can
> > > > > > > >>>>>>>>>>>> start
> > > > > > > >>>>>>>>>>>>>> writing POCs to help guide the review process
> this
> > > > > > > >> naturally
> > > > > > > >>>>> will
> > > > > > > >>>>>>> be
> > > > > > > >>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>> Pull
> > > > > > > >>>>>>>>>>>>>> request with actual code.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> *Now the most important thing is that we need to
> > > agree
> > > > > > > >> that
> > > > > > > >>>>> any
> > > > > > > >>>>>>> work
> > > > > > > >>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>> does not align with this formal process will be
> > > > ignored
> > > > > > > >> and
> > > > > > > >>>>> the
> > > > > > > >>>>>>>>>>> author
> > > > > > > >>>>>>>>>>>>> will
> > > > > > > >>>>>>>>>>>>>> be asked to start with a DIP*
> > > > > > > >>>>>>>>>>>>>> *That is what i meant with  “If it didn’t happen
> > on
> > > > the
> > > > > > > >>>>> mailing
> > > > > > > >>>>>>> list,
> > > > > > > >>>>>>>>>>>> it
> > > > > > > >>>>>>>>>>>>>> didn’t happen.”*
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Thanks and happy coding!
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <
> > > > > > > >>> gian@apache.org>
> > > > > > > >>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> One of the advantages I see with a more formal
> > > > process
> > > > > is
> > > > > > > >>>>> (like
> > > > > > > >>>>>>>>>>> Kafka
> > > > > > > >>>>>>>>>>>>>> KIPs)
> > > > > > > >>>>>>>>>>>>>>> is that it levels the playing field a bit and
> > sets
> > > > some
> > > > > > > >>>>> ground
> > > > > > > >>>>>>>>>>> rules
> > > > > > > >>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>> working together. In a way it can help
> encourage
> > > > > > > >>>>> contributions
> > > > > > > >>>>>> by
> > > > > > > >>>>>>>>>>>>> making
> > > > > > > >>>>>>>>>>>>>> it
> > > > > > > >>>>>>>>>>>>>>> clear what is expected of potential
> contributors.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> We have a design review process today that is
> not
> > > as
> > > > > > > >> formal
> > > > > > > >>>>> as
> > > > > > > >>>>>>>>>>> KIPs,
> > > > > > > >>>>>>>>>>>>> but
> > > > > > > >>>>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>>> somewhat heavier than the one you describe.
> Maybe
> > > we
> > > > > > > >> could
> > > > > > > >>>>> tweak
> > > > > > > >>>>>>>>>>> our
> > > > > > > >>>>>>>>>>>>>>> current one by starting to do design reviews
> > > > separately
> > > > > > > >>> from
> > > > > > > >>>>>> PRs.
> > > > > > > >>>>>>>>>>>> i.e.,
> > > > > > > >>>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>> anything that meets our 'design review'
> criteria,
> > > do
> > > > > that
> > > > > > > >>> on
> > > > > > > >>>>> the
> > > > > > > >>>>>>>>>>> dev
> > > > > > > >>>>>>>>>>>>> list
> > > > > > > >>>>>>>>>>>>>>> or in a separate issue, and keep the PR focused
> > on
> > > > > > > >>> code-level
> > > > > > > >>>>>>>>>>> stuff.
> > > > > > > >>>>>>>>>>>>> That
> > > > > > > >>>>>>>>>>>>>>> way we don't end up trying to do both at once.
> > And
> > > it
> > > > > > > >> makes
> > > > > > > >>>>> it
> > > > > > > >>>>>>>>>>> easier
> > > > > > > >>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>> start talking about design before the code is
> > > ready,
> > > > > > > >> which
> > > > > > > >>>>> would
> > > > > > > >>>>>>> be
> > > > > > > >>>>>>>>>>>>>> better.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <
> > > > > > > >>> jhyde@apache.org
> > > > > > > >>>>>>
> > > > > > > >>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> It’s really hard to say no to a contribution
> > when
> > > > > > > >> someone
> > > > > > > >>>>> has
> > > > > > > >>>>>> put
> > > > > > > >>>>>>>>>>>> in
> > > > > > > >>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>> significant amount of work.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> The following approach is simple and works
> > really
> > > > > well:
> > > > > > > >>>>> Before
> > > > > > > >>>>>>>>>>> you
> > > > > > > >>>>>>>>>>>>>> start
> > > > > > > >>>>>>>>>>>>>>>> work, log a case, describing the problem. When
> > you
> > > > > have
> > > > > > > >>> some
> > > > > > > >>>>>>>>>>> ideas
> > > > > > > >>>>>>>>>>>>>> about
> > > > > > > >>>>>>>>>>>>>>>> design, add those to the case. When you have a
> > > code
> > > > > > > >>> branch,
> > > > > > > >>>>> add
> > > > > > > >>>>>>>>>>> its
> > > > > > > >>>>>>>>>>>>> URL
> > > > > > > >>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>> the case. And so forth. At any point in the
> > > > > proceedings,
> > > > > > > >>>>> people
> > > > > > > >>>>>>>>>>> can
> > > > > > > >>>>>>>>>>>>>> chime
> > > > > > > >>>>>>>>>>>>>>>> in with their opinions.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> In my opinion, a formal “design review”
> process
> > is
> > > > not
> > > > > > > >>>>>> necessary.
> > > > > > > >>>>>>>>>>>>> Just
> > > > > > > >>>>>>>>>>>>>>>> build consensus iteratively, by starting the
> > > > > > > >> conversation
> > > > > > > >>>>> early
> > > > > > > >>>>>>>>>>> in
> > > > > > > >>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>> process.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Julian
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <
> > > > > > > >>> gian@apache.org
> > > > > > > >>>>>>
> > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> In this particular case: please consider the
> PR
> > > as
> > > > a
> > > > > > > >>>>> proposal.
> > > > > > > >>>>>>>>>>>>> Don't
> > > > > > > >>>>>>>>>>>>>>> feel
> > > > > > > >>>>>>>>>>>>>>>>> like just because there is code there that
> > takes
> > > a
> > > > > > > >>> certain
> > > > > > > >>>>>>>>>>>>> approach,
> > > > > > > >>>>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>> the approach is somehow sacred. I had to
> > > implement
> > > > > > > >>>>> something
> > > > > > > >>>>>> to
> > > > > > > >>>>>>>>>>>>>>>> crystallize
> > > > > > > >>>>>>>>>>>>>>>>> my own thinking about how the problem could
> be
> > > > > > > >>> approached.
> > > > > > > >>>>> I
> > > > > > > >>>>>>>>>>>> won't
> > > > > > > >>>>>>>>>>>>> be
> > > > > > > >>>>>>>>>>>>>>>>> disappointed if, as a community, we decide a
> > > > > different
> > > > > > > >>>>>>>>>>> direction
> > > > > > > >>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>>>> better
> > > > > > > >>>>>>>>>>>>>>>>> and the code all gets thrown away. That's one
> > of
> > > > the
> > > > > > > >>>>> reasons
> > > > > > > >>>>>>>>>>>> that I
> > > > > > > >>>>>>>>>>>>>>>> removed
> > > > > > > >>>>>>>>>>>>>>>>> the 0.14.0 milestone that was added to the
> > patch.
> > > > (I
> > > > > > > >>> don't
> > > > > > > >>>>>> want
> > > > > > > >>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>> rush
> > > > > > > >>>>>>>>>>>>>>>> it,
> > > > > > > >>>>>>>>>>>>>>>>> nor do I think that's a good idea.)
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> In general: Sounds like we could do with some
> > > more
> > > > > > > >>>>>>>>>>> formalization
> > > > > > > >>>>>>>>>>>>>> around
> > > > > > > >>>>>>>>>>>>>>>>> what a proposal looks like, which sorts of
> > > changes
> > > > > need
> > > > > > > >>>>> one,
> > > > > > > >>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>> when
> > > > > > > >>>>>>>>>>>>>>> in
> > > > > > > >>>>>>>>>>>>>>>>> the dev cycle it is appropriate. FWIW I think
> > > > Kafka's
> > > > > > > >>>>> process
> > > > > > > >>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>> more
> > > > > > > >>>>>>>>>>>>>>> or
> > > > > > > >>>>>>>>>>>>>>>>> less fine, and would be okay with adopting it
> > for
> > > > > Druid
> > > > > > > >>> if
> > > > > > > >>>>>>>>>>> people
> > > > > > > >>>>>>>>>>>>>> like
> > > > > > > >>>>>>>>>>>>>>>> it.
> > > > > > > >>>>>>>>>>>>>>>>> Right now our standards for what requires a
> > > "design
> > > > > > > >>> review"
> > > > > > > >>>>>> are
> > > > > > > >>>>>>>>>>>>> very
> > > > > > > >>>>>>>>>>>>>>>>> similar to the Kafka community standards for
> > what
> > > > > > > >>> requires
> > > > > > > >>>>> a
> > > > > > > >>>>>>>>>>> KIP,
> > > > > > > >>>>>>>>>>>>> so
> > > > > > > >>>>>>>>>>>>>> we
> > > > > > > >>>>>>>>>>>>>>>>> have some familiarity with those concepts.
> > > However
> > > > we
> > > > > > > >>> don't
> > > > > > > >>>>>>>>>>>>> separate
> > > > > > > >>>>>>>>>>>>>> PR
> > > > > > > >>>>>>>>>>>>>>>>> review and proposal discussion as strictly as
> > > they
> > > > > do,
> > > > > > > >>>>> which
> > > > > > > >>>>>>>>>>>> seems
> > > > > > > >>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>> be
> > > > > > > >>>>>>>>>>>>>>>>> the foundation for the feeling of exclusion
> > that
> > > is
> > > > > > > >> being
> > > > > > > >>>>> felt
> > > > > > > >>>>>>>>>>>>> here.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Separately: I just redid the description on
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > https://github.com/apache/incubator-druid/pull/6794
> > > > > to
> > > > > > > >>> be
> > > > > > > >>>>>> more
> > > > > > > >>>>>>>>>>>>>>>> proposal-y.
> > > > > > > >>>>>>>>>>>>>>>>> I followed the KIP style:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > > >>>>>>>>>>>>>>>> .
> > > > > > > >>>>>>>>>>>>>>>>> Please refresh the page and see if it looks
> > more
> > > > > > > >> useful.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Gian
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <
> > > > > > > >>>>> jhyde@apache.org
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Slim,
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> I agree with your points that offline
> > > development
> > > > is
> > > > > > > >> bad
> > > > > > > >>>>> for
> > > > > > > >>>>>>>>>>>>>>> community.
> > > > > > > >>>>>>>>>>>>>>>>>> But I don’t think you need much mentor help.
> > You
> > > > > have
> > > > > > > >>>>> raised
> > > > > > > >>>>>>>>>>>> valid
> > > > > > > >>>>>>>>>>>>>>>> issues
> > > > > > > >>>>>>>>>>>>>>>>>> and the Druid community needs to decide what
> > its
> > > > > > > >>>>> development
> > > > > > > >>>>>>>>>>>>>> practices
> > > > > > > >>>>>>>>>>>>>>>>>> should be.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Julian
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim
> Bouguerra <
> > > > > > > >>>>>>>>>>> bslim@apache.org>
> > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Hello everyone and hope you all have very
> > good
> > > > > > > >>> holidays.
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> First, this email is not directed on the
> > author
> > > > or
> > > > > > > >> the
> > > > > > > >>> PR
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > https://github.com/apache/incubator-druid/pull/6794
> > > > > > > >>> it
> > > > > > > >>>>>>>>>>> self,
> > > > > > > >>>>>>>>>>>>> but
> > > > > > > >>>>>>>>>>>>>> i
> > > > > > > >>>>>>>>>>>>>>>> see
> > > > > > > >>>>>>>>>>>>>>>>>>> this PR as a perfect example.
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> One of the foundation of Apache Way or
> what i
> > > > would
> > > > > > > >>>>> simply
> > > > > > > >>>>>>>>>>> call
> > > > > > > >>>>>>>>>>>>>> open
> > > > > > > >>>>>>>>>>>>>>>>>> source
> > > > > > > >>>>>>>>>>>>>>>>>>> community driven development is that
> > "Technical
> > > > > > > >>> decisions
> > > > > > > >>>>>> are
> > > > > > > >>>>>>>>>>>>>>>> discussed,
> > > > > > > >>>>>>>>>>>>>>>>>>> decided, and archived publicly.
> > > > > > > >>>>>>>>>>>>>>>>>>> developpement"
> > > > > > > >>>>>>>>>>>>>>>>>>> Which means that big technical  changes
> such
> > as
> > > > the
> > > > > > > >> one
> > > > > > > >>>>>>>>>>> brought
> > > > > > > >>>>>>>>>>>>> by
> > > > > > > >>>>>>>>>>>>>>>> #/6794
> > > > > > > >>>>>>>>>>>>>>>>>>> should have started as a proposal and round
> > of
> > > > > > > >>>>> discussions
> > > > > > > >>>>>>>>>>>> about
> > > > > > > >>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>> major
> > > > > > > >>>>>>>>>>>>>>>>>>> changes designs not as 11K line of code.
> > > > > > > >>>>>>>>>>>>>>>>>>> I believe such openness will promote a lot
> of
> > > > good
> > > > > > > >>>>> benefits
> > > > > > > >>>>>>>>>>>> such
> > > > > > > >>>>>>>>>>>>>> as:
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> - ensures community health and growth.
> > > > > > > >>>>>>>>>>>>>>>>>>> - ensures everyone can participate not only
> > the
> > > > > > > >> authors
> > > > > > > >>>>> and
> > > > > > > >>>>>>>>>>> his
> > > > > > > >>>>>>>>>>>>>>>>>> co-workers.
> > > > > > > >>>>>>>>>>>>>>>>>>> - ensures that the project is driven by the
> > > > > community
> > > > > > > >>> and
> > > > > > > >>>>>>>>>>> not a
> > > > > > > >>>>>>>>>>>>>> given
> > > > > > > >>>>>>>>>>>>>>>>>>> company or an individual.
> > > > > > > >>>>>>>>>>>>>>>>>>> - ensures that there is consensus (not
> saying
> > > > 100%
> > > > > > > >>>>>>>>>>> agreement;)
> > > > > > > >>>>>>>>>>>>>>> however
> > > > > > > >>>>>>>>>>>>>>>> it
> > > > > > > >>>>>>>>>>>>>>>>>>> means that all individuals will accept the
> > > > current
> > > > > > > >>>>> progress
> > > > > > > >>>>>>>>>>> on
> > > > > > > >>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>> project
> > > > > > > >>>>>>>>>>>>>>>>>>> until some better proposal is put forth.
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Personally such BIG offline PR makes me
> feel
> > > > > excluded
> > > > > > > >>> and
> > > > > > > >>>>>>>>>>>> doesn't
> > > > > > > >>>>>>>>>>>>>>> give
> > > > > > > >>>>>>>>>>>>>>>>>> me a
> > > > > > > >>>>>>>>>>>>>>>>>>> sense that i belong to  a community at all.
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> To prevent such off list development i
> think
> > > as a
> > > > > > > >> Druid
> > > > > > > >>>>>>>>>>>> Community
> > > > > > > >>>>>>>>>>>>>> we
> > > > > > > >>>>>>>>>>>>>>>> need
> > > > > > > >>>>>>>>>>>>>>>>>>> to stick to the apache way “If it didn’t
> > happen
> > > > on
> > > > > > > >> the
> > > > > > > >>>>>>>>>>> mailing
> > > > > > > >>>>>>>>>>>>>> list,
> > > > > > > >>>>>>>>>>>>>>> it
> > > > > > > >>>>>>>>>>>>>>>>>>> didn’t happen.”
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> I would appreciate if some of the Apache
> > mentor
> > > > > help
> > > > > > > >>> with
> > > > > > > >>>>>>>>>>> this.
> > > > > > > >>>>>>>>>>>>>>>>>>> Thanks
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>
> > > > >
> ---------------------------------------------------------------------
> > > > > > > >>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > > > > > > >>> dev-unsubscribe@druid.apache.org
> > > > > > > >>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > > >>>>> dev-help@druid.apache.org
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>
> > > > >
> ---------------------------------------------------------------------
> > > > > > > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > > > > > > >> dev-unsubscribe@druid.apache.org
> > > > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > > >>> dev-help@druid.apache.org
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> --
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> B-Slim
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>
> > > > > > >
> > > > >
> > >
> _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > >>>>>>>>> To unsubscribe, e-mail:
> > dev-unsubscribe@druid.apache.org
> > > > > > > >>>>>>>>> For additional commands, e-mail:
> > > dev-help@druid.apache.org
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>
> > > > >
> ---------------------------------------------------------------------
> > > > > > > >>>>>>> To unsubscribe, e-mail:
> dev-unsubscribe@druid.apache.org
> > > > > > > >>>>>>> For additional commands, e-mail:
> > dev-help@druid.apache.org
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > > > > For additional commands, e-mail: dev-help@druid.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Off list major development

[Jonathan Wei <jo...@apache.org>]

Hi all,

An issue has been opened by a community member suggesting that we create a
template for proposals:
https://github.com/apache/incubator-druid/issues/6949

Having a template sounds convenient, and based on the discussion in this
thread, I'm suggesting we adopt something based on the Kafka proposal
format.

I'm planning on creating such a template if there are no objections or
alternative suggestions, so please take a look if you have thoughts on this.

Thanks,
Jon

On Tue, Jan 15, 2019 at 12:07 PM Jihoon Son <gh...@gmail.com> wrote:

> Good point.
> If some authors raise PRs without noticing the need for a proposal, we
> shouldn't ask them to close their PRs only because of the absence of the
> proposal.
>
> "Design review" without a proposal for simple PRs would be good if we can
> determine well what PRs need and what don't.
> But, how do we know? Even for the same PR, someone may think it needs a
> proposal but another may not.
>
> If someone don't notice the need for a proposal and raise a PR without it,
> I'm fine with that.
> However, we should still encourage writing a proposal before writing code
> because we can avoid unnecessary effort.
>
> I think this kind of issue usually happens for first time contributors and
> they will be better once they get used to Druid development.
> And I believe someday even first contributors would follow this policy once
> it gets settled down well in the community as Kafka community does.
>
> Jihoon
>
> On Tue, Jan 15, 2019 at 4:31 AM Roman Leventov <le...@gmail.com>
> wrote:
>
> > In such small PRs, authors likely won't be aware that they need to
> create a
> > proposal in the first place. The first reviewer just adds the "Design
> > Review" tag. It's also absolutely not about considering designs and
> gauging
> > the proposal, it's just verifying that a configuration / parameter / HTTP
> > endpoint name is reasonable and aligned with the rest of Druid. So I
> think
> > that a separate proposal issue for such PRs is unnecessary bureaucracy.
> >
> > On Tue, 15 Jan 2019 at 07:45, Jihoon Son <gh...@gmail.com> wrote:
> >
> > > Roman,
> > >
> > > > Jihoon in
> > >
> > >
> >
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> > > and later Gian in this thread suggested that _every_ piece of work that
> > > should be labelled as "Design Review" according to the current rules
> > should
> > > be accompanied by an issue. I don't agree with this, there are some PRs
> > as
> > > small as a few dozens of lines of code, that add some configuration
> > > parameter and therefore should be labelled "Design Review". I don't
> > thing a
> > > separate proposal issue is needed for them, and even for a little
> larger
> > > PRs too.
> > >
> > > What I'm concerned with is how people feel if their design is not
> > accepted
> > > even though they wrote code. Of course, as Clint said, sometimes code
> > helps
> > > better understanding of the proposal. But, I believe this is the case
> > when
> > > the proposal is quite complicated and not easy to understand without
> > code.
> > > Also the authors should be aware of that they might rewrite the entire
> > code
> > > if the design should be changed.
> > >
> > > If writing code is simple, I don't see why the authors don't wait until
> > the
> > > review for their proposal is finished.
> > >
> > > Jihoon
> > >
> > > On Fri, Jan 11, 2019 at 9:51 AM Fangjin Yang <fa...@imply.io> wrote:
> > >
> > > > I agree with Gian, as an Apache committer, your responsibility is for
> > the
> > > > betterment of the project. I agree it is in the best interest of the
> > > > project to stop thinking about what orgs people belong to. We are
> all a
> > > > part of the Apache software foundation, regardless of what our roles
> > and
> > > > titles are outside of it.
> > > >
> > > > On Fri, Jan 11, 2019 at 2:22 AM Roman Leventov <
> leventov.ru@gmail.com>
> > > > wrote:
> > > >
> > > > > It's not that people from one org could abuse the project and push
> > some
> > > > > change, but that they have similar perspective (bubble effect) and
> > some
> > > > > important aspects of a large feature could escape their attention.
> > > > >
> > > > > I suggest it to be not a rigid rule, but a recommendation for
> authors
> > > of
> > > > > large proposals to try to attract reviewers from other orgs.
> > > > >
> > > > > On Fri, 11 Jan 2019 at 02:51, Julian Hyde <jh...@apache.org>
> wrote:
> > > > >
> > > > > > I agree with Gian.
> > > > > >
> > > > > > As an Apache committer, you only have one affiliation: you are
> > > working
> > > > in
> > > > > > the best interests of the project.
> > > > > >
> > > > > > Obviously, in the real world there are other pressures. But we do
> > our
> > > > > best
> > > > > > to compensate for them.
> > > > > >
> > > > > > Also, as a a community we try to design our process so as to
> avoid
> > > > undue
> > > > > > influences. For instance, when I advocate for logging cases
> early,
> > I
> > > am
> > > > > > trying to mitigate the effect of product managers and VPs of
> > > > engineering,
> > > > > > who like to have their say in meeting rooms rather than on public
> > > > mailing
> > > > > > lists. That’s just one example; if we see other influences at
> play,
> > > > let’s
> > > > > > evolve our process to try to level the playing field.
> > > > > >
> > > > > > Julian
> > > > > >
> > > > > >
> > > > > > > On Jan 10, 2019, at 10:40 AM, Gian Merlino <gi...@apache.org>
> > > wrote:
> > > > > > >
> > > > > > >>> What do community members think about also making a
> requirement
> > > > that
> > > > > > >>> "Design Review" PRs and proposals are reviewed by at least
> two
> > > > people
> > > > > > > with
> > > > > > >>> different affiliation?
> > > > > > >> This seems pretty reasonable to me. I haven't found anything
> in
> > > > Apache
> > > > > > >> voting procedure docs (
> > > > https://www.apache.org/foundation/voting.html)
> > > > > > that
> > > > > > >> seems to explicitly forbid something like this yet at least.
> > > > > > >
> > > > > > > On the other hand, my understanding of the Apache Way would
> mean
> > > that
> > > > > > this
> > > > > > > kind of rule doesn't make sense. In particular from
> > > > > > > https://www.apache.org/foundation/how-it-works.html: "We
> firmly
> > > > > believe
> > > > > > in
> > > > > > > hats. Your role at the ASF is one assigned to you personally,
> and
> > > is
> > > > > > > bestowed on you by your peers. It is not tied to your job or
> > > current
> > > > > > > employer or company." That sentiment seems incompatible with
> > making
> > > > > > > explicit rules about organizational diversity in voting. I have
> > > also
> > > > > > heard
> > > > > > > a few people say things like: people are supposed to represent
> > > > > > themselves,
> > > > > > > not their employers.
> > > > > > >
> > > > > > > Obviously, though, people's actions and opinions are influenced
> > by
> > > > > their
> > > > > > > employer. IMO, a better way to approach a potential problem
> there
> > > is
> > > > > that
> > > > > > > if people from a particular organization end up behaving
> > abusively,
> > > > > then
> > > > > > > PMC members from other organizations (or in extreme situations
> > the
> > > > > Apache
> > > > > > > board itself) should tell them to knock it off. And then do
> more
> > > > > serious
> > > > > > > things if the inappropriate behavior continues. This kind
> > > > > > >
> > > > > > > The other thing you brought up, promoting more shared ownership
> > and
> > > > > > > awareness, I am hopeful that separating proposals from PRs will
> > > help
> > > > > with
> > > > > > > that. One reason is that it takes much less time to understand
> a
> > > well
> > > > > > > written proposal than it takes to understand a PR. Code is very
> > > clear
> > > > > but
> > > > > > > it is also very verbose and takes a while to read and
> understand.
> > > So
> > > > in
> > > > > > > theory changing how we operate, in this way, should promote
> more
> > > > > > > understanding of more people across the code base.
> > > > > > >
> > > > > > > On Thu, Jan 10, 2019 at 1:34 AM Clint Wylie <
> > clint.wylie@imply.io>
> > > > > > wrote:
> > > > > > >
> > > > > > >>>
> > > > > > >>> What do community members think about also making a
> requirement
> > > > that
> > > > > > >>> "Design Review" PRs and proposals are reviewed by at least
> two
> > > > people
> > > > > > >> with
> > > > > > >>> different affiliation?
> > > > > > >>
> > > > > > >>
> > > > > > >> This seems pretty reasonable to me. I haven't found anything
> in
> > > > Apache
> > > > > > >> voting procedure docs (
> > > > https://www.apache.org/foundation/voting.html)
> > > > > > that
> > > > > > >> seems to explicitly forbid something like this yet at least.
> > > > > > >>
> > > > > > >> On Wed, Jan 9, 2019 at 8:52 PM Roman Leventov <
> > > > leventov.ru@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> What do community members think about also making a
> requirement
> > > > that
> > > > > > >>> "Design Review" PRs and proposals are reviewed by at least
> two
> > > > people
> > > > > > >> with
> > > > > > >>> different affiliation? IMO it's a good idea, because it
> ensures
> > > > that
> > > > > > >>> different interests are taken into account. Also it pushes
> > people
> > > > to
> > > > > > >> engage
> > > > > > >>> with work done in other parts of Druid, improving shared code
> > > > owning
> > > > > > and
> > > > > > >>> awareness.
> > > > > > >>>
> > > > > > >>> Or it's against the Apache Way because it's assumed that
> there
> > > are
> > > > no
> > > > > > >>> company boundaries within the community?
> > > > > > >>>
> > > > > > >>> On Thu, 10 Jan 2019 at 11:45, Roman Leventov <
> > > > leventov.ru@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>> I see two important reasons why it makes sense to file an
> > issue
> > > > and
> > > > > > >>>> probably announce it in the mailing list, before writing a
> lot
> > > of
> > > > > > code,
> > > > > > >>>> despite not having a clear picture of what it will be and
> any
> > > > > > >> performance
> > > > > > >>>> data:
> > > > > > >>>> 1) somebody could already work on this problem privately in
> > > > > parallel,
> > > > > > >> it
> > > > > > >>>> allows to avoid clash in people's works
> > > > > > >>>> 2) some people could quickly think about the problem field
> and
> > > > share
> > > > > > >>>> high-level ideas that could wildly change the direction in
> > which
> > > > the
> > > > > > >>> author
> > > > > > >>>> (the person who is going to write code) will move in his
> work
> > > from
> > > > > > >> early
> > > > > > >>> on.
> > > > > > >>>>
> > > > > > >>>> Jihoon in
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> > > > > > >>> and
> > > > > > >>>> later Gian in this thread suggested that _every_ piece of
> work
> > > > that
> > > > > > >>> should
> > > > > > >>>> be labelled as "Design Review" according to the current
> rules
> > > > should
> > > > > > be
> > > > > > >>>> accompanied by an issue. I don't agree with this, there are
> > some
> > > > PRs
> > > > > > as
> > > > > > >>>> small as a few dozens of lines of code, that add some
> > > > configuration
> > > > > > >>>> parameter and therefore should be labelled "Design Review".
> I
> > > > don't
> > > > > > >>> thing a
> > > > > > >>>> separate proposal issue is needed for them, and even for a
> > > little
> > > > > > >> larger
> > > > > > >>>> PRs too.
> > > > > > >>>>
> > > > > > >>>> For the same reason, I also don't see the point of renaming
> > > > "Design
> > > > > > >>>> Review" into "Proposal", as well as separating "Design
> Review"
> > > > into
> > > > > > >>>> "Proposal" and something like "API Review". I think a single
> > > > "Design
> > > > > > >>>> Review" tag handles it well.
> > > > > > >>>>
> > > > > > >>>> Gian mentioned an idea that PRs that follow a "Design
> Review"
> > > > > proposal
> > > > > > >>>> issue shouldn't be "Design Review" themselves. I don't agree
> > > with
> > > > > > >> this, I
> > > > > > >>>> think that actual code and performance data are important
> > inputs
> > > > > that
> > > > > > >>>> should be re-evaluated at least by two people. I even think
> > that
> > > > > it's
> > > > > > >>> very
> > > > > > >>>> desirable that at least two people read _each line of
> > production
> > > > > code_
> > > > > > >> in
> > > > > > >>>> large PRs, although it's not what was done historically in
> > > Druid,
> > > > > > >> because
> > > > > > >>>> large bodies of newly added code, with whole new classes and
> > > > > > subsystems
> > > > > > >>>> added, are also coincidentally tested worse than already
> > > existing
> > > > > > >> classes
> > > > > > >>>> and subsystems, including in production. It seems to me that
> > > those
> > > > > > huge
> > > > > > >>>> code influxes is a major source of bugs, that could later
> take
> > > > years
> > > > > > to
> > > > > > >>>> squeeze out from the codebase.
> > > > > > >>>>
> > > > > > >>>> On Wed, 9 Jan 2019 at 08:24, Clint Wylie <
> > clint.wylie@imply.io>
> > > > > > wrote:
> > > > > > >>>>
> > > > > > >>>>> Apologies for the delayed response.
> > > > > > >>>>>
> > > > > > >>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > > > > > >>> slim.bouguerra@gmail.com>
> > > > > > >>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> I am wondering here what is the case where code first is
> > > better?
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>> I don't think it's wrong to share ideas as early as
> possible,
> > > and
> > > > > > >> after
> > > > > > >>>>> this discussion I think I am in favor of it too. I just
> meant
> > > > that
> > > > > I
> > > > > > >>> don't
> > > > > > >>>>> think it's always necessarily the most productive
> discussion
> > > > until
> > > > > > >> code
> > > > > > >>>>> exists sometimes, with the types of thing I am thinking of
> > are
> > > > > almost
> > > > > > >>>>> entirely limited to cases where things might sound good to
> > > anyone
> > > > > on
> > > > > > >>> paper
> > > > > > >>>>> but in reality need a large amount of experiments conducted
> > and
> > > > > > >>>>> observations collected to determine that something is
> > actually
> > > > > worth
> > > > > > >>>>> doing,
> > > > > > >>>>> which I imagine is mainly things like reworking internals
> for
> > > > > > >>> performance
> > > > > > >>>>> improvements.
> > > > > > >>>>>
> > > > > > >>>>> In the case of my combined proposal PR, I needed to prove
> > that
> > > > the
> > > > > > >>> thing I
> > > > > > >>>>> was working on was a good idea... and it wasn't directly.
> > But I
> > > > > came
> > > > > > >> up
> > > > > > >>>>> with another idea during the course of experiment turned
> into
> > > > > > >> something
> > > > > > >>>>> compelling, so an initial proposal would have looked quite
> a
> > > lot
> > > > > > >>> different
> > > > > > >>>>> than what I ended up with. Once I had proven to myself that
> > it
> > > > was
> > > > > a
> > > > > > >>> good
> > > > > > >>>>> idea, then I was comfortable sharing with the wider
> > community.
> > > > I'm
> > > > > > not
> > > > > > >>>>> certain how this would play out in an always proposal first
> > > > model,
> > > > > > >> maybe
> > > > > > >>>>> the first proposal exists, I personally reject it after
> > > updating
> > > > > with
> > > > > > >>>>> experiment results show it's a bad idea, continue
> > experimenting
> > > > and
> > > > > > >>> raise
> > > > > > >>>>> a
> > > > > > >>>>> new one after the experiments start looking promising?
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>>> Let's not be naive this is very rare that a contributor
> will
> > > > > accept
> > > > > > >>> that
> > > > > > >>>>>> his work is to be thrown, usually devs takes coding as
> > > personal
> > > > > > >>> creation
> > > > > > >>>>>> and they get attached to it.
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>> I agree, just because a handful of the committers have this
> > > > > attitude,
> > > > > > >> it
> > > > > > >>>>> isn't fair to expect the wider community to also, that's
> why
> > I
> > > am
> > > > > in
> > > > > > >>> favor
> > > > > > >>>>> of formalizing the process.
> > > > > > >>>>>
> > > > > > >>>>> Can you please explain what is overbearing ? what can be
> > > changed
> > > > to
> > > > > > >> make
> > > > > > >>>>> it
> > > > > > >>>>>> easy ?
> > > > > > >>>>>> Most of the points are kind of the actual questions that
> you
> > > > want
> > > > > to
> > > > > > >>>>>> address before hand anyway isn't ?
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>> Sorry for the confusion, I said it's "not overbearing", I
> > think
> > > > > it's
> > > > > > >>> fine.
> > > > > > >>>>>
> > > > > > >>>>> What are the complaints ?
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> Is this and other previous threads not a complaint about
> > > opening
> > > > a
> > > > > > >> large
> > > > > > >>>>> PR
> > > > > > >>>>> without a proposal? :) I just mean that formalizing the
> > > process,
> > > > > even
> > > > > > >>> if a
> > > > > > >>>>> proposal has a reference PR opened with it near
> concurrently,
> > > > could
> > > > > > >>>>> prevent
> > > > > > >>>>> these discussions from happening in the future because
> ground
> > > > rules
> > > > > > >> have
> > > > > > >>>>> been set and we are all on the same page I guess.
> > > > > > >>>>>
> > > > > > >>>>> I think we should also probably consider renaming the
> "design
> > > > > review"
> > > > > > >>>>> label
> > > > > > >>>>> to "proposal" or something to make it more clear that a PR
> is
> > > > > > >> associated
> > > > > > >>>>> with a proposal. It might also be worth considering using
> > > github
> > > > > > >>> projects
> > > > > > >>>>> for particularly large things that involve multiple follow
> up
> > > > PRs,
> > > > > > >> but I
> > > > > > >>>>> haven't used them in much depth to know if they add
> anything.
> > > > > > >>>>>
> > > > > > >>>>> It seems like we are all converging on agreement to do a
> > github
> > > > > issue
> > > > > > >>>>> proposal for larger changes (I would vote for announcing
> them
> > > on
> > > > > the
> > > > > > >> dev
> > > > > > >>>>> list too for more visibility), so that design review is
> > > separated
> > > > > > from
> > > > > > >>>>> code
> > > > > > >>>>> review. I guess my main concern was not wanting to
> discourage
> > > > > > >>>>> experimentation by walling it off behind mandatory
> > discussions,
> > > > but
> > > > > > >> the
> > > > > > >>>>> more I think about it,  it doesn't technically change much
> > > about
> > > > > this
> > > > > > >>>>> process, it just requires a more formal proposal to
> accompany
> > > any
> > > > > > >>>>> experiments that are shared as a PR.
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> On Tue, Jan 8, 2019 at 12:28 PM Gian Merlino <
> > gian@apache.org>
> > > > > > wrote:
> > > > > > >>>>>
> > > > > > >>>>>> I think for us, choosing to use GitHub issues as
> discussion
> > > > > threads
> > > > > > >>> for
> > > > > > >>>>>> potential 'major' contributions would be a good idea,
> > > especially
> > > > > if
> > > > > > >> we
> > > > > > >>>>>> encourage people to start them before PRs show up.
> > Definitely
> > > > > agree
> > > > > > >>> that
> > > > > > >>>>>> all contributors should go through the same process -- I
> > > > couldn't
> > > > > > >> see
> > > > > > >>> it
> > > > > > >>>>>> working well any other way.
> > > > > > >>>>>>
> > > > > > >>>>>> On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <
> > jhyde@apache.org
> > > >
> > > > > > >> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> Statically, yes, GitHub PRs are the same as GitHub cases.
> > But
> > > > > > >>>>>> dynamically,
> > > > > > >>>>>>> they are different, because you can only log a PR when
> you
> > > have
> > > > > > >>>>> finished
> > > > > > >>>>>>> work.
> > > > > > >>>>>>>
> > > > > > >>>>>>> A lot of other Apache projects use JIRA, so there is a
> > clear
> > > > > > >>>>> distinction
> > > > > > >>>>>>> between cases and contributions. JIRA cases, especially
> > when
> > > > > > >> logged
> > > > > > >>>>> early
> > > > > > >>>>>>> in the lifecycle of a contribution, become long-running
> > > > > > >> conversation
> > > > > > >>>>>>> threads with a lot of community participation. If the
> Druid
> > > > chose
> > > > > > >> to
> > > > > > >>>>> do
> > > > > > >>>>>> so,
> > > > > > >>>>>>> GitHub cases could be the same.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Be careful that you do not treat “potential contributors”
> > (by
> > > > > > >> which
> > > > > > >>> I
> > > > > > >>>>>>> presume you mean non-committers) differently from
> > committers
> > > > and
> > > > > > >> PMC
> > > > > > >>>>>>> members. Anyone starting a major piece of work should
> > follow
> > > > the
> > > > > > >>> same
> > > > > > >>>>>>> process. (Experienced committers probably have a somewhat
> > > > better
> > > > > > >>> idea
> > > > > > >>>>>> what
> > > > > > >>>>>>> work will turn out to be “major”, so they get a little
> more
> > > > > > >> leeway.)
> > > > > > >>>>>>>
> > > > > > >>>>>>> Julian
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>> On Jan 7, 2019, at 12:10 PM, Gian Merlino <
> > gian@apache.org>
> > > > > > >>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I don't think there's a need to raise issues for every
> > > > change: a
> > > > > > >>>>> small
> > > > > > >>>>>>> bug
> > > > > > >>>>>>>> fix or doc fix should just go straight to PR. (GitHub
> PRs
> > > show
> > > > > > >> up
> > > > > > >>> as
> > > > > > >>>>>>> issues
> > > > > > >>>>>>>> in the issue-search UI/API, so it's not like this means
> > the
> > > > > > >> patch
> > > > > > >>>>> has
> > > > > > >>>>>> no
> > > > > > >>>>>>>> corresponding issue -- in a sense the PR _is_ the
> issue.)
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I do think it makes sense to encourage potential
> > > contributors
> > > > to
> > > > > > >>>>> write
> > > > > > >>>>>> to
> > > > > > >>>>>>>> the dev list or raise an issue if they aren't sure if
> > > > something
> > > > > > >>>>> would
> > > > > > >>>>>>> need
> > > > > > >>>>>>>> to go through a more heavy weight process.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Fwiw we do have a set of 'design review' criteria
> already
> > > (we
> > > > > > >> had
> > > > > > >>> a
> > > > > > >>>>>>>> discussion about this a couple years ago) at:
> > > > > > >>>>>>>>
> http://druid.io/community/#getting-your-changes-accepted.
> > > So
> > > > we
> > > > > > >>>>>>> wouldn't be
> > > > > > >>>>>>>> starting from zero on defining that. We set it up back
> > when
> > > we
> > > > > > >>> were
> > > > > > >>>>>>> trying
> > > > > > >>>>>>>> to _streamline_ our process -- we used to require two
> > > > non-author
> > > > > > >>> +1s
> > > > > > >>>>>> for
> > > > > > >>>>>>>> _every_ change, even minor ones. The introduction of
> > design
> > > > > > >> review
> > > > > > >>>>>>> criteria
> > > > > > >>>>>>>> was meant to classify which PRs need that level of
> review
> > > and
> > > > > > >>> which
> > > > > > >>>>>> ones
> > > > > > >>>>>>>> are minor and can be merged with less review. I do think
> > it
> > > > > > >> helped
> > > > > > >>>>> with
> > > > > > >>>>>>>> getting minor PRs merged more quickly. The list of
> > criteria
> > > > is,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> - Major architectural changes or API changes
> > > > > > >>>>>>>> - HTTP requests and responses (e. g. a new HTTP
> endpoint)
> > > > > > >>>>>>>> - Interfaces for extensions
> > > > > > >>>>>>>> - Server configuration (e. g. altering the behavior of a
> > > > config
> > > > > > >>>>>> property)
> > > > > > >>>>>>>> - Emitted metrics
> > > > > > >>>>>>>> - Other major changes, judged by the discretion of Druid
> > > > > > >>> committers
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Some of it is subjective, but it has been in place for a
> > > > while,
> > > > > > >> so
> > > > > > >>>>> it's
> > > > > > >>>>>>> at
> > > > > > >>>>>>>> least something we are relatively familiar with.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <
> > > jhyde@apache.org
> > > > >
> > > > > > >>>>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Small contributions don’t need any design review,
> whereas
> > > > large
> > > > > > >>>>>>>>> contributions need significant review. I don’t think we
> > > > should
> > > > > > >>>>> require
> > > > > > >>>>>>> an
> > > > > > >>>>>>>>> additional step for those (many) small contributions.
> But
> > > who
> > > > > > >>>>> decides
> > > > > > >>>>>>>>> whether a contribution fits into the small or large
> > > category?
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> I think the solution is for authors to log a case (or
> > send
> > > an
> > > > > > >>>>> email to
> > > > > > >>>>>>>>> dev) before they start work on any contribution. Then
> > > > > > >> committers
> > > > > > >>>>> can
> > > > > > >>>>>>>>> request a more heavy-weight process if they think it is
> > > > needed.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Julian
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <
> > > gian@apache.org>
> > > > > > >>>>> wrote:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> It sounds like splitting design from code review is a
> > > common
> > > > > > >>> theme
> > > > > > >>>>>> in a
> > > > > > >>>>>>>>> few
> > > > > > >>>>>>>>>> of the posts here. How does everyone feel about
> making a
> > > > point
> > > > > > >>> of
> > > > > > >>>>>>>>>> encouraging design reviews to be done as issues,
> > separate
> > > > from
> > > > > > >>> the
> > > > > > >>>>>> pull
> > > > > > >>>>>>>>>> request, with the expectations that (1) the design
> > review
> > > > > > >> issue
> > > > > > >>>>>>>>>> ("proposal") should generally appear somewhat _before_
> > the
> > > > > > >> pull
> > > > > > >>>>>>> request;
> > > > > > >>>>>>>>>> (2) pull requests should _not_ have design review
> happen
> > > on
> > > > > > >>> them,
> > > > > > >>>>>>> meaning
> > > > > > >>>>>>>>>> there should no longer be PRs with design review tags,
> > and
> > > > we
> > > > > > >>>>> should
> > > > > > >>>>>>> move
> > > > > > >>>>>>>>>> the design review approval process to the issue rather
> > > than
> > > > > > >> the
> > > > > > >>>>> PR.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> For (1), even if we encourage design review
> discussions
> > to
> > > > > > >> start
> > > > > > >>>>>>> before a
> > > > > > >>>>>>>>>> pull request appears, I don't see an issue with them
> > > running
> > > > > > >>>>>>> concurrently
> > > > > > >>>>>>>>>> for a while at some point.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <
> > > > > > >> jonwei@apache.org>
> > > > > > >>>>>> wrote:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Thanks for raising these concerns!
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> My initial thoughts:
> > > > > > >>>>>>>>>>> - I agree that separation of design review and
> > code-level
> > > > > > >>> review
> > > > > > >>>>> for
> > > > > > >>>>>>>>> major
> > > > > > >>>>>>>>>>> changes would be more efficient
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> - I agree that a clear, more formalized process for
> > > > handling
> > > > > > >>>>> major
> > > > > > >>>>>>>>> changes
> > > > > > >>>>>>>>>>> would be helpful for contributors:
> > > > > > >>>>>>>>>>> - Define what is considered a major change
> > > > > > >>>>>>>>>>> - Define a standard proposal structure, KIP-style
> > > proposal
> > > > > > >>> format
> > > > > > >>>>>>>>> sounds
> > > > > > >>>>>>>>>>> good to me
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> - I think it's too rigid to have a policy of "no code
> > at
> > > > all
> > > > > > >>> with
> > > > > > >>>>>> the
> > > > > > >>>>>>>>>>> initial proposal"
> > > > > > >>>>>>>>>>> - Code samples can be useful references for
> > understanding
> > > > > > >>> aspects
> > > > > > >>>>>> of a
> > > > > > >>>>>>>>>>> design
> > > > > > >>>>>>>>>>> - In some cases it's necessary to run experiments to
> > > fully
> > > > > > >>>>>> understand
> > > > > > >>>>>>> a
> > > > > > >>>>>>>>>>> problem and determine an appropriate design, or to
> > > > determine
> > > > > > >>>>> whether
> > > > > > >>>>>>>>>>> something is even worth doing before committing to
> the
> > > work
> > > > > > >> of
> > > > > > >>>>>>> fleshing
> > > > > > >>>>>>>>> out
> > > > > > >>>>>>>>>>> a proposal, prototype code is a natural outcome of
> that
> > > and
> > > > > > >> I'm
> > > > > > >>>>> not
> > > > > > >>>>>>>>> against
> > > > > > >>>>>>>>>>> someone providing such code for reference
> > > > > > >>>>>>>>>>> - I tend to view design/code as things that are often
> > > > > > >> developed
> > > > > > >>>>>>>>>>> simultaneously in an intertwined way
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Let's not be naive this is very rare that a
> > contributor
> > > > will
> > > > > > >>>>> accept
> > > > > > >>>>>>>>> that
> > > > > > >>>>>>>>>>> his work is to be thrown, usually devs takes coding
> as
> > > > > > >> personal
> > > > > > >>>>>>> creation
> > > > > > >>>>>>>>>>> and they get attached to it.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> If we have a clear review process that emphasizes the
> > > need
> > > > > > >> for
> > > > > > >>>>> early
> > > > > > >>>>>>>>>>> consensus building, with separate design and code
> > review,
> > > > > > >> then
> > > > > > >>> I
> > > > > > >>>>>> feel
> > > > > > >>>>>>>>> we've
> > > > > > >>>>>>>>>>> done enough and don't need a hard rule against having
> > > some
> > > > > > >> code
> > > > > > >>>>>> linked
> > > > > > >>>>>>>>> with
> > > > > > >>>>>>>>>>> the initial proposal. If a potential contributor then
> > > still
> > > > > > >>>>> wants to
> > > > > > >>>>>>> go
> > > > > > >>>>>>>>>>> ahead and write a lot of code that may be rejected or
> > > > change
> > > > > > >>>>>>>>> significantly,
> > > > > > >>>>>>>>>>> the risks were made clear.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Once code is written hard to think abstract.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I can see the validity of the concern, but I
> personally
> > > > don't
> > > > > > >>>>> see it
> > > > > > >>>>>>> as
> > > > > > >>>>>>>>> a
> > > > > > >>>>>>>>>>> huge risk. My impression from the Druid PR reviews
> I've
> > > > seen
> > > > > > >> is
> > > > > > >>>>> that
> > > > > > >>>>>>> our
> > > > > > >>>>>>>>>>> reviewers are able to keep abstract design vs.
> > > > implementation
> > > > > > >>>>>> details
> > > > > > >>>>>>>>>>> separate and consider alternate designs when
> reviewing.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> To summarize I think it's probably enough to have a
> > > policy
> > > > > > >>> along
> > > > > > >>>>> the
> > > > > > >>>>>>>>> lines
> > > > > > >>>>>>>>>>> of:
> > > > > > >>>>>>>>>>> - Create more formalized guidelines for proposals and
> > > what
> > > > > > >>>>> changes
> > > > > > >>>>>>>>> require
> > > > > > >>>>>>>>>>> proposals
> > > > > > >>>>>>>>>>> - Separate design and code review for major changes,
> > with
> > > > > > >>> design
> > > > > > >>>>>>> review
> > > > > > >>>>>>>>>>> first, code-level review after reaching consensus on
> > the
> > > > > > >>> design.
> > > > > > >>>>>>>>>>> - Code before the design review is completed is just
> > for
> > > > > > >>>>> reference,
> > > > > > >>>>>>> not
> > > > > > >>>>>>>>>>> regarded as a candidate for review/merging.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> - Jon
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > > > > > >>>>>>>>> slim.bouguerra@gmail.com>
> > > > > > >>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <
> > > > > > >>>>> clint.wylie@imply.io>
> > > > > > >>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> I am definitely biased in this matter as an owner
> of
> > > > > > >> another
> > > > > > >>>>> large
> > > > > > >>>>>>> PR
> > > > > > >>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>> wasn't preceded by a direct proposal or dev list
> > > > > > >> discussion,
> > > > > > >>>>> and
> > > > > > >>>>>> in
> > > > > > >>>>>>>>>>>> general
> > > > > > >>>>>>>>>>>>> I agree that proposal first is usually better, but
> I
> > > > think
> > > > > > >> in
> > > > > > >>>>> some
> > > > > > >>>>>>>>>>> rarer
> > > > > > >>>>>>>>>>>>> cases approaching a problem code first *is* the
> most
> > > > > > >>>>> appropriate
> > > > > > >>>>>> way
> > > > > > >>>>>>>>> to
> > > > > > >>>>>>>>>>>>> have a discussion.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I am wondering here what is the case where code
> first
> > is
> > > > > > >>> better?
> > > > > > >>>>>>>>>>>> In general when you are writing code you have an
> idea
> > > > about
> > > > > > >>> what
> > > > > > >>>>>> you
> > > > > > >>>>>>>>> want
> > > > > > >>>>>>>>>>>> to change, why you want to change and why you want
> to
> > > > change
> > > > > > >>> it.
> > > > > > >>>>>>>>>>>> I do not see what is wrong with sharing this
> primitive
> > > > ideas
> > > > > > >>> and
> > > > > > >>>>>>>>> thoughts
> > > > > > >>>>>>>>>>>> as an abstract proposal (at least to avoid
> > overlapping)
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I see nothing wrong with it so long as the author
> > > > > > >>>>>>>>>>>>> accepts that the PR is treated as a combined
> proposal
> > > and
> > > > > > >>>>> proof of
> > > > > > >>>>>>>>>>>> concept,
> > > > > > >>>>>>>>>>>>> and fair game to be radically changed via
> discussion
> > or
> > > > > > >> even
> > > > > > >>>>>>> rejected,
> > > > > > >>>>>>>>>>>>> which sounds like Gian's attitude on the matter and
> > is
> > > > mine
> > > > > > >>> as
> > > > > > >>>>>> well
> > > > > > >>>>>>>>>>> with
> > > > > > >>>>>>>>>>>> my
> > > > > > >>>>>>>>>>>>> compression stuff.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Let's not be naive this is very rare that a
> > contributor
> > > > will
> > > > > > >>>>> accept
> > > > > > >>>>>>>>> that
> > > > > > >>>>>>>>>>>> his work is to be thrown, usually devs takes coding
> as
> > > > > > >>> personal
> > > > > > >>>>>>>>> creation
> > > > > > >>>>>>>>>>>> and they get attached to it.
> > > > > > >>>>>>>>>>>> To my point you can take a look on some old issue in
> > the
> > > > > > >> Druid
> > > > > > >>>>>> forum
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> > > > > > >>>>>>>>>>>> and am sure other communities have similar problems.
> > > > > > >>>>>>>>>>>> So leaving the door open to some side cases is not a
> > > good
> > > > > > >> idea
> > > > > > >>>>> in
> > > > > > >>>>>> my
> > > > > > >>>>>>>>>>>> opinion and will lead to similar issue in the
> future.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> This seems to me especially likely to happen in
> cases
> > > > > > >>>>>>>>>>>>> where an approach still needs proven to be a viable
> > > idea
> > > > > > >> *to
> > > > > > >>>>> the
> > > > > > >>>>>>>>>>> author*,
> > > > > > >>>>>>>>>>>>> so that a much more productive discussion can be
> had
> > in
> > > > the
> > > > > > >>>>> first
> > > > > > >>>>>>>>>>> place.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> I think there is a trade off, I don't think we want
> > to
> > > > > > >>>>> discourage
> > > > > > >>>>>>>>>>>>> experimentation by walling it off behind mandatory
> > > > > > >>> discussions
> > > > > > >>>>>>> before
> > > > > > >>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>> can even start, but I do think formalizing the
> > process
> > > > for
> > > > > > >>>>> large
> > > > > > >>>>>>>>>>> changes
> > > > > > >>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>> a good thing, especially since we probably can't
> > expect
> > > > the
> > > > > > >>>>> wider
> > > > > > >>>>>>>>>>>> community
> > > > > > >>>>>>>>>>>>> to have the same attitude towards a large PR
> getting
> > > > > > >>> discarded
> > > > > > >>>>> as
> > > > > > >>>>>> a
> > > > > > >>>>>>>>>>>>> committer might. I think the Kafka approach is
> > > > reasonable,
> > > > > > >> a
> > > > > > >>>>> bit
> > > > > > >>>>>>> more
> > > > > > >>>>>>>>>>>>> formal than our design review process but not
> > > > overbearing.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Can you please explain what is overbearing ? what
> can
> > be
> > > > > > >>>>> changed to
> > > > > > >>>>>>>>> make
> > > > > > >>>>>>>>>>> it
> > > > > > >>>>>>>>>>>> easy ?
> > > > > > >>>>>>>>>>>> Most of the points are kind of the actual questions
> > that
> > > > you
> > > > > > >>>>> want
> > > > > > >>>>>> to
> > > > > > >>>>>>>>>>>> address before hand anyway isn't ?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Going code first
> > > > > > >>>>>>>>>>>>> should be in general discouraged, but when it does
> > > > happen,
> > > > > > >> it
> > > > > > >>>>>> seems
> > > > > > >>>>>>>>>>> like
> > > > > > >>>>>>>>>>>>> opening DIP/an issue/starting a mailing list thread
> > or
> > > > > > >>>>> whatever we
> > > > > > >>>>>>> go
> > > > > > >>>>>>>>>>>> with
> > > > > > >>>>>>>>>>>>> to have a more high level design discussion
> alongside
> > > the
> > > > > > >>>>>> reference
> > > > > > >>>>>>> PR
> > > > > > >>>>>>>>>>>>> could alleviate some of these complaints?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> What are the complaints ?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> +1 for "DIP" heh, I think making
> > > > > > >>>>>>>>>>>>> them in the form of github issues is probably
> > > > appropriate,
> > > > > > >>>>> with a
> > > > > > >>>>>>> dev
> > > > > > >>>>>>>>>>>> list
> > > > > > >>>>>>>>>>>>> thread to announce them perhaps?
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I think  github issue with [Proposal] header like
> > > > > > >>>>>>>>>>>>
> https://github.com/apache/incubator-druid/issues/4349
> > > is
> > > > > > >> good
> > > > > > >>>>> to
> > > > > > >>>>>> me,
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Thanks!
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <
> > > > > > >>>>> bslim@apache.org>
> > > > > > >>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Thanks everyone for interacting with this thread.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> The fact that i, Roman, Jihoon  and others in the
> > past
> > > > (FJ
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ
> > > > > > >>>>> )
> > > > > > >>>>>>>>>>>>>> raised this point indicates that PRs without a
> > > proposal
> > > > > > >> are
> > > > > > >>>>>> indeed
> > > > > > >>>>>>> an
> > > > > > >>>>>>>>>>>>> issue
> > > > > > >>>>>>>>>>>>>> and we need to solve it.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Something Similar to KIP maybe called DIPs is fine
> > > with
> > > > > > >> me.
> > > > > > >>>>>>>>>>>>>> What i strive to see is the following:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> [Step 1] formalize what is the kind of work that
> > > needs a
> > > > > > >>>>> formal
> > > > > > >>>>>>>>>>>>> Proposal, I
> > > > > > >>>>>>>>>>>>>> think Roman and Jihoon has already covered that
> > pretty
> > > > > > >> well.
> > > > > > >>>>> am
> > > > > > >>>>>> +1
> > > > > > >>>>>>> on
> > > > > > >>>>>>>>>>>>> that.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> I am strongly in favor of the separation of
> Proposal
> > > > > > >> Review
> > > > > > >>>>> and
> > > > > > >>>>>>>>>>> (later)
> > > > > > >>>>>>>>>>>>>> Code review PRs. My  main reasons:
> > > > > > >>>>>>>>>>>>>> Most importantly code reviewing will introduce lot
> > of
> > > > > > >> noise
> > > > > > >>>>> and
> > > > > > >>>>>>> will
> > > > > > >>>>>>>>>>>>>> ultimately make  the GitHub page unreadable.
> > > > > > >>>>>>>>>>>>>> Avoid overlapping of work.
> > > > > > >>>>>>>>>>>>>> Once code is written hard to think abstract.
> > > > > > >>>>>>>>>>>>>> Separate page for Design review later can always
> be
> > > used
> > > > > > >> it
> > > > > > >>>>> as a
> > > > > > >>>>>>>>>>> Design
> > > > > > >>>>>>>>>>>>>> document that is readable and code free-ish.
> > > > > > >>>>>>>>>>>>>> As i said the goal of this first round is to see
> if
> > > the
> > > > > > >>>>> community
> > > > > > >>>>>>>>>>> agree
> > > > > > >>>>>>>>>>>>>> about such change, then make the process of design
> > > more
> > > > > > >>>>> inclusive
> > > > > > >>>>>>>>>>> thus
> > > > > > >>>>>>>>>>>>>> other contributors can submit a counter proposals.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> [Step 2] IF everybody agree about that point Step
> 2
> > is
> > > > to
> > > > > > >>>>> define
> > > > > > >>>>>>>>>>> which
> > > > > > >>>>>>>>>>>>>> medium is used to Publish a primitive form of a
> CODE
> > > > FREE
> > > > > > >>>>>> Abstract
> > > > > > >>>>>>>>>>>>> Proposal
> > > > > > >>>>>>>>>>>>>> containing at least the following bullet points.
> > > > > > >>>>>>>>>>>>>> - The problem description and motivation
> > > > > > >>>>>>>>>>>>>> - Overview of the proposed change
> > > > > > >>>>>>>>>>>>>> - Operational impact (compatibility/ plans to
> > > upgrades)
> > > > > > >>> public
> > > > > > >>>>>> API
> > > > > > >>>>>>>>>>>>> changes,
> > > > > > >>>>>>>>>>>>>> configuration changes, algorithm, and so on
> > > > > > >>>>>>>>>>>>>> - Expected benefits and drawbacks
> > > > > > >>>>>>>>>>>>>> - Rationale and alternatives
> > > > > > >>>>>>>>>>>>>> - Estimate Time to Deliver if possible.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> The way i think this can be is a Github issue
> where
> > > > member
> > > > > > >>> of
> > > > > > >>>>> the
> > > > > > >>>>>>>>>>>>> community
> > > > > > >>>>>>>>>>>>>> will interact via comments and the author will be
> > > > updating
> > > > > > >>> the
> > > > > > >>>>>>>>>>>>> description
> > > > > > >>>>>>>>>>>>>> in the light of comments provided by the
> community.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> During and near the end of the design discussions
> > the
> > > > > > >>> author/s
> > > > > > >>>>>> can
> > > > > > >>>>>>>>>>>> start
> > > > > > >>>>>>>>>>>>>> writing POCs to help guide the review process this
> > > > > > >> naturally
> > > > > > >>>>> will
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>> Pull
> > > > > > >>>>>>>>>>>>>> request with actual code.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> *Now the most important thing is that we need to
> > agree
> > > > > > >> that
> > > > > > >>>>> any
> > > > > > >>>>>>> work
> > > > > > >>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>> does not align with this formal process will be
> > > ignored
> > > > > > >> and
> > > > > > >>>>> the
> > > > > > >>>>>>>>>>> author
> > > > > > >>>>>>>>>>>>> will
> > > > > > >>>>>>>>>>>>>> be asked to start with a DIP*
> > > > > > >>>>>>>>>>>>>> *That is what i meant with  “If it didn’t happen
> on
> > > the
> > > > > > >>>>> mailing
> > > > > > >>>>>>> list,
> > > > > > >>>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>>> didn’t happen.”*
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Thanks and happy coding!
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <
> > > > > > >>> gian@apache.org>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> One of the advantages I see with a more formal
> > > process
> > > > is
> > > > > > >>>>> (like
> > > > > > >>>>>>>>>>> Kafka
> > > > > > >>>>>>>>>>>>>> KIPs)
> > > > > > >>>>>>>>>>>>>>> is that it levels the playing field a bit and
> sets
> > > some
> > > > > > >>>>> ground
> > > > > > >>>>>>>>>>> rules
> > > > > > >>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>> working together. In a way it can help encourage
> > > > > > >>>>> contributions
> > > > > > >>>>>> by
> > > > > > >>>>>>>>>>>>> making
> > > > > > >>>>>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>>>> clear what is expected of potential contributors.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> We have a design review process today that is not
> > as
> > > > > > >> formal
> > > > > > >>>>> as
> > > > > > >>>>>>>>>>> KIPs,
> > > > > > >>>>>>>>>>>>> but
> > > > > > >>>>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>> somewhat heavier than the one you describe. Maybe
> > we
> > > > > > >> could
> > > > > > >>>>> tweak
> > > > > > >>>>>>>>>>> our
> > > > > > >>>>>>>>>>>>>>> current one by starting to do design reviews
> > > separately
> > > > > > >>> from
> > > > > > >>>>>> PRs.
> > > > > > >>>>>>>>>>>> i.e.,
> > > > > > >>>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>> anything that meets our 'design review' criteria,
> > do
> > > > that
> > > > > > >>> on
> > > > > > >>>>> the
> > > > > > >>>>>>>>>>> dev
> > > > > > >>>>>>>>>>>>> list
> > > > > > >>>>>>>>>>>>>>> or in a separate issue, and keep the PR focused
> on
> > > > > > >>> code-level
> > > > > > >>>>>>>>>>> stuff.
> > > > > > >>>>>>>>>>>>> That
> > > > > > >>>>>>>>>>>>>>> way we don't end up trying to do both at once.
> And
> > it
> > > > > > >> makes
> > > > > > >>>>> it
> > > > > > >>>>>>>>>>> easier
> > > > > > >>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>> start talking about design before the code is
> > ready,
> > > > > > >> which
> > > > > > >>>>> would
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>>>>>>> better.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <
> > > > > > >>> jhyde@apache.org
> > > > > > >>>>>>
> > > > > > >>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> It’s really hard to say no to a contribution
> when
> > > > > > >> someone
> > > > > > >>>>> has
> > > > > > >>>>>> put
> > > > > > >>>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>> significant amount of work.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> The following approach is simple and works
> really
> > > > well:
> > > > > > >>>>> Before
> > > > > > >>>>>>>>>>> you
> > > > > > >>>>>>>>>>>>>> start
> > > > > > >>>>>>>>>>>>>>>> work, log a case, describing the problem. When
> you
> > > > have
> > > > > > >>> some
> > > > > > >>>>>>>>>>> ideas
> > > > > > >>>>>>>>>>>>>> about
> > > > > > >>>>>>>>>>>>>>>> design, add those to the case. When you have a
> > code
> > > > > > >>> branch,
> > > > > > >>>>> add
> > > > > > >>>>>>>>>>> its
> > > > > > >>>>>>>>>>>>> URL
> > > > > > >>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>> the case. And so forth. At any point in the
> > > > proceedings,
> > > > > > >>>>> people
> > > > > > >>>>>>>>>>> can
> > > > > > >>>>>>>>>>>>>> chime
> > > > > > >>>>>>>>>>>>>>>> in with their opinions.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> In my opinion, a formal “design review” process
> is
> > > not
> > > > > > >>>>>> necessary.
> > > > > > >>>>>>>>>>>>> Just
> > > > > > >>>>>>>>>>>>>>>> build consensus iteratively, by starting the
> > > > > > >> conversation
> > > > > > >>>>> early
> > > > > > >>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>> process.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Julian
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <
> > > > > > >>> gian@apache.org
> > > > > > >>>>>>
> > > > > > >>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> In this particular case: please consider the PR
> > as
> > > a
> > > > > > >>>>> proposal.
> > > > > > >>>>>>>>>>>>> Don't
> > > > > > >>>>>>>>>>>>>>> feel
> > > > > > >>>>>>>>>>>>>>>>> like just because there is code there that
> takes
> > a
> > > > > > >>> certain
> > > > > > >>>>>>>>>>>>> approach,
> > > > > > >>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>> the approach is somehow sacred. I had to
> > implement
> > > > > > >>>>> something
> > > > > > >>>>>> to
> > > > > > >>>>>>>>>>>>>>>> crystallize
> > > > > > >>>>>>>>>>>>>>>>> my own thinking about how the problem could be
> > > > > > >>> approached.
> > > > > > >>>>> I
> > > > > > >>>>>>>>>>>> won't
> > > > > > >>>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>>>>>> disappointed if, as a community, we decide a
> > > > different
> > > > > > >>>>>>>>>>> direction
> > > > > > >>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>>> better
> > > > > > >>>>>>>>>>>>>>>>> and the code all gets thrown away. That's one
> of
> > > the
> > > > > > >>>>> reasons
> > > > > > >>>>>>>>>>>> that I
> > > > > > >>>>>>>>>>>>>>>> removed
> > > > > > >>>>>>>>>>>>>>>>> the 0.14.0 milestone that was added to the
> patch.
> > > (I
> > > > > > >>> don't
> > > > > > >>>>>> want
> > > > > > >>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>> rush
> > > > > > >>>>>>>>>>>>>>>> it,
> > > > > > >>>>>>>>>>>>>>>>> nor do I think that's a good idea.)
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> In general: Sounds like we could do with some
> > more
> > > > > > >>>>>>>>>>> formalization
> > > > > > >>>>>>>>>>>>>> around
> > > > > > >>>>>>>>>>>>>>>>> what a proposal looks like, which sorts of
> > changes
> > > > need
> > > > > > >>>>> one,
> > > > > > >>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>> when
> > > > > > >>>>>>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>>>>>> the dev cycle it is appropriate. FWIW I think
> > > Kafka's
> > > > > > >>>>> process
> > > > > > >>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>> more
> > > > > > >>>>>>>>>>>>>>> or
> > > > > > >>>>>>>>>>>>>>>>> less fine, and would be okay with adopting it
> for
> > > > Druid
> > > > > > >>> if
> > > > > > >>>>>>>>>>> people
> > > > > > >>>>>>>>>>>>>> like
> > > > > > >>>>>>>>>>>>>>>> it.
> > > > > > >>>>>>>>>>>>>>>>> Right now our standards for what requires a
> > "design
> > > > > > >>> review"
> > > > > > >>>>>> are
> > > > > > >>>>>>>>>>>>> very
> > > > > > >>>>>>>>>>>>>>>>> similar to the Kafka community standards for
> what
> > > > > > >>> requires
> > > > > > >>>>> a
> > > > > > >>>>>>>>>>> KIP,
> > > > > > >>>>>>>>>>>>> so
> > > > > > >>>>>>>>>>>>>> we
> > > > > > >>>>>>>>>>>>>>>>> have some familiarity with those concepts.
> > However
> > > we
> > > > > > >>> don't
> > > > > > >>>>>>>>>>>>> separate
> > > > > > >>>>>>>>>>>>>> PR
> > > > > > >>>>>>>>>>>>>>>>> review and proposal discussion as strictly as
> > they
> > > > do,
> > > > > > >>>>> which
> > > > > > >>>>>>>>>>>> seems
> > > > > > >>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>>>>>> the foundation for the feeling of exclusion
> that
> > is
> > > > > > >> being
> > > > > > >>>>> felt
> > > > > > >>>>>>>>>>>>> here.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Separately: I just redid the description on
> > > > > > >>>>>>>>>>>>>>>>>
> > > https://github.com/apache/incubator-druid/pull/6794
> > > > to
> > > > > > >>> be
> > > > > > >>>>>> more
> > > > > > >>>>>>>>>>>>>>>> proposal-y.
> > > > > > >>>>>>>>>>>>>>>>> I followed the KIP style:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > >>>>>>>>>>>>>>>> .
> > > > > > >>>>>>>>>>>>>>>>> Please refresh the page and see if it looks
> more
> > > > > > >> useful.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Gian
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <
> > > > > > >>>>> jhyde@apache.org
> > > > > > >>>>>>>
> > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Slim,
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> I agree with your points that offline
> > development
> > > is
> > > > > > >> bad
> > > > > > >>>>> for
> > > > > > >>>>>>>>>>>>>>> community.
> > > > > > >>>>>>>>>>>>>>>>>> But I don’t think you need much mentor help.
> You
> > > > have
> > > > > > >>>>> raised
> > > > > > >>>>>>>>>>>> valid
> > > > > > >>>>>>>>>>>>>>>> issues
> > > > > > >>>>>>>>>>>>>>>>>> and the Druid community needs to decide what
> its
> > > > > > >>>>> development
> > > > > > >>>>>>>>>>>>>> practices
> > > > > > >>>>>>>>>>>>>>>>>> should be.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Julian
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
> > > > > > >>>>>>>>>>> bslim@apache.org>
> > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Hello everyone and hope you all have very
> good
> > > > > > >>> holidays.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> First, this email is not directed on the
> author
> > > or
> > > > > > >> the
> > > > > > >>> PR
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > https://github.com/apache/incubator-druid/pull/6794
> > > > > > >>> it
> > > > > > >>>>>>>>>>> self,
> > > > > > >>>>>>>>>>>>> but
> > > > > > >>>>>>>>>>>>>> i
> > > > > > >>>>>>>>>>>>>>>> see
> > > > > > >>>>>>>>>>>>>>>>>>> this PR as a perfect example.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> One of the foundation of Apache Way or what i
> > > would
> > > > > > >>>>> simply
> > > > > > >>>>>>>>>>> call
> > > > > > >>>>>>>>>>>>>> open
> > > > > > >>>>>>>>>>>>>>>>>> source
> > > > > > >>>>>>>>>>>>>>>>>>> community driven development is that
> "Technical
> > > > > > >>> decisions
> > > > > > >>>>>> are
> > > > > > >>>>>>>>>>>>>>>> discussed,
> > > > > > >>>>>>>>>>>>>>>>>>> decided, and archived publicly.
> > > > > > >>>>>>>>>>>>>>>>>>> developpement"
> > > > > > >>>>>>>>>>>>>>>>>>> Which means that big technical  changes such
> as
> > > the
> > > > > > >> one
> > > > > > >>>>>>>>>>> brought
> > > > > > >>>>>>>>>>>>> by
> > > > > > >>>>>>>>>>>>>>>> #/6794
> > > > > > >>>>>>>>>>>>>>>>>>> should have started as a proposal and round
> of
> > > > > > >>>>> discussions
> > > > > > >>>>>>>>>>>> about
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>> major
> > > > > > >>>>>>>>>>>>>>>>>>> changes designs not as 11K line of code.
> > > > > > >>>>>>>>>>>>>>>>>>> I believe such openness will promote a lot of
> > > good
> > > > > > >>>>> benefits
> > > > > > >>>>>>>>>>>> such
> > > > > > >>>>>>>>>>>>>> as:
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> - ensures community health and growth.
> > > > > > >>>>>>>>>>>>>>>>>>> - ensures everyone can participate not only
> the
> > > > > > >> authors
> > > > > > >>>>> and
> > > > > > >>>>>>>>>>> his
> > > > > > >>>>>>>>>>>>>>>>>> co-workers.
> > > > > > >>>>>>>>>>>>>>>>>>> - ensures that the project is driven by the
> > > > community
> > > > > > >>> and
> > > > > > >>>>>>>>>>> not a
> > > > > > >>>>>>>>>>>>>> given
> > > > > > >>>>>>>>>>>>>>>>>>> company or an individual.
> > > > > > >>>>>>>>>>>>>>>>>>> - ensures that there is consensus (not saying
> > > 100%
> > > > > > >>>>>>>>>>> agreement;)
> > > > > > >>>>>>>>>>>>>>> however
> > > > > > >>>>>>>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>>>>>>>> means that all individuals will accept the
> > > current
> > > > > > >>>>> progress
> > > > > > >>>>>>>>>>> on
> > > > > > >>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>> project
> > > > > > >>>>>>>>>>>>>>>>>>> until some better proposal is put forth.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Personally such BIG offline PR makes me feel
> > > > excluded
> > > > > > >>> and
> > > > > > >>>>>>>>>>>> doesn't
> > > > > > >>>>>>>>>>>>>>> give
> > > > > > >>>>>>>>>>>>>>>>>> me a
> > > > > > >>>>>>>>>>>>>>>>>>> sense that i belong to  a community at all.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> To prevent such off list development i think
> > as a
> > > > > > >> Druid
> > > > > > >>>>>>>>>>>> Community
> > > > > > >>>>>>>>>>>>>> we
> > > > > > >>>>>>>>>>>>>>>> need
> > > > > > >>>>>>>>>>>>>>>>>>> to stick to the apache way “If it didn’t
> happen
> > > on
> > > > > > >> the
> > > > > > >>>>>>>>>>> mailing
> > > > > > >>>>>>>>>>>>>> list,
> > > > > > >>>>>>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>>>>>>>> didn’t happen.”
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> I would appreciate if some of the Apache
> mentor
> > > > help
> > > > > > >>> with
> > > > > > >>>>>>>>>>> this.
> > > > > > >>>>>>>>>>>>>>>>>>> Thanks
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>
> > > > ---------------------------------------------------------------------
> > > > > > >>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > > > > > >>> dev-unsubscribe@druid.apache.org
> > > > > > >>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > >>>>> dev-help@druid.apache.org
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>
> > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > > > > > >> dev-unsubscribe@druid.apache.org
> > > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > >>> dev-help@druid.apache.org
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> --
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> B-Slim
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>
> > > > > > >>>
> > > > > >
> > > >
> > _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>
> > > > >
> ---------------------------------------------------------------------
> > > > > > >>>>>>>>> To unsubscribe, e-mail:
> dev-unsubscribe@druid.apache.org
> > > > > > >>>>>>>>> For additional commands, e-mail:
> > dev-help@druid.apache.org
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>
> > > > ---------------------------------------------------------------------
> > > > > > >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > > > >>>>>>> For additional commands, e-mail:
> dev-help@druid.apache.org
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > > > For additional commands, e-mail: dev-help@druid.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Off list major development

[Jihoon Son <gh...@gmail.com>]

Good point.
If some authors raise PRs without noticing the need for a proposal, we
shouldn't ask them to close their PRs only because of the absence of the
proposal.

"Design review" without a proposal for simple PRs would be good if we can
determine well what PRs need and what don't.
But, how do we know? Even for the same PR, someone may think it needs a
proposal but another may not.

If someone don't notice the need for a proposal and raise a PR without it,
I'm fine with that.
However, we should still encourage writing a proposal before writing code
because we can avoid unnecessary effort.

I think this kind of issue usually happens for first time contributors and
they will be better once they get used to Druid development.
And I believe someday even first contributors would follow this policy once
it gets settled down well in the community as Kafka community does.

Jihoon

On Tue, Jan 15, 2019 at 4:31 AM Roman Leventov <le...@gmail.com>
wrote:

> In such small PRs, authors likely won't be aware that they need to create a
> proposal in the first place. The first reviewer just adds the "Design
> Review" tag. It's also absolutely not about considering designs and gauging
> the proposal, it's just verifying that a configuration / parameter / HTTP
> endpoint name is reasonable and aligned with the rest of Druid. So I think
> that a separate proposal issue for such PRs is unnecessary bureaucracy.
>
> On Tue, 15 Jan 2019 at 07:45, Jihoon Son <gh...@gmail.com> wrote:
>
> > Roman,
> >
> > > Jihoon in
> >
> >
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> > and later Gian in this thread suggested that _every_ piece of work that
> > should be labelled as "Design Review" according to the current rules
> should
> > be accompanied by an issue. I don't agree with this, there are some PRs
> as
> > small as a few dozens of lines of code, that add some configuration
> > parameter and therefore should be labelled "Design Review". I don't
> thing a
> > separate proposal issue is needed for them, and even for a little larger
> > PRs too.
> >
> > What I'm concerned with is how people feel if their design is not
> accepted
> > even though they wrote code. Of course, as Clint said, sometimes code
> helps
> > better understanding of the proposal. But, I believe this is the case
> when
> > the proposal is quite complicated and not easy to understand without
> code.
> > Also the authors should be aware of that they might rewrite the entire
> code
> > if the design should be changed.
> >
> > If writing code is simple, I don't see why the authors don't wait until
> the
> > review for their proposal is finished.
> >
> > Jihoon
> >
> > On Fri, Jan 11, 2019 at 9:51 AM Fangjin Yang <fa...@imply.io> wrote:
> >
> > > I agree with Gian, as an Apache committer, your responsibility is for
> the
> > > betterment of the project. I agree it is in the best interest of the
> > > project to stop thinking about what orgs people belong to. We are all a
> > > part of the Apache software foundation, regardless of what our roles
> and
> > > titles are outside of it.
> > >
> > > On Fri, Jan 11, 2019 at 2:22 AM Roman Leventov <le...@gmail.com>
> > > wrote:
> > >
> > > > It's not that people from one org could abuse the project and push
> some
> > > > change, but that they have similar perspective (bubble effect) and
> some
> > > > important aspects of a large feature could escape their attention.
> > > >
> > > > I suggest it to be not a rigid rule, but a recommendation for authors
> > of
> > > > large proposals to try to attract reviewers from other orgs.
> > > >
> > > > On Fri, 11 Jan 2019 at 02:51, Julian Hyde <jh...@apache.org> wrote:
> > > >
> > > > > I agree with Gian.
> > > > >
> > > > > As an Apache committer, you only have one affiliation: you are
> > working
> > > in
> > > > > the best interests of the project.
> > > > >
> > > > > Obviously, in the real world there are other pressures. But we do
> our
> > > > best
> > > > > to compensate for them.
> > > > >
> > > > > Also, as a a community we try to design our process so as to avoid
> > > undue
> > > > > influences. For instance, when I advocate for logging cases early,
> I
> > am
> > > > > trying to mitigate the effect of product managers and VPs of
> > > engineering,
> > > > > who like to have their say in meeting rooms rather than on public
> > > mailing
> > > > > lists. That’s just one example; if we see other influences at play,
> > > let’s
> > > > > evolve our process to try to level the playing field.
> > > > >
> > > > > Julian
> > > > >
> > > > >
> > > > > > On Jan 10, 2019, at 10:40 AM, Gian Merlino <gi...@apache.org>
> > wrote:
> > > > > >
> > > > > >>> What do community members think about also making a requirement
> > > that
> > > > > >>> "Design Review" PRs and proposals are reviewed by at least two
> > > people
> > > > > > with
> > > > > >>> different affiliation?
> > > > > >> This seems pretty reasonable to me. I haven't found anything in
> > > Apache
> > > > > >> voting procedure docs (
> > > https://www.apache.org/foundation/voting.html)
> > > > > that
> > > > > >> seems to explicitly forbid something like this yet at least.
> > > > > >
> > > > > > On the other hand, my understanding of the Apache Way would mean
> > that
> > > > > this
> > > > > > kind of rule doesn't make sense. In particular from
> > > > > > https://www.apache.org/foundation/how-it-works.html: "We firmly
> > > > believe
> > > > > in
> > > > > > hats. Your role at the ASF is one assigned to you personally, and
> > is
> > > > > > bestowed on you by your peers. It is not tied to your job or
> > current
> > > > > > employer or company." That sentiment seems incompatible with
> making
> > > > > > explicit rules about organizational diversity in voting. I have
> > also
> > > > > heard
> > > > > > a few people say things like: people are supposed to represent
> > > > > themselves,
> > > > > > not their employers.
> > > > > >
> > > > > > Obviously, though, people's actions and opinions are influenced
> by
> > > > their
> > > > > > employer. IMO, a better way to approach a potential problem there
> > is
> > > > that
> > > > > > if people from a particular organization end up behaving
> abusively,
> > > > then
> > > > > > PMC members from other organizations (or in extreme situations
> the
> > > > Apache
> > > > > > board itself) should tell them to knock it off. And then do more
> > > > serious
> > > > > > things if the inappropriate behavior continues. This kind
> > > > > >
> > > > > > The other thing you brought up, promoting more shared ownership
> and
> > > > > > awareness, I am hopeful that separating proposals from PRs will
> > help
> > > > with
> > > > > > that. One reason is that it takes much less time to understand a
> > well
> > > > > > written proposal than it takes to understand a PR. Code is very
> > clear
> > > > but
> > > > > > it is also very verbose and takes a while to read and understand.
> > So
> > > in
> > > > > > theory changing how we operate, in this way, should promote more
> > > > > > understanding of more people across the code base.
> > > > > >
> > > > > > On Thu, Jan 10, 2019 at 1:34 AM Clint Wylie <
> clint.wylie@imply.io>
> > > > > wrote:
> > > > > >
> > > > > >>>
> > > > > >>> What do community members think about also making a requirement
> > > that
> > > > > >>> "Design Review" PRs and proposals are reviewed by at least two
> > > people
> > > > > >> with
> > > > > >>> different affiliation?
> > > > > >>
> > > > > >>
> > > > > >> This seems pretty reasonable to me. I haven't found anything in
> > > Apache
> > > > > >> voting procedure docs (
> > > https://www.apache.org/foundation/voting.html)
> > > > > that
> > > > > >> seems to explicitly forbid something like this yet at least.
> > > > > >>
> > > > > >> On Wed, Jan 9, 2019 at 8:52 PM Roman Leventov <
> > > leventov.ru@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> What do community members think about also making a requirement
> > > that
> > > > > >>> "Design Review" PRs and proposals are reviewed by at least two
> > > people
> > > > > >> with
> > > > > >>> different affiliation? IMO it's a good idea, because it ensures
> > > that
> > > > > >>> different interests are taken into account. Also it pushes
> people
> > > to
> > > > > >> engage
> > > > > >>> with work done in other parts of Druid, improving shared code
> > > owning
> > > > > and
> > > > > >>> awareness.
> > > > > >>>
> > > > > >>> Or it's against the Apache Way because it's assumed that there
> > are
> > > no
> > > > > >>> company boundaries within the community?
> > > > > >>>
> > > > > >>> On Thu, 10 Jan 2019 at 11:45, Roman Leventov <
> > > leventov.ru@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> I see two important reasons why it makes sense to file an
> issue
> > > and
> > > > > >>>> probably announce it in the mailing list, before writing a lot
> > of
> > > > > code,
> > > > > >>>> despite not having a clear picture of what it will be and any
> > > > > >> performance
> > > > > >>>> data:
> > > > > >>>> 1) somebody could already work on this problem privately in
> > > > parallel,
> > > > > >> it
> > > > > >>>> allows to avoid clash in people's works
> > > > > >>>> 2) some people could quickly think about the problem field and
> > > share
> > > > > >>>> high-level ideas that could wildly change the direction in
> which
> > > the
> > > > > >>> author
> > > > > >>>> (the person who is going to write code) will move in his work
> > from
> > > > > >> early
> > > > > >>> on.
> > > > > >>>>
> > > > > >>>> Jihoon in
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> > > > > >>> and
> > > > > >>>> later Gian in this thread suggested that _every_ piece of work
> > > that
> > > > > >>> should
> > > > > >>>> be labelled as "Design Review" according to the current rules
> > > should
> > > > > be
> > > > > >>>> accompanied by an issue. I don't agree with this, there are
> some
> > > PRs
> > > > > as
> > > > > >>>> small as a few dozens of lines of code, that add some
> > > configuration
> > > > > >>>> parameter and therefore should be labelled "Design Review". I
> > > don't
> > > > > >>> thing a
> > > > > >>>> separate proposal issue is needed for them, and even for a
> > little
> > > > > >> larger
> > > > > >>>> PRs too.
> > > > > >>>>
> > > > > >>>> For the same reason, I also don't see the point of renaming
> > > "Design
> > > > > >>>> Review" into "Proposal", as well as separating "Design Review"
> > > into
> > > > > >>>> "Proposal" and something like "API Review". I think a single
> > > "Design
> > > > > >>>> Review" tag handles it well.
> > > > > >>>>
> > > > > >>>> Gian mentioned an idea that PRs that follow a "Design Review"
> > > > proposal
> > > > > >>>> issue shouldn't be "Design Review" themselves. I don't agree
> > with
> > > > > >> this, I
> > > > > >>>> think that actual code and performance data are important
> inputs
> > > > that
> > > > > >>>> should be re-evaluated at least by two people. I even think
> that
> > > > it's
> > > > > >>> very
> > > > > >>>> desirable that at least two people read _each line of
> production
> > > > code_
> > > > > >> in
> > > > > >>>> large PRs, although it's not what was done historically in
> > Druid,
> > > > > >> because
> > > > > >>>> large bodies of newly added code, with whole new classes and
> > > > > subsystems
> > > > > >>>> added, are also coincidentally tested worse than already
> > existing
> > > > > >> classes
> > > > > >>>> and subsystems, including in production. It seems to me that
> > those
> > > > > huge
> > > > > >>>> code influxes is a major source of bugs, that could later take
> > > years
> > > > > to
> > > > > >>>> squeeze out from the codebase.
> > > > > >>>>
> > > > > >>>> On Wed, 9 Jan 2019 at 08:24, Clint Wylie <
> clint.wylie@imply.io>
> > > > > wrote:
> > > > > >>>>
> > > > > >>>>> Apologies for the delayed response.
> > > > > >>>>>
> > > > > >>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > > > > >>> slim.bouguerra@gmail.com>
> > > > > >>>>> wrote:
> > > > > >>>>>
> > > > > >>>>>> I am wondering here what is the case where code first is
> > better?
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>> I don't think it's wrong to share ideas as early as possible,
> > and
> > > > > >> after
> > > > > >>>>> this discussion I think I am in favor of it too. I just meant
> > > that
> > > > I
> > > > > >>> don't
> > > > > >>>>> think it's always necessarily the most productive discussion
> > > until
> > > > > >> code
> > > > > >>>>> exists sometimes, with the types of thing I am thinking of
> are
> > > > almost
> > > > > >>>>> entirely limited to cases where things might sound good to
> > anyone
> > > > on
> > > > > >>> paper
> > > > > >>>>> but in reality need a large amount of experiments conducted
> and
> > > > > >>>>> observations collected to determine that something is
> actually
> > > > worth
> > > > > >>>>> doing,
> > > > > >>>>> which I imagine is mainly things like reworking internals for
> > > > > >>> performance
> > > > > >>>>> improvements.
> > > > > >>>>>
> > > > > >>>>> In the case of my combined proposal PR, I needed to prove
> that
> > > the
> > > > > >>> thing I
> > > > > >>>>> was working on was a good idea... and it wasn't directly.
> But I
> > > > came
> > > > > >> up
> > > > > >>>>> with another idea during the course of experiment turned into
> > > > > >> something
> > > > > >>>>> compelling, so an initial proposal would have looked quite a
> > lot
> > > > > >>> different
> > > > > >>>>> than what I ended up with. Once I had proven to myself that
> it
> > > was
> > > > a
> > > > > >>> good
> > > > > >>>>> idea, then I was comfortable sharing with the wider
> community.
> > > I'm
> > > > > not
> > > > > >>>>> certain how this would play out in an always proposal first
> > > model,
> > > > > >> maybe
> > > > > >>>>> the first proposal exists, I personally reject it after
> > updating
> > > > with
> > > > > >>>>> experiment results show it's a bad idea, continue
> experimenting
> > > and
> > > > > >>> raise
> > > > > >>>>> a
> > > > > >>>>> new one after the experiments start looking promising?
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>> Let's not be naive this is very rare that a contributor will
> > > > accept
> > > > > >>> that
> > > > > >>>>>> his work is to be thrown, usually devs takes coding as
> > personal
> > > > > >>> creation
> > > > > >>>>>> and they get attached to it.
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>> I agree, just because a handful of the committers have this
> > > > attitude,
> > > > > >> it
> > > > > >>>>> isn't fair to expect the wider community to also, that's why
> I
> > am
> > > > in
> > > > > >>> favor
> > > > > >>>>> of formalizing the process.
> > > > > >>>>>
> > > > > >>>>> Can you please explain what is overbearing ? what can be
> > changed
> > > to
> > > > > >> make
> > > > > >>>>> it
> > > > > >>>>>> easy ?
> > > > > >>>>>> Most of the points are kind of the actual questions that you
> > > want
> > > > to
> > > > > >>>>>> address before hand anyway isn't ?
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>> Sorry for the confusion, I said it's "not overbearing", I
> think
> > > > it's
> > > > > >>> fine.
> > > > > >>>>>
> > > > > >>>>> What are the complaints ?
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> Is this and other previous threads not a complaint about
> > opening
> > > a
> > > > > >> large
> > > > > >>>>> PR
> > > > > >>>>> without a proposal? :) I just mean that formalizing the
> > process,
> > > > even
> > > > > >>> if a
> > > > > >>>>> proposal has a reference PR opened with it near concurrently,
> > > could
> > > > > >>>>> prevent
> > > > > >>>>> these discussions from happening in the future because ground
> > > rules
> > > > > >> have
> > > > > >>>>> been set and we are all on the same page I guess.
> > > > > >>>>>
> > > > > >>>>> I think we should also probably consider renaming the "design
> > > > review"
> > > > > >>>>> label
> > > > > >>>>> to "proposal" or something to make it more clear that a PR is
> > > > > >> associated
> > > > > >>>>> with a proposal. It might also be worth considering using
> > github
> > > > > >>> projects
> > > > > >>>>> for particularly large things that involve multiple follow up
> > > PRs,
> > > > > >> but I
> > > > > >>>>> haven't used them in much depth to know if they add anything.
> > > > > >>>>>
> > > > > >>>>> It seems like we are all converging on agreement to do a
> github
> > > > issue
> > > > > >>>>> proposal for larger changes (I would vote for announcing them
> > on
> > > > the
> > > > > >> dev
> > > > > >>>>> list too for more visibility), so that design review is
> > separated
> > > > > from
> > > > > >>>>> code
> > > > > >>>>> review. I guess my main concern was not wanting to discourage
> > > > > >>>>> experimentation by walling it off behind mandatory
> discussions,
> > > but
> > > > > >> the
> > > > > >>>>> more I think about it,  it doesn't technically change much
> > about
> > > > this
> > > > > >>>>> process, it just requires a more formal proposal to accompany
> > any
> > > > > >>>>> experiments that are shared as a PR.
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> On Tue, Jan 8, 2019 at 12:28 PM Gian Merlino <
> gian@apache.org>
> > > > > wrote:
> > > > > >>>>>
> > > > > >>>>>> I think for us, choosing to use GitHub issues as discussion
> > > > threads
> > > > > >>> for
> > > > > >>>>>> potential 'major' contributions would be a good idea,
> > especially
> > > > if
> > > > > >> we
> > > > > >>>>>> encourage people to start them before PRs show up.
> Definitely
> > > > agree
> > > > > >>> that
> > > > > >>>>>> all contributors should go through the same process -- I
> > > couldn't
> > > > > >> see
> > > > > >>> it
> > > > > >>>>>> working well any other way.
> > > > > >>>>>>
> > > > > >>>>>> On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <
> jhyde@apache.org
> > >
> > > > > >> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Statically, yes, GitHub PRs are the same as GitHub cases.
> But
> > > > > >>>>>> dynamically,
> > > > > >>>>>>> they are different, because you can only log a PR when you
> > have
> > > > > >>>>> finished
> > > > > >>>>>>> work.
> > > > > >>>>>>>
> > > > > >>>>>>> A lot of other Apache projects use JIRA, so there is a
> clear
> > > > > >>>>> distinction
> > > > > >>>>>>> between cases and contributions. JIRA cases, especially
> when
> > > > > >> logged
> > > > > >>>>> early
> > > > > >>>>>>> in the lifecycle of a contribution, become long-running
> > > > > >> conversation
> > > > > >>>>>>> threads with a lot of community participation. If the Druid
> > > chose
> > > > > >> to
> > > > > >>>>> do
> > > > > >>>>>> so,
> > > > > >>>>>>> GitHub cases could be the same.
> > > > > >>>>>>>
> > > > > >>>>>>> Be careful that you do not treat “potential contributors”
> (by
> > > > > >> which
> > > > > >>> I
> > > > > >>>>>>> presume you mean non-committers) differently from
> committers
> > > and
> > > > > >> PMC
> > > > > >>>>>>> members. Anyone starting a major piece of work should
> follow
> > > the
> > > > > >>> same
> > > > > >>>>>>> process. (Experienced committers probably have a somewhat
> > > better
> > > > > >>> idea
> > > > > >>>>>> what
> > > > > >>>>>>> work will turn out to be “major”, so they get a little more
> > > > > >> leeway.)
> > > > > >>>>>>>
> > > > > >>>>>>> Julian
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>> On Jan 7, 2019, at 12:10 PM, Gian Merlino <
> gian@apache.org>
> > > > > >>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>> I don't think there's a need to raise issues for every
> > > change: a
> > > > > >>>>> small
> > > > > >>>>>>> bug
> > > > > >>>>>>>> fix or doc fix should just go straight to PR. (GitHub PRs
> > show
> > > > > >> up
> > > > > >>> as
> > > > > >>>>>>> issues
> > > > > >>>>>>>> in the issue-search UI/API, so it's not like this means
> the
> > > > > >> patch
> > > > > >>>>> has
> > > > > >>>>>> no
> > > > > >>>>>>>> corresponding issue -- in a sense the PR _is_ the issue.)
> > > > > >>>>>>>>
> > > > > >>>>>>>> I do think it makes sense to encourage potential
> > contributors
> > > to
> > > > > >>>>> write
> > > > > >>>>>> to
> > > > > >>>>>>>> the dev list or raise an issue if they aren't sure if
> > > something
> > > > > >>>>> would
> > > > > >>>>>>> need
> > > > > >>>>>>>> to go through a more heavy weight process.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Fwiw we do have a set of 'design review' criteria already
> > (we
> > > > > >> had
> > > > > >>> a
> > > > > >>>>>>>> discussion about this a couple years ago) at:
> > > > > >>>>>>>> http://druid.io/community/#getting-your-changes-accepted.
> > So
> > > we
> > > > > >>>>>>> wouldn't be
> > > > > >>>>>>>> starting from zero on defining that. We set it up back
> when
> > we
> > > > > >>> were
> > > > > >>>>>>> trying
> > > > > >>>>>>>> to _streamline_ our process -- we used to require two
> > > non-author
> > > > > >>> +1s
> > > > > >>>>>> for
> > > > > >>>>>>>> _every_ change, even minor ones. The introduction of
> design
> > > > > >> review
> > > > > >>>>>>> criteria
> > > > > >>>>>>>> was meant to classify which PRs need that level of review
> > and
> > > > > >>> which
> > > > > >>>>>> ones
> > > > > >>>>>>>> are minor and can be merged with less review. I do think
> it
> > > > > >> helped
> > > > > >>>>> with
> > > > > >>>>>>>> getting minor PRs merged more quickly. The list of
> criteria
> > > is,
> > > > > >>>>>>>>
> > > > > >>>>>>>> - Major architectural changes or API changes
> > > > > >>>>>>>> - HTTP requests and responses (e. g. a new HTTP endpoint)
> > > > > >>>>>>>> - Interfaces for extensions
> > > > > >>>>>>>> - Server configuration (e. g. altering the behavior of a
> > > config
> > > > > >>>>>> property)
> > > > > >>>>>>>> - Emitted metrics
> > > > > >>>>>>>> - Other major changes, judged by the discretion of Druid
> > > > > >>> committers
> > > > > >>>>>>>>
> > > > > >>>>>>>> Some of it is subjective, but it has been in place for a
> > > while,
> > > > > >> so
> > > > > >>>>> it's
> > > > > >>>>>>> at
> > > > > >>>>>>>> least something we are relatively familiar with.
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <
> > jhyde@apache.org
> > > >
> > > > > >>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Small contributions don’t need any design review, whereas
> > > large
> > > > > >>>>>>>>> contributions need significant review. I don’t think we
> > > should
> > > > > >>>>> require
> > > > > >>>>>>> an
> > > > > >>>>>>>>> additional step for those (many) small contributions. But
> > who
> > > > > >>>>> decides
> > > > > >>>>>>>>> whether a contribution fits into the small or large
> > category?
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> I think the solution is for authors to log a case (or
> send
> > an
> > > > > >>>>> email to
> > > > > >>>>>>>>> dev) before they start work on any contribution. Then
> > > > > >> committers
> > > > > >>>>> can
> > > > > >>>>>>>>> request a more heavy-weight process if they think it is
> > > needed.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Julian
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <
> > gian@apache.org>
> > > > > >>>>> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> It sounds like splitting design from code review is a
> > common
> > > > > >>> theme
> > > > > >>>>>> in a
> > > > > >>>>>>>>> few
> > > > > >>>>>>>>>> of the posts here. How does everyone feel about making a
> > > point
> > > > > >>> of
> > > > > >>>>>>>>>> encouraging design reviews to be done as issues,
> separate
> > > from
> > > > > >>> the
> > > > > >>>>>> pull
> > > > > >>>>>>>>>> request, with the expectations that (1) the design
> review
> > > > > >> issue
> > > > > >>>>>>>>>> ("proposal") should generally appear somewhat _before_
> the
> > > > > >> pull
> > > > > >>>>>>> request;
> > > > > >>>>>>>>>> (2) pull requests should _not_ have design review happen
> > on
> > > > > >>> them,
> > > > > >>>>>>> meaning
> > > > > >>>>>>>>>> there should no longer be PRs with design review tags,
> and
> > > we
> > > > > >>>>> should
> > > > > >>>>>>> move
> > > > > >>>>>>>>>> the design review approval process to the issue rather
> > than
> > > > > >> the
> > > > > >>>>> PR.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> For (1), even if we encourage design review discussions
> to
> > > > > >> start
> > > > > >>>>>>> before a
> > > > > >>>>>>>>>> pull request appears, I don't see an issue with them
> > running
> > > > > >>>>>>> concurrently
> > > > > >>>>>>>>>> for a while at some point.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <
> > > > > >> jonwei@apache.org>
> > > > > >>>>>> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Thanks for raising these concerns!
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> My initial thoughts:
> > > > > >>>>>>>>>>> - I agree that separation of design review and
> code-level
> > > > > >>> review
> > > > > >>>>> for
> > > > > >>>>>>>>> major
> > > > > >>>>>>>>>>> changes would be more efficient
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> - I agree that a clear, more formalized process for
> > > handling
> > > > > >>>>> major
> > > > > >>>>>>>>> changes
> > > > > >>>>>>>>>>> would be helpful for contributors:
> > > > > >>>>>>>>>>> - Define what is considered a major change
> > > > > >>>>>>>>>>> - Define a standard proposal structure, KIP-style
> > proposal
> > > > > >>> format
> > > > > >>>>>>>>> sounds
> > > > > >>>>>>>>>>> good to me
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> - I think it's too rigid to have a policy of "no code
> at
> > > all
> > > > > >>> with
> > > > > >>>>>> the
> > > > > >>>>>>>>>>> initial proposal"
> > > > > >>>>>>>>>>> - Code samples can be useful references for
> understanding
> > > > > >>> aspects
> > > > > >>>>>> of a
> > > > > >>>>>>>>>>> design
> > > > > >>>>>>>>>>> - In some cases it's necessary to run experiments to
> > fully
> > > > > >>>>>> understand
> > > > > >>>>>>> a
> > > > > >>>>>>>>>>> problem and determine an appropriate design, or to
> > > determine
> > > > > >>>>> whether
> > > > > >>>>>>>>>>> something is even worth doing before committing to the
> > work
> > > > > >> of
> > > > > >>>>>>> fleshing
> > > > > >>>>>>>>> out
> > > > > >>>>>>>>>>> a proposal, prototype code is a natural outcome of that
> > and
> > > > > >> I'm
> > > > > >>>>> not
> > > > > >>>>>>>>> against
> > > > > >>>>>>>>>>> someone providing such code for reference
> > > > > >>>>>>>>>>> - I tend to view design/code as things that are often
> > > > > >> developed
> > > > > >>>>>>>>>>> simultaneously in an intertwined way
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> Let's not be naive this is very rare that a
> contributor
> > > will
> > > > > >>>>> accept
> > > > > >>>>>>>>> that
> > > > > >>>>>>>>>>> his work is to be thrown, usually devs takes coding as
> > > > > >> personal
> > > > > >>>>>>> creation
> > > > > >>>>>>>>>>> and they get attached to it.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> If we have a clear review process that emphasizes the
> > need
> > > > > >> for
> > > > > >>>>> early
> > > > > >>>>>>>>>>> consensus building, with separate design and code
> review,
> > > > > >> then
> > > > > >>> I
> > > > > >>>>>> feel
> > > > > >>>>>>>>> we've
> > > > > >>>>>>>>>>> done enough and don't need a hard rule against having
> > some
> > > > > >> code
> > > > > >>>>>> linked
> > > > > >>>>>>>>> with
> > > > > >>>>>>>>>>> the initial proposal. If a potential contributor then
> > still
> > > > > >>>>> wants to
> > > > > >>>>>>> go
> > > > > >>>>>>>>>>> ahead and write a lot of code that may be rejected or
> > > change
> > > > > >>>>>>>>> significantly,
> > > > > >>>>>>>>>>> the risks were made clear.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> Once code is written hard to think abstract.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I can see the validity of the concern, but I personally
> > > don't
> > > > > >>>>> see it
> > > > > >>>>>>> as
> > > > > >>>>>>>>> a
> > > > > >>>>>>>>>>> huge risk. My impression from the Druid PR reviews I've
> > > seen
> > > > > >> is
> > > > > >>>>> that
> > > > > >>>>>>> our
> > > > > >>>>>>>>>>> reviewers are able to keep abstract design vs.
> > > implementation
> > > > > >>>>>> details
> > > > > >>>>>>>>>>> separate and consider alternate designs when reviewing.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> To summarize I think it's probably enough to have a
> > policy
> > > > > >>> along
> > > > > >>>>> the
> > > > > >>>>>>>>> lines
> > > > > >>>>>>>>>>> of:
> > > > > >>>>>>>>>>> - Create more formalized guidelines for proposals and
> > what
> > > > > >>>>> changes
> > > > > >>>>>>>>> require
> > > > > >>>>>>>>>>> proposals
> > > > > >>>>>>>>>>> - Separate design and code review for major changes,
> with
> > > > > >>> design
> > > > > >>>>>>> review
> > > > > >>>>>>>>>>> first, code-level review after reaching consensus on
> the
> > > > > >>> design.
> > > > > >>>>>>>>>>> - Code before the design review is completed is just
> for
> > > > > >>>>> reference,
> > > > > >>>>>>> not
> > > > > >>>>>>>>>>> regarded as a candidate for review/merging.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> - Jon
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > > > > >>>>>>>>> slim.bouguerra@gmail.com>
> > > > > >>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <
> > > > > >>>>> clint.wylie@imply.io>
> > > > > >>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> I am definitely biased in this matter as an owner of
> > > > > >> another
> > > > > >>>>> large
> > > > > >>>>>>> PR
> > > > > >>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>> wasn't preceded by a direct proposal or dev list
> > > > > >> discussion,
> > > > > >>>>> and
> > > > > >>>>>> in
> > > > > >>>>>>>>>>>> general
> > > > > >>>>>>>>>>>>> I agree that proposal first is usually better, but I
> > > think
> > > > > >> in
> > > > > >>>>> some
> > > > > >>>>>>>>>>> rarer
> > > > > >>>>>>>>>>>>> cases approaching a problem code first *is* the most
> > > > > >>>>> appropriate
> > > > > >>>>>> way
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>>>>> have a discussion.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> I am wondering here what is the case where code first
> is
> > > > > >>> better?
> > > > > >>>>>>>>>>>> In general when you are writing code you have an idea
> > > about
> > > > > >>> what
> > > > > >>>>>> you
> > > > > >>>>>>>>> want
> > > > > >>>>>>>>>>>> to change, why you want to change and why you want to
> > > change
> > > > > >>> it.
> > > > > >>>>>>>>>>>> I do not see what is wrong with sharing this primitive
> > > ideas
> > > > > >>> and
> > > > > >>>>>>>>> thoughts
> > > > > >>>>>>>>>>>> as an abstract proposal (at least to avoid
> overlapping)
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> I see nothing wrong with it so long as the author
> > > > > >>>>>>>>>>>>> accepts that the PR is treated as a combined proposal
> > and
> > > > > >>>>> proof of
> > > > > >>>>>>>>>>>> concept,
> > > > > >>>>>>>>>>>>> and fair game to be radically changed via discussion
> or
> > > > > >> even
> > > > > >>>>>>> rejected,
> > > > > >>>>>>>>>>>>> which sounds like Gian's attitude on the matter and
> is
> > > mine
> > > > > >>> as
> > > > > >>>>>> well
> > > > > >>>>>>>>>>> with
> > > > > >>>>>>>>>>>> my
> > > > > >>>>>>>>>>>>> compression stuff.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Let's not be naive this is very rare that a
> contributor
> > > will
> > > > > >>>>> accept
> > > > > >>>>>>>>> that
> > > > > >>>>>>>>>>>> his work is to be thrown, usually devs takes coding as
> > > > > >>> personal
> > > > > >>>>>>>>> creation
> > > > > >>>>>>>>>>>> and they get attached to it.
> > > > > >>>>>>>>>>>> To my point you can take a look on some old issue in
> the
> > > > > >> Druid
> > > > > >>>>>> forum
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> > > > > >>>>>>>>>>>> and am sure other communities have similar problems.
> > > > > >>>>>>>>>>>> So leaving the door open to some side cases is not a
> > good
> > > > > >> idea
> > > > > >>>>> in
> > > > > >>>>>> my
> > > > > >>>>>>>>>>>> opinion and will lead to similar issue in the future.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> This seems to me especially likely to happen in cases
> > > > > >>>>>>>>>>>>> where an approach still needs proven to be a viable
> > idea
> > > > > >> *to
> > > > > >>>>> the
> > > > > >>>>>>>>>>> author*,
> > > > > >>>>>>>>>>>>> so that a much more productive discussion can be had
> in
> > > the
> > > > > >>>>> first
> > > > > >>>>>>>>>>> place.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> I think there is a trade off, I don't think we want
> to
> > > > > >>>>> discourage
> > > > > >>>>>>>>>>>>> experimentation by walling it off behind mandatory
> > > > > >>> discussions
> > > > > >>>>>>> before
> > > > > >>>>>>>>>>> it
> > > > > >>>>>>>>>>>>> can even start, but I do think formalizing the
> process
> > > for
> > > > > >>>>> large
> > > > > >>>>>>>>>>> changes
> > > > > >>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>> a good thing, especially since we probably can't
> expect
> > > the
> > > > > >>>>> wider
> > > > > >>>>>>>>>>>> community
> > > > > >>>>>>>>>>>>> to have the same attitude towards a large PR getting
> > > > > >>> discarded
> > > > > >>>>> as
> > > > > >>>>>> a
> > > > > >>>>>>>>>>>>> committer might. I think the Kafka approach is
> > > reasonable,
> > > > > >> a
> > > > > >>>>> bit
> > > > > >>>>>>> more
> > > > > >>>>>>>>>>>>> formal than our design review process but not
> > > overbearing.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Can you please explain what is overbearing ? what can
> be
> > > > > >>>>> changed to
> > > > > >>>>>>>>> make
> > > > > >>>>>>>>>>> it
> > > > > >>>>>>>>>>>> easy ?
> > > > > >>>>>>>>>>>> Most of the points are kind of the actual questions
> that
> > > you
> > > > > >>>>> want
> > > > > >>>>>> to
> > > > > >>>>>>>>>>>> address before hand anyway isn't ?
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Going code first
> > > > > >>>>>>>>>>>>> should be in general discouraged, but when it does
> > > happen,
> > > > > >> it
> > > > > >>>>>> seems
> > > > > >>>>>>>>>>> like
> > > > > >>>>>>>>>>>>> opening DIP/an issue/starting a mailing list thread
> or
> > > > > >>>>> whatever we
> > > > > >>>>>>> go
> > > > > >>>>>>>>>>>> with
> > > > > >>>>>>>>>>>>> to have a more high level design discussion alongside
> > the
> > > > > >>>>>> reference
> > > > > >>>>>>> PR
> > > > > >>>>>>>>>>>>> could alleviate some of these complaints?
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> What are the complaints ?
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> +1 for "DIP" heh, I think making
> > > > > >>>>>>>>>>>>> them in the form of github issues is probably
> > > appropriate,
> > > > > >>>>> with a
> > > > > >>>>>>> dev
> > > > > >>>>>>>>>>>> list
> > > > > >>>>>>>>>>>>> thread to announce them perhaps?
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> I think  github issue with [Proposal] header like
> > > > > >>>>>>>>>>>> https://github.com/apache/incubator-druid/issues/4349
> > is
> > > > > >> good
> > > > > >>>>> to
> > > > > >>>>>> me,
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Thanks!
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <
> > > > > >>>>> bslim@apache.org>
> > > > > >>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Thanks everyone for interacting with this thread.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> The fact that i, Roman, Jihoon  and others in the
> past
> > > (FJ
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ
> > > > > >>>>> )
> > > > > >>>>>>>>>>>>>> raised this point indicates that PRs without a
> > proposal
> > > > > >> are
> > > > > >>>>>> indeed
> > > > > >>>>>>> an
> > > > > >>>>>>>>>>>>> issue
> > > > > >>>>>>>>>>>>>> and we need to solve it.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Something Similar to KIP maybe called DIPs is fine
> > with
> > > > > >> me.
> > > > > >>>>>>>>>>>>>> What i strive to see is the following:
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> [Step 1] formalize what is the kind of work that
> > needs a
> > > > > >>>>> formal
> > > > > >>>>>>>>>>>>> Proposal, I
> > > > > >>>>>>>>>>>>>> think Roman and Jihoon has already covered that
> pretty
> > > > > >> well.
> > > > > >>>>> am
> > > > > >>>>>> +1
> > > > > >>>>>>> on
> > > > > >>>>>>>>>>>>> that.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> I am strongly in favor of the separation of Proposal
> > > > > >> Review
> > > > > >>>>> and
> > > > > >>>>>>>>>>> (later)
> > > > > >>>>>>>>>>>>>> Code review PRs. My  main reasons:
> > > > > >>>>>>>>>>>>>> Most importantly code reviewing will introduce lot
> of
> > > > > >> noise
> > > > > >>>>> and
> > > > > >>>>>>> will
> > > > > >>>>>>>>>>>>>> ultimately make  the GitHub page unreadable.
> > > > > >>>>>>>>>>>>>> Avoid overlapping of work.
> > > > > >>>>>>>>>>>>>> Once code is written hard to think abstract.
> > > > > >>>>>>>>>>>>>> Separate page for Design review later can always be
> > used
> > > > > >> it
> > > > > >>>>> as a
> > > > > >>>>>>>>>>> Design
> > > > > >>>>>>>>>>>>>> document that is readable and code free-ish.
> > > > > >>>>>>>>>>>>>> As i said the goal of this first round is to see if
> > the
> > > > > >>>>> community
> > > > > >>>>>>>>>>> agree
> > > > > >>>>>>>>>>>>>> about such change, then make the process of design
> > more
> > > > > >>>>> inclusive
> > > > > >>>>>>>>>>> thus
> > > > > >>>>>>>>>>>>>> other contributors can submit a counter proposals.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> [Step 2] IF everybody agree about that point Step 2
> is
> > > to
> > > > > >>>>> define
> > > > > >>>>>>>>>>> which
> > > > > >>>>>>>>>>>>>> medium is used to Publish a primitive form of a CODE
> > > FREE
> > > > > >>>>>> Abstract
> > > > > >>>>>>>>>>>>> Proposal
> > > > > >>>>>>>>>>>>>> containing at least the following bullet points.
> > > > > >>>>>>>>>>>>>> - The problem description and motivation
> > > > > >>>>>>>>>>>>>> - Overview of the proposed change
> > > > > >>>>>>>>>>>>>> - Operational impact (compatibility/ plans to
> > upgrades)
> > > > > >>> public
> > > > > >>>>>> API
> > > > > >>>>>>>>>>>>> changes,
> > > > > >>>>>>>>>>>>>> configuration changes, algorithm, and so on
> > > > > >>>>>>>>>>>>>> - Expected benefits and drawbacks
> > > > > >>>>>>>>>>>>>> - Rationale and alternatives
> > > > > >>>>>>>>>>>>>> - Estimate Time to Deliver if possible.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> The way i think this can be is a Github issue where
> > > member
> > > > > >>> of
> > > > > >>>>> the
> > > > > >>>>>>>>>>>>> community
> > > > > >>>>>>>>>>>>>> will interact via comments and the author will be
> > > updating
> > > > > >>> the
> > > > > >>>>>>>>>>>>> description
> > > > > >>>>>>>>>>>>>> in the light of comments provided by the community.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> During and near the end of the design discussions
> the
> > > > > >>> author/s
> > > > > >>>>>> can
> > > > > >>>>>>>>>>>> start
> > > > > >>>>>>>>>>>>>> writing POCs to help guide the review process this
> > > > > >> naturally
> > > > > >>>>> will
> > > > > >>>>>>> be
> > > > > >>>>>>>>>>> a
> > > > > >>>>>>>>>>>>> Pull
> > > > > >>>>>>>>>>>>>> request with actual code.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> *Now the most important thing is that we need to
> agree
> > > > > >> that
> > > > > >>>>> any
> > > > > >>>>>>> work
> > > > > >>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>> does not align with this formal process will be
> > ignored
> > > > > >> and
> > > > > >>>>> the
> > > > > >>>>>>>>>>> author
> > > > > >>>>>>>>>>>>> will
> > > > > >>>>>>>>>>>>>> be asked to start with a DIP*
> > > > > >>>>>>>>>>>>>> *That is what i meant with  “If it didn’t happen on
> > the
> > > > > >>>>> mailing
> > > > > >>>>>>> list,
> > > > > >>>>>>>>>>>> it
> > > > > >>>>>>>>>>>>>> didn’t happen.”*
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Thanks and happy coding!
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <
> > > > > >>> gian@apache.org>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> One of the advantages I see with a more formal
> > process
> > > is
> > > > > >>>>> (like
> > > > > >>>>>>>>>>> Kafka
> > > > > >>>>>>>>>>>>>> KIPs)
> > > > > >>>>>>>>>>>>>>> is that it levels the playing field a bit and sets
> > some
> > > > > >>>>> ground
> > > > > >>>>>>>>>>> rules
> > > > > >>>>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>>> working together. In a way it can help encourage
> > > > > >>>>> contributions
> > > > > >>>>>> by
> > > > > >>>>>>>>>>>>> making
> > > > > >>>>>>>>>>>>>> it
> > > > > >>>>>>>>>>>>>>> clear what is expected of potential contributors.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> We have a design review process today that is not
> as
> > > > > >> formal
> > > > > >>>>> as
> > > > > >>>>>>>>>>> KIPs,
> > > > > >>>>>>>>>>>>> but
> > > > > >>>>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>>>> somewhat heavier than the one you describe. Maybe
> we
> > > > > >> could
> > > > > >>>>> tweak
> > > > > >>>>>>>>>>> our
> > > > > >>>>>>>>>>>>>>> current one by starting to do design reviews
> > separately
> > > > > >>> from
> > > > > >>>>>> PRs.
> > > > > >>>>>>>>>>>> i.e.,
> > > > > >>>>>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>>> anything that meets our 'design review' criteria,
> do
> > > that
> > > > > >>> on
> > > > > >>>>> the
> > > > > >>>>>>>>>>> dev
> > > > > >>>>>>>>>>>>> list
> > > > > >>>>>>>>>>>>>>> or in a separate issue, and keep the PR focused on
> > > > > >>> code-level
> > > > > >>>>>>>>>>> stuff.
> > > > > >>>>>>>>>>>>> That
> > > > > >>>>>>>>>>>>>>> way we don't end up trying to do both at once. And
> it
> > > > > >> makes
> > > > > >>>>> it
> > > > > >>>>>>>>>>> easier
> > > > > >>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>> start talking about design before the code is
> ready,
> > > > > >> which
> > > > > >>>>> would
> > > > > >>>>>>> be
> > > > > >>>>>>>>>>>>>> better.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <
> > > > > >>> jhyde@apache.org
> > > > > >>>>>>
> > > > > >>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> It’s really hard to say no to a contribution when
> > > > > >> someone
> > > > > >>>>> has
> > > > > >>>>>> put
> > > > > >>>>>>>>>>>> in
> > > > > >>>>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>> significant amount of work.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> The following approach is simple and works really
> > > well:
> > > > > >>>>> Before
> > > > > >>>>>>>>>>> you
> > > > > >>>>>>>>>>>>>> start
> > > > > >>>>>>>>>>>>>>>> work, log a case, describing the problem. When you
> > > have
> > > > > >>> some
> > > > > >>>>>>>>>>> ideas
> > > > > >>>>>>>>>>>>>> about
> > > > > >>>>>>>>>>>>>>>> design, add those to the case. When you have a
> code
> > > > > >>> branch,
> > > > > >>>>> add
> > > > > >>>>>>>>>>> its
> > > > > >>>>>>>>>>>>> URL
> > > > > >>>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>> the case. And so forth. At any point in the
> > > proceedings,
> > > > > >>>>> people
> > > > > >>>>>>>>>>> can
> > > > > >>>>>>>>>>>>>> chime
> > > > > >>>>>>>>>>>>>>>> in with their opinions.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> In my opinion, a formal “design review” process is
> > not
> > > > > >>>>>> necessary.
> > > > > >>>>>>>>>>>>> Just
> > > > > >>>>>>>>>>>>>>>> build consensus iteratively, by starting the
> > > > > >> conversation
> > > > > >>>>> early
> > > > > >>>>>>>>>>> in
> > > > > >>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>> process.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Julian
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <
> > > > > >>> gian@apache.org
> > > > > >>>>>>
> > > > > >>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> In this particular case: please consider the PR
> as
> > a
> > > > > >>>>> proposal.
> > > > > >>>>>>>>>>>>> Don't
> > > > > >>>>>>>>>>>>>>> feel
> > > > > >>>>>>>>>>>>>>>>> like just because there is code there that takes
> a
> > > > > >>> certain
> > > > > >>>>>>>>>>>>> approach,
> > > > > >>>>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>> the approach is somehow sacred. I had to
> implement
> > > > > >>>>> something
> > > > > >>>>>> to
> > > > > >>>>>>>>>>>>>>>> crystallize
> > > > > >>>>>>>>>>>>>>>>> my own thinking about how the problem could be
> > > > > >>> approached.
> > > > > >>>>> I
> > > > > >>>>>>>>>>>> won't
> > > > > >>>>>>>>>>>>> be
> > > > > >>>>>>>>>>>>>>>>> disappointed if, as a community, we decide a
> > > different
> > > > > >>>>>>>>>>> direction
> > > > > >>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>>>>> better
> > > > > >>>>>>>>>>>>>>>>> and the code all gets thrown away. That's one of
> > the
> > > > > >>>>> reasons
> > > > > >>>>>>>>>>>> that I
> > > > > >>>>>>>>>>>>>>>> removed
> > > > > >>>>>>>>>>>>>>>>> the 0.14.0 milestone that was added to the patch.
> > (I
> > > > > >>> don't
> > > > > >>>>>> want
> > > > > >>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>> rush
> > > > > >>>>>>>>>>>>>>>> it,
> > > > > >>>>>>>>>>>>>>>>> nor do I think that's a good idea.)
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> In general: Sounds like we could do with some
> more
> > > > > >>>>>>>>>>> formalization
> > > > > >>>>>>>>>>>>>> around
> > > > > >>>>>>>>>>>>>>>>> what a proposal looks like, which sorts of
> changes
> > > need
> > > > > >>>>> one,
> > > > > >>>>>>>>>>> and
> > > > > >>>>>>>>>>>>> when
> > > > > >>>>>>>>>>>>>>> in
> > > > > >>>>>>>>>>>>>>>>> the dev cycle it is appropriate. FWIW I think
> > Kafka's
> > > > > >>>>> process
> > > > > >>>>>>>>>>> is
> > > > > >>>>>>>>>>>>> more
> > > > > >>>>>>>>>>>>>>> or
> > > > > >>>>>>>>>>>>>>>>> less fine, and would be okay with adopting it for
> > > Druid
> > > > > >>> if
> > > > > >>>>>>>>>>> people
> > > > > >>>>>>>>>>>>>> like
> > > > > >>>>>>>>>>>>>>>> it.
> > > > > >>>>>>>>>>>>>>>>> Right now our standards for what requires a
> "design
> > > > > >>> review"
> > > > > >>>>>> are
> > > > > >>>>>>>>>>>>> very
> > > > > >>>>>>>>>>>>>>>>> similar to the Kafka community standards for what
> > > > > >>> requires
> > > > > >>>>> a
> > > > > >>>>>>>>>>> KIP,
> > > > > >>>>>>>>>>>>> so
> > > > > >>>>>>>>>>>>>> we
> > > > > >>>>>>>>>>>>>>>>> have some familiarity with those concepts.
> However
> > we
> > > > > >>> don't
> > > > > >>>>>>>>>>>>> separate
> > > > > >>>>>>>>>>>>>> PR
> > > > > >>>>>>>>>>>>>>>>> review and proposal discussion as strictly as
> they
> > > do,
> > > > > >>>>> which
> > > > > >>>>>>>>>>>> seems
> > > > > >>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>> be
> > > > > >>>>>>>>>>>>>>>>> the foundation for the feeling of exclusion that
> is
> > > > > >> being
> > > > > >>>>> felt
> > > > > >>>>>>>>>>>>> here.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Separately: I just redid the description on
> > > > > >>>>>>>>>>>>>>>>>
> > https://github.com/apache/incubator-druid/pull/6794
> > > to
> > > > > >>> be
> > > > > >>>>>> more
> > > > > >>>>>>>>>>>>>>>> proposal-y.
> > > > > >>>>>>>>>>>>>>>>> I followed the KIP style:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > >>>>>>>>>>>>>>>> .
> > > > > >>>>>>>>>>>>>>>>> Please refresh the page and see if it looks more
> > > > > >> useful.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Gian
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <
> > > > > >>>>> jhyde@apache.org
> > > > > >>>>>>>
> > > > > >>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Slim,
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> I agree with your points that offline
> development
> > is
> > > > > >> bad
> > > > > >>>>> for
> > > > > >>>>>>>>>>>>>>> community.
> > > > > >>>>>>>>>>>>>>>>>> But I don’t think you need much mentor help. You
> > > have
> > > > > >>>>> raised
> > > > > >>>>>>>>>>>> valid
> > > > > >>>>>>>>>>>>>>>> issues
> > > > > >>>>>>>>>>>>>>>>>> and the Druid community needs to decide what its
> > > > > >>>>> development
> > > > > >>>>>>>>>>>>>> practices
> > > > > >>>>>>>>>>>>>>>>>> should be.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Julian
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
> > > > > >>>>>>>>>>> bslim@apache.org>
> > > > > >>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Hello everyone and hope you all have very good
> > > > > >>> holidays.
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> First, this email is not directed on the author
> > or
> > > > > >> the
> > > > > >>> PR
> > > > > >>>>>>>>>>>>>>>>>>>
> > > https://github.com/apache/incubator-druid/pull/6794
> > > > > >>> it
> > > > > >>>>>>>>>>> self,
> > > > > >>>>>>>>>>>>> but
> > > > > >>>>>>>>>>>>>> i
> > > > > >>>>>>>>>>>>>>>> see
> > > > > >>>>>>>>>>>>>>>>>>> this PR as a perfect example.
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> One of the foundation of Apache Way or what i
> > would
> > > > > >>>>> simply
> > > > > >>>>>>>>>>> call
> > > > > >>>>>>>>>>>>>> open
> > > > > >>>>>>>>>>>>>>>>>> source
> > > > > >>>>>>>>>>>>>>>>>>> community driven development is that "Technical
> > > > > >>> decisions
> > > > > >>>>>> are
> > > > > >>>>>>>>>>>>>>>> discussed,
> > > > > >>>>>>>>>>>>>>>>>>> decided, and archived publicly.
> > > > > >>>>>>>>>>>>>>>>>>> developpement"
> > > > > >>>>>>>>>>>>>>>>>>> Which means that big technical  changes such as
> > the
> > > > > >> one
> > > > > >>>>>>>>>>> brought
> > > > > >>>>>>>>>>>>> by
> > > > > >>>>>>>>>>>>>>>> #/6794
> > > > > >>>>>>>>>>>>>>>>>>> should have started as a proposal and round of
> > > > > >>>>> discussions
> > > > > >>>>>>>>>>>> about
> > > > > >>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>> major
> > > > > >>>>>>>>>>>>>>>>>>> changes designs not as 11K line of code.
> > > > > >>>>>>>>>>>>>>>>>>> I believe such openness will promote a lot of
> > good
> > > > > >>>>> benefits
> > > > > >>>>>>>>>>>> such
> > > > > >>>>>>>>>>>>>> as:
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> - ensures community health and growth.
> > > > > >>>>>>>>>>>>>>>>>>> - ensures everyone can participate not only the
> > > > > >> authors
> > > > > >>>>> and
> > > > > >>>>>>>>>>> his
> > > > > >>>>>>>>>>>>>>>>>> co-workers.
> > > > > >>>>>>>>>>>>>>>>>>> - ensures that the project is driven by the
> > > community
> > > > > >>> and
> > > > > >>>>>>>>>>> not a
> > > > > >>>>>>>>>>>>>> given
> > > > > >>>>>>>>>>>>>>>>>>> company or an individual.
> > > > > >>>>>>>>>>>>>>>>>>> - ensures that there is consensus (not saying
> > 100%
> > > > > >>>>>>>>>>> agreement;)
> > > > > >>>>>>>>>>>>>>> however
> > > > > >>>>>>>>>>>>>>>> it
> > > > > >>>>>>>>>>>>>>>>>>> means that all individuals will accept the
> > current
> > > > > >>>>> progress
> > > > > >>>>>>>>>>> on
> > > > > >>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>> project
> > > > > >>>>>>>>>>>>>>>>>>> until some better proposal is put forth.
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Personally such BIG offline PR makes me feel
> > > excluded
> > > > > >>> and
> > > > > >>>>>>>>>>>> doesn't
> > > > > >>>>>>>>>>>>>>> give
> > > > > >>>>>>>>>>>>>>>>>> me a
> > > > > >>>>>>>>>>>>>>>>>>> sense that i belong to  a community at all.
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> To prevent such off list development i think
> as a
> > > > > >> Druid
> > > > > >>>>>>>>>>>> Community
> > > > > >>>>>>>>>>>>>> we
> > > > > >>>>>>>>>>>>>>>> need
> > > > > >>>>>>>>>>>>>>>>>>> to stick to the apache way “If it didn’t happen
> > on
> > > > > >> the
> > > > > >>>>>>>>>>> mailing
> > > > > >>>>>>>>>>>>>> list,
> > > > > >>>>>>>>>>>>>>> it
> > > > > >>>>>>>>>>>>>>>>>>> didn’t happen.”
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> I would appreciate if some of the Apache mentor
> > > help
> > > > > >>> with
> > > > > >>>>>>>>>>> this.
> > > > > >>>>>>>>>>>>>>>>>>> Thanks
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>
> > > ---------------------------------------------------------------------
> > > > > >>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > > > > >>> dev-unsubscribe@druid.apache.org
> > > > > >>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > >>>>> dev-help@druid.apache.org
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > > > > >> dev-unsubscribe@druid.apache.org
> > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > >>> dev-help@druid.apache.org
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> --
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> B-Slim
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>
> > > > > >>>
> > > > >
> > >
> _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>
> > > > ---------------------------------------------------------------------
> > > > > >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > > >>>>>>>>> For additional commands, e-mail:
> dev-help@druid.apache.org
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>
> > > ---------------------------------------------------------------------
> > > > > >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > > >>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > > For additional commands, e-mail: dev-help@druid.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Off list major development

[Roman Leventov <le...@gmail.com>]

In such small PRs, authors likely won't be aware that they need to create a
proposal in the first place. The first reviewer just adds the "Design
Review" tag. It's also absolutely not about considering designs and gauging
the proposal, it's just verifying that a configuration / parameter / HTTP
endpoint name is reasonable and aligned with the rest of Druid. So I think
that a separate proposal issue for such PRs is unnecessary bureaucracy.

On Tue, 15 Jan 2019 at 07:45, Jihoon Son <gh...@gmail.com> wrote:

> Roman,
>
> > Jihoon in
>
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> and later Gian in this thread suggested that _every_ piece of work that
> should be labelled as "Design Review" according to the current rules should
> be accompanied by an issue. I don't agree with this, there are some PRs as
> small as a few dozens of lines of code, that add some configuration
> parameter and therefore should be labelled "Design Review". I don't thing a
> separate proposal issue is needed for them, and even for a little larger
> PRs too.
>
> What I'm concerned with is how people feel if their design is not accepted
> even though they wrote code. Of course, as Clint said, sometimes code helps
> better understanding of the proposal. But, I believe this is the case when
> the proposal is quite complicated and not easy to understand without code.
> Also the authors should be aware of that they might rewrite the entire code
> if the design should be changed.
>
> If writing code is simple, I don't see why the authors don't wait until the
> review for their proposal is finished.
>
> Jihoon
>
> On Fri, Jan 11, 2019 at 9:51 AM Fangjin Yang <fa...@imply.io> wrote:
>
> > I agree with Gian, as an Apache committer, your responsibility is for the
> > betterment of the project. I agree it is in the best interest of the
> > project to stop thinking about what orgs people belong to. We are all a
> > part of the Apache software foundation, regardless of what our roles and
> > titles are outside of it.
> >
> > On Fri, Jan 11, 2019 at 2:22 AM Roman Leventov <le...@gmail.com>
> > wrote:
> >
> > > It's not that people from one org could abuse the project and push some
> > > change, but that they have similar perspective (bubble effect) and some
> > > important aspects of a large feature could escape their attention.
> > >
> > > I suggest it to be not a rigid rule, but a recommendation for authors
> of
> > > large proposals to try to attract reviewers from other orgs.
> > >
> > > On Fri, 11 Jan 2019 at 02:51, Julian Hyde <jh...@apache.org> wrote:
> > >
> > > > I agree with Gian.
> > > >
> > > > As an Apache committer, you only have one affiliation: you are
> working
> > in
> > > > the best interests of the project.
> > > >
> > > > Obviously, in the real world there are other pressures. But we do our
> > > best
> > > > to compensate for them.
> > > >
> > > > Also, as a a community we try to design our process so as to avoid
> > undue
> > > > influences. For instance, when I advocate for logging cases early, I
> am
> > > > trying to mitigate the effect of product managers and VPs of
> > engineering,
> > > > who like to have their say in meeting rooms rather than on public
> > mailing
> > > > lists. That’s just one example; if we see other influences at play,
> > let’s
> > > > evolve our process to try to level the playing field.
> > > >
> > > > Julian
> > > >
> > > >
> > > > > On Jan 10, 2019, at 10:40 AM, Gian Merlino <gi...@apache.org>
> wrote:
> > > > >
> > > > >>> What do community members think about also making a requirement
> > that
> > > > >>> "Design Review" PRs and proposals are reviewed by at least two
> > people
> > > > > with
> > > > >>> different affiliation?
> > > > >> This seems pretty reasonable to me. I haven't found anything in
> > Apache
> > > > >> voting procedure docs (
> > https://www.apache.org/foundation/voting.html)
> > > > that
> > > > >> seems to explicitly forbid something like this yet at least.
> > > > >
> > > > > On the other hand, my understanding of the Apache Way would mean
> that
> > > > this
> > > > > kind of rule doesn't make sense. In particular from
> > > > > https://www.apache.org/foundation/how-it-works.html: "We firmly
> > > believe
> > > > in
> > > > > hats. Your role at the ASF is one assigned to you personally, and
> is
> > > > > bestowed on you by your peers. It is not tied to your job or
> current
> > > > > employer or company." That sentiment seems incompatible with making
> > > > > explicit rules about organizational diversity in voting. I have
> also
> > > > heard
> > > > > a few people say things like: people are supposed to represent
> > > > themselves,
> > > > > not their employers.
> > > > >
> > > > > Obviously, though, people's actions and opinions are influenced by
> > > their
> > > > > employer. IMO, a better way to approach a potential problem there
> is
> > > that
> > > > > if people from a particular organization end up behaving abusively,
> > > then
> > > > > PMC members from other organizations (or in extreme situations the
> > > Apache
> > > > > board itself) should tell them to knock it off. And then do more
> > > serious
> > > > > things if the inappropriate behavior continues. This kind
> > > > >
> > > > > The other thing you brought up, promoting more shared ownership and
> > > > > awareness, I am hopeful that separating proposals from PRs will
> help
> > > with
> > > > > that. One reason is that it takes much less time to understand a
> well
> > > > > written proposal than it takes to understand a PR. Code is very
> clear
> > > but
> > > > > it is also very verbose and takes a while to read and understand.
> So
> > in
> > > > > theory changing how we operate, in this way, should promote more
> > > > > understanding of more people across the code base.
> > > > >
> > > > > On Thu, Jan 10, 2019 at 1:34 AM Clint Wylie <cl...@imply.io>
> > > > wrote:
> > > > >
> > > > >>>
> > > > >>> What do community members think about also making a requirement
> > that
> > > > >>> "Design Review" PRs and proposals are reviewed by at least two
> > people
> > > > >> with
> > > > >>> different affiliation?
> > > > >>
> > > > >>
> > > > >> This seems pretty reasonable to me. I haven't found anything in
> > Apache
> > > > >> voting procedure docs (
> > https://www.apache.org/foundation/voting.html)
> > > > that
> > > > >> seems to explicitly forbid something like this yet at least.
> > > > >>
> > > > >> On Wed, Jan 9, 2019 at 8:52 PM Roman Leventov <
> > leventov.ru@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> What do community members think about also making a requirement
> > that
> > > > >>> "Design Review" PRs and proposals are reviewed by at least two
> > people
> > > > >> with
> > > > >>> different affiliation? IMO it's a good idea, because it ensures
> > that
> > > > >>> different interests are taken into account. Also it pushes people
> > to
> > > > >> engage
> > > > >>> with work done in other parts of Druid, improving shared code
> > owning
> > > > and
> > > > >>> awareness.
> > > > >>>
> > > > >>> Or it's against the Apache Way because it's assumed that there
> are
> > no
> > > > >>> company boundaries within the community?
> > > > >>>
> > > > >>> On Thu, 10 Jan 2019 at 11:45, Roman Leventov <
> > leventov.ru@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> I see two important reasons why it makes sense to file an issue
> > and
> > > > >>>> probably announce it in the mailing list, before writing a lot
> of
> > > > code,
> > > > >>>> despite not having a clear picture of what it will be and any
> > > > >> performance
> > > > >>>> data:
> > > > >>>> 1) somebody could already work on this problem privately in
> > > parallel,
> > > > >> it
> > > > >>>> allows to avoid clash in people's works
> > > > >>>> 2) some people could quickly think about the problem field and
> > share
> > > > >>>> high-level ideas that could wildly change the direction in which
> > the
> > > > >>> author
> > > > >>>> (the person who is going to write code) will move in his work
> from
> > > > >> early
> > > > >>> on.
> > > > >>>>
> > > > >>>> Jihoon in
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> > > > >>> and
> > > > >>>> later Gian in this thread suggested that _every_ piece of work
> > that
> > > > >>> should
> > > > >>>> be labelled as "Design Review" according to the current rules
> > should
> > > > be
> > > > >>>> accompanied by an issue. I don't agree with this, there are some
> > PRs
> > > > as
> > > > >>>> small as a few dozens of lines of code, that add some
> > configuration
> > > > >>>> parameter and therefore should be labelled "Design Review". I
> > don't
> > > > >>> thing a
> > > > >>>> separate proposal issue is needed for them, and even for a
> little
> > > > >> larger
> > > > >>>> PRs too.
> > > > >>>>
> > > > >>>> For the same reason, I also don't see the point of renaming
> > "Design
> > > > >>>> Review" into "Proposal", as well as separating "Design Review"
> > into
> > > > >>>> "Proposal" and something like "API Review". I think a single
> > "Design
> > > > >>>> Review" tag handles it well.
> > > > >>>>
> > > > >>>> Gian mentioned an idea that PRs that follow a "Design Review"
> > > proposal
> > > > >>>> issue shouldn't be "Design Review" themselves. I don't agree
> with
> > > > >> this, I
> > > > >>>> think that actual code and performance data are important inputs
> > > that
> > > > >>>> should be re-evaluated at least by two people. I even think that
> > > it's
> > > > >>> very
> > > > >>>> desirable that at least two people read _each line of production
> > > code_
> > > > >> in
> > > > >>>> large PRs, although it's not what was done historically in
> Druid,
> > > > >> because
> > > > >>>> large bodies of newly added code, with whole new classes and
> > > > subsystems
> > > > >>>> added, are also coincidentally tested worse than already
> existing
> > > > >> classes
> > > > >>>> and subsystems, including in production. It seems to me that
> those
> > > > huge
> > > > >>>> code influxes is a major source of bugs, that could later take
> > years
> > > > to
> > > > >>>> squeeze out from the codebase.
> > > > >>>>
> > > > >>>> On Wed, 9 Jan 2019 at 08:24, Clint Wylie <cl...@imply.io>
> > > > wrote:
> > > > >>>>
> > > > >>>>> Apologies for the delayed response.
> > > > >>>>>
> > > > >>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > > > >>> slim.bouguerra@gmail.com>
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> I am wondering here what is the case where code first is
> better?
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>> I don't think it's wrong to share ideas as early as possible,
> and
> > > > >> after
> > > > >>>>> this discussion I think I am in favor of it too. I just meant
> > that
> > > I
> > > > >>> don't
> > > > >>>>> think it's always necessarily the most productive discussion
> > until
> > > > >> code
> > > > >>>>> exists sometimes, with the types of thing I am thinking of are
> > > almost
> > > > >>>>> entirely limited to cases where things might sound good to
> anyone
> > > on
> > > > >>> paper
> > > > >>>>> but in reality need a large amount of experiments conducted and
> > > > >>>>> observations collected to determine that something is actually
> > > worth
> > > > >>>>> doing,
> > > > >>>>> which I imagine is mainly things like reworking internals for
> > > > >>> performance
> > > > >>>>> improvements.
> > > > >>>>>
> > > > >>>>> In the case of my combined proposal PR, I needed to prove that
> > the
> > > > >>> thing I
> > > > >>>>> was working on was a good idea... and it wasn't directly. But I
> > > came
> > > > >> up
> > > > >>>>> with another idea during the course of experiment turned into
> > > > >> something
> > > > >>>>> compelling, so an initial proposal would have looked quite a
> lot
> > > > >>> different
> > > > >>>>> than what I ended up with. Once I had proven to myself that it
> > was
> > > a
> > > > >>> good
> > > > >>>>> idea, then I was comfortable sharing with the wider community.
> > I'm
> > > > not
> > > > >>>>> certain how this would play out in an always proposal first
> > model,
> > > > >> maybe
> > > > >>>>> the first proposal exists, I personally reject it after
> updating
> > > with
> > > > >>>>> experiment results show it's a bad idea, continue experimenting
> > and
> > > > >>> raise
> > > > >>>>> a
> > > > >>>>> new one after the experiments start looking promising?
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>> Let's not be naive this is very rare that a contributor will
> > > accept
> > > > >>> that
> > > > >>>>>> his work is to be thrown, usually devs takes coding as
> personal
> > > > >>> creation
> > > > >>>>>> and they get attached to it.
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>> I agree, just because a handful of the committers have this
> > > attitude,
> > > > >> it
> > > > >>>>> isn't fair to expect the wider community to also, that's why I
> am
> > > in
> > > > >>> favor
> > > > >>>>> of formalizing the process.
> > > > >>>>>
> > > > >>>>> Can you please explain what is overbearing ? what can be
> changed
> > to
> > > > >> make
> > > > >>>>> it
> > > > >>>>>> easy ?
> > > > >>>>>> Most of the points are kind of the actual questions that you
> > want
> > > to
> > > > >>>>>> address before hand anyway isn't ?
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>> Sorry for the confusion, I said it's "not overbearing", I think
> > > it's
> > > > >>> fine.
> > > > >>>>>
> > > > >>>>> What are the complaints ?
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> Is this and other previous threads not a complaint about
> opening
> > a
> > > > >> large
> > > > >>>>> PR
> > > > >>>>> without a proposal? :) I just mean that formalizing the
> process,
> > > even
> > > > >>> if a
> > > > >>>>> proposal has a reference PR opened with it near concurrently,
> > could
> > > > >>>>> prevent
> > > > >>>>> these discussions from happening in the future because ground
> > rules
> > > > >> have
> > > > >>>>> been set and we are all on the same page I guess.
> > > > >>>>>
> > > > >>>>> I think we should also probably consider renaming the "design
> > > review"
> > > > >>>>> label
> > > > >>>>> to "proposal" or something to make it more clear that a PR is
> > > > >> associated
> > > > >>>>> with a proposal. It might also be worth considering using
> github
> > > > >>> projects
> > > > >>>>> for particularly large things that involve multiple follow up
> > PRs,
> > > > >> but I
> > > > >>>>> haven't used them in much depth to know if they add anything.
> > > > >>>>>
> > > > >>>>> It seems like we are all converging on agreement to do a github
> > > issue
> > > > >>>>> proposal for larger changes (I would vote for announcing them
> on
> > > the
> > > > >> dev
> > > > >>>>> list too for more visibility), so that design review is
> separated
> > > > from
> > > > >>>>> code
> > > > >>>>> review. I guess my main concern was not wanting to discourage
> > > > >>>>> experimentation by walling it off behind mandatory discussions,
> > but
> > > > >> the
> > > > >>>>> more I think about it,  it doesn't technically change much
> about
> > > this
> > > > >>>>> process, it just requires a more formal proposal to accompany
> any
> > > > >>>>> experiments that are shared as a PR.
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Tue, Jan 8, 2019 at 12:28 PM Gian Merlino <gi...@apache.org>
> > > > wrote:
> > > > >>>>>
> > > > >>>>>> I think for us, choosing to use GitHub issues as discussion
> > > threads
> > > > >>> for
> > > > >>>>>> potential 'major' contributions would be a good idea,
> especially
> > > if
> > > > >> we
> > > > >>>>>> encourage people to start them before PRs show up. Definitely
> > > agree
> > > > >>> that
> > > > >>>>>> all contributors should go through the same process -- I
> > couldn't
> > > > >> see
> > > > >>> it
> > > > >>>>>> working well any other way.
> > > > >>>>>>
> > > > >>>>>> On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <jhyde@apache.org
> >
> > > > >> wrote:
> > > > >>>>>>
> > > > >>>>>>> Statically, yes, GitHub PRs are the same as GitHub cases. But
> > > > >>>>>> dynamically,
> > > > >>>>>>> they are different, because you can only log a PR when you
> have
> > > > >>>>> finished
> > > > >>>>>>> work.
> > > > >>>>>>>
> > > > >>>>>>> A lot of other Apache projects use JIRA, so there is a clear
> > > > >>>>> distinction
> > > > >>>>>>> between cases and contributions. JIRA cases, especially when
> > > > >> logged
> > > > >>>>> early
> > > > >>>>>>> in the lifecycle of a contribution, become long-running
> > > > >> conversation
> > > > >>>>>>> threads with a lot of community participation. If the Druid
> > chose
> > > > >> to
> > > > >>>>> do
> > > > >>>>>> so,
> > > > >>>>>>> GitHub cases could be the same.
> > > > >>>>>>>
> > > > >>>>>>> Be careful that you do not treat “potential contributors” (by
> > > > >> which
> > > > >>> I
> > > > >>>>>>> presume you mean non-committers) differently from committers
> > and
> > > > >> PMC
> > > > >>>>>>> members. Anyone starting a major piece of work should follow
> > the
> > > > >>> same
> > > > >>>>>>> process. (Experienced committers probably have a somewhat
> > better
> > > > >>> idea
> > > > >>>>>> what
> > > > >>>>>>> work will turn out to be “major”, so they get a little more
> > > > >> leeway.)
> > > > >>>>>>>
> > > > >>>>>>> Julian
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>> On Jan 7, 2019, at 12:10 PM, Gian Merlino <gi...@apache.org>
> > > > >>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> I don't think there's a need to raise issues for every
> > change: a
> > > > >>>>> small
> > > > >>>>>>> bug
> > > > >>>>>>>> fix or doc fix should just go straight to PR. (GitHub PRs
> show
> > > > >> up
> > > > >>> as
> > > > >>>>>>> issues
> > > > >>>>>>>> in the issue-search UI/API, so it's not like this means the
> > > > >> patch
> > > > >>>>> has
> > > > >>>>>> no
> > > > >>>>>>>> corresponding issue -- in a sense the PR _is_ the issue.)
> > > > >>>>>>>>
> > > > >>>>>>>> I do think it makes sense to encourage potential
> contributors
> > to
> > > > >>>>> write
> > > > >>>>>> to
> > > > >>>>>>>> the dev list or raise an issue if they aren't sure if
> > something
> > > > >>>>> would
> > > > >>>>>>> need
> > > > >>>>>>>> to go through a more heavy weight process.
> > > > >>>>>>>>
> > > > >>>>>>>> Fwiw we do have a set of 'design review' criteria already
> (we
> > > > >> had
> > > > >>> a
> > > > >>>>>>>> discussion about this a couple years ago) at:
> > > > >>>>>>>> http://druid.io/community/#getting-your-changes-accepted.
> So
> > we
> > > > >>>>>>> wouldn't be
> > > > >>>>>>>> starting from zero on defining that. We set it up back when
> we
> > > > >>> were
> > > > >>>>>>> trying
> > > > >>>>>>>> to _streamline_ our process -- we used to require two
> > non-author
> > > > >>> +1s
> > > > >>>>>> for
> > > > >>>>>>>> _every_ change, even minor ones. The introduction of design
> > > > >> review
> > > > >>>>>>> criteria
> > > > >>>>>>>> was meant to classify which PRs need that level of review
> and
> > > > >>> which
> > > > >>>>>> ones
> > > > >>>>>>>> are minor and can be merged with less review. I do think it
> > > > >> helped
> > > > >>>>> with
> > > > >>>>>>>> getting minor PRs merged more quickly. The list of criteria
> > is,
> > > > >>>>>>>>
> > > > >>>>>>>> - Major architectural changes or API changes
> > > > >>>>>>>> - HTTP requests and responses (e. g. a new HTTP endpoint)
> > > > >>>>>>>> - Interfaces for extensions
> > > > >>>>>>>> - Server configuration (e. g. altering the behavior of a
> > config
> > > > >>>>>> property)
> > > > >>>>>>>> - Emitted metrics
> > > > >>>>>>>> - Other major changes, judged by the discretion of Druid
> > > > >>> committers
> > > > >>>>>>>>
> > > > >>>>>>>> Some of it is subjective, but it has been in place for a
> > while,
> > > > >> so
> > > > >>>>> it's
> > > > >>>>>>> at
> > > > >>>>>>>> least something we are relatively familiar with.
> > > > >>>>>>>>
> > > > >>>>>>>> On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <
> jhyde@apache.org
> > >
> > > > >>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Small contributions don’t need any design review, whereas
> > large
> > > > >>>>>>>>> contributions need significant review. I don’t think we
> > should
> > > > >>>>> require
> > > > >>>>>>> an
> > > > >>>>>>>>> additional step for those (many) small contributions. But
> who
> > > > >>>>> decides
> > > > >>>>>>>>> whether a contribution fits into the small or large
> category?
> > > > >>>>>>>>>
> > > > >>>>>>>>> I think the solution is for authors to log a case (or send
> an
> > > > >>>>> email to
> > > > >>>>>>>>> dev) before they start work on any contribution. Then
> > > > >> committers
> > > > >>>>> can
> > > > >>>>>>>>> request a more heavy-weight process if they think it is
> > needed.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Julian
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <
> gian@apache.org>
> > > > >>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> It sounds like splitting design from code review is a
> common
> > > > >>> theme
> > > > >>>>>> in a
> > > > >>>>>>>>> few
> > > > >>>>>>>>>> of the posts here. How does everyone feel about making a
> > point
> > > > >>> of
> > > > >>>>>>>>>> encouraging design reviews to be done as issues, separate
> > from
> > > > >>> the
> > > > >>>>>> pull
> > > > >>>>>>>>>> request, with the expectations that (1) the design review
> > > > >> issue
> > > > >>>>>>>>>> ("proposal") should generally appear somewhat _before_ the
> > > > >> pull
> > > > >>>>>>> request;
> > > > >>>>>>>>>> (2) pull requests should _not_ have design review happen
> on
> > > > >>> them,
> > > > >>>>>>> meaning
> > > > >>>>>>>>>> there should no longer be PRs with design review tags, and
> > we
> > > > >>>>> should
> > > > >>>>>>> move
> > > > >>>>>>>>>> the design review approval process to the issue rather
> than
> > > > >> the
> > > > >>>>> PR.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> For (1), even if we encourage design review discussions to
> > > > >> start
> > > > >>>>>>> before a
> > > > >>>>>>>>>> pull request appears, I don't see an issue with them
> running
> > > > >>>>>>> concurrently
> > > > >>>>>>>>>> for a while at some point.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <
> > > > >> jonwei@apache.org>
> > > > >>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Thanks for raising these concerns!
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> My initial thoughts:
> > > > >>>>>>>>>>> - I agree that separation of design review and code-level
> > > > >>> review
> > > > >>>>> for
> > > > >>>>>>>>> major
> > > > >>>>>>>>>>> changes would be more efficient
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> - I agree that a clear, more formalized process for
> > handling
> > > > >>>>> major
> > > > >>>>>>>>> changes
> > > > >>>>>>>>>>> would be helpful for contributors:
> > > > >>>>>>>>>>> - Define what is considered a major change
> > > > >>>>>>>>>>> - Define a standard proposal structure, KIP-style
> proposal
> > > > >>> format
> > > > >>>>>>>>> sounds
> > > > >>>>>>>>>>> good to me
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> - I think it's too rigid to have a policy of "no code at
> > all
> > > > >>> with
> > > > >>>>>> the
> > > > >>>>>>>>>>> initial proposal"
> > > > >>>>>>>>>>> - Code samples can be useful references for understanding
> > > > >>> aspects
> > > > >>>>>> of a
> > > > >>>>>>>>>>> design
> > > > >>>>>>>>>>> - In some cases it's necessary to run experiments to
> fully
> > > > >>>>>> understand
> > > > >>>>>>> a
> > > > >>>>>>>>>>> problem and determine an appropriate design, or to
> > determine
> > > > >>>>> whether
> > > > >>>>>>>>>>> something is even worth doing before committing to the
> work
> > > > >> of
> > > > >>>>>>> fleshing
> > > > >>>>>>>>> out
> > > > >>>>>>>>>>> a proposal, prototype code is a natural outcome of that
> and
> > > > >> I'm
> > > > >>>>> not
> > > > >>>>>>>>> against
> > > > >>>>>>>>>>> someone providing such code for reference
> > > > >>>>>>>>>>> - I tend to view design/code as things that are often
> > > > >> developed
> > > > >>>>>>>>>>> simultaneously in an intertwined way
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Let's not be naive this is very rare that a contributor
> > will
> > > > >>>>> accept
> > > > >>>>>>>>> that
> > > > >>>>>>>>>>> his work is to be thrown, usually devs takes coding as
> > > > >> personal
> > > > >>>>>>> creation
> > > > >>>>>>>>>>> and they get attached to it.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> If we have a clear review process that emphasizes the
> need
> > > > >> for
> > > > >>>>> early
> > > > >>>>>>>>>>> consensus building, with separate design and code review,
> > > > >> then
> > > > >>> I
> > > > >>>>>> feel
> > > > >>>>>>>>> we've
> > > > >>>>>>>>>>> done enough and don't need a hard rule against having
> some
> > > > >> code
> > > > >>>>>> linked
> > > > >>>>>>>>> with
> > > > >>>>>>>>>>> the initial proposal. If a potential contributor then
> still
> > > > >>>>> wants to
> > > > >>>>>>> go
> > > > >>>>>>>>>>> ahead and write a lot of code that may be rejected or
> > change
> > > > >>>>>>>>> significantly,
> > > > >>>>>>>>>>> the risks were made clear.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Once code is written hard to think abstract.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I can see the validity of the concern, but I personally
> > don't
> > > > >>>>> see it
> > > > >>>>>>> as
> > > > >>>>>>>>> a
> > > > >>>>>>>>>>> huge risk. My impression from the Druid PR reviews I've
> > seen
> > > > >> is
> > > > >>>>> that
> > > > >>>>>>> our
> > > > >>>>>>>>>>> reviewers are able to keep abstract design vs.
> > implementation
> > > > >>>>>> details
> > > > >>>>>>>>>>> separate and consider alternate designs when reviewing.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> To summarize I think it's probably enough to have a
> policy
> > > > >>> along
> > > > >>>>> the
> > > > >>>>>>>>> lines
> > > > >>>>>>>>>>> of:
> > > > >>>>>>>>>>> - Create more formalized guidelines for proposals and
> what
> > > > >>>>> changes
> > > > >>>>>>>>> require
> > > > >>>>>>>>>>> proposals
> > > > >>>>>>>>>>> - Separate design and code review for major changes, with
> > > > >>> design
> > > > >>>>>>> review
> > > > >>>>>>>>>>> first, code-level review after reaching consensus on the
> > > > >>> design.
> > > > >>>>>>>>>>> - Code before the design review is completed is just for
> > > > >>>>> reference,
> > > > >>>>>>> not
> > > > >>>>>>>>>>> regarded as a candidate for review/merging.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> - Jon
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > > > >>>>>>>>> slim.bouguerra@gmail.com>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <
> > > > >>>>> clint.wylie@imply.io>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> I am definitely biased in this matter as an owner of
> > > > >> another
> > > > >>>>> large
> > > > >>>>>>> PR
> > > > >>>>>>>>>>>> that
> > > > >>>>>>>>>>>>> wasn't preceded by a direct proposal or dev list
> > > > >> discussion,
> > > > >>>>> and
> > > > >>>>>> in
> > > > >>>>>>>>>>>> general
> > > > >>>>>>>>>>>>> I agree that proposal first is usually better, but I
> > think
> > > > >> in
> > > > >>>>> some
> > > > >>>>>>>>>>> rarer
> > > > >>>>>>>>>>>>> cases approaching a problem code first *is* the most
> > > > >>>>> appropriate
> > > > >>>>>> way
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>>> have a discussion.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I am wondering here what is the case where code first is
> > > > >>> better?
> > > > >>>>>>>>>>>> In general when you are writing code you have an idea
> > about
> > > > >>> what
> > > > >>>>>> you
> > > > >>>>>>>>> want
> > > > >>>>>>>>>>>> to change, why you want to change and why you want to
> > change
> > > > >>> it.
> > > > >>>>>>>>>>>> I do not see what is wrong with sharing this primitive
> > ideas
> > > > >>> and
> > > > >>>>>>>>> thoughts
> > > > >>>>>>>>>>>> as an abstract proposal (at least to avoid overlapping)
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I see nothing wrong with it so long as the author
> > > > >>>>>>>>>>>>> accepts that the PR is treated as a combined proposal
> and
> > > > >>>>> proof of
> > > > >>>>>>>>>>>> concept,
> > > > >>>>>>>>>>>>> and fair game to be radically changed via discussion or
> > > > >> even
> > > > >>>>>>> rejected,
> > > > >>>>>>>>>>>>> which sounds like Gian's attitude on the matter and is
> > mine
> > > > >>> as
> > > > >>>>>> well
> > > > >>>>>>>>>>> with
> > > > >>>>>>>>>>>> my
> > > > >>>>>>>>>>>>> compression stuff.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Let's not be naive this is very rare that a contributor
> > will
> > > > >>>>> accept
> > > > >>>>>>>>> that
> > > > >>>>>>>>>>>> his work is to be thrown, usually devs takes coding as
> > > > >>> personal
> > > > >>>>>>>>> creation
> > > > >>>>>>>>>>>> and they get attached to it.
> > > > >>>>>>>>>>>> To my point you can take a look on some old issue in the
> > > > >> Druid
> > > > >>>>>> forum
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> > > > >>>>>>>>>>>> and am sure other communities have similar problems.
> > > > >>>>>>>>>>>> So leaving the door open to some side cases is not a
> good
> > > > >> idea
> > > > >>>>> in
> > > > >>>>>> my
> > > > >>>>>>>>>>>> opinion and will lead to similar issue in the future.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> This seems to me especially likely to happen in cases
> > > > >>>>>>>>>>>>> where an approach still needs proven to be a viable
> idea
> > > > >> *to
> > > > >>>>> the
> > > > >>>>>>>>>>> author*,
> > > > >>>>>>>>>>>>> so that a much more productive discussion can be had in
> > the
> > > > >>>>> first
> > > > >>>>>>>>>>> place.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> I think there is a trade off, I don't think we want to
> > > > >>>>> discourage
> > > > >>>>>>>>>>>>> experimentation by walling it off behind mandatory
> > > > >>> discussions
> > > > >>>>>>> before
> > > > >>>>>>>>>>> it
> > > > >>>>>>>>>>>>> can even start, but I do think formalizing the process
> > for
> > > > >>>>> large
> > > > >>>>>>>>>>> changes
> > > > >>>>>>>>>>>> is
> > > > >>>>>>>>>>>>> a good thing, especially since we probably can't expect
> > the
> > > > >>>>> wider
> > > > >>>>>>>>>>>> community
> > > > >>>>>>>>>>>>> to have the same attitude towards a large PR getting
> > > > >>> discarded
> > > > >>>>> as
> > > > >>>>>> a
> > > > >>>>>>>>>>>>> committer might. I think the Kafka approach is
> > reasonable,
> > > > >> a
> > > > >>>>> bit
> > > > >>>>>>> more
> > > > >>>>>>>>>>>>> formal than our design review process but not
> > overbearing.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Can you please explain what is overbearing ? what can be
> > > > >>>>> changed to
> > > > >>>>>>>>> make
> > > > >>>>>>>>>>> it
> > > > >>>>>>>>>>>> easy ?
> > > > >>>>>>>>>>>> Most of the points are kind of the actual questions that
> > you
> > > > >>>>> want
> > > > >>>>>> to
> > > > >>>>>>>>>>>> address before hand anyway isn't ?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Going code first
> > > > >>>>>>>>>>>>> should be in general discouraged, but when it does
> > happen,
> > > > >> it
> > > > >>>>>> seems
> > > > >>>>>>>>>>> like
> > > > >>>>>>>>>>>>> opening DIP/an issue/starting a mailing list thread or
> > > > >>>>> whatever we
> > > > >>>>>>> go
> > > > >>>>>>>>>>>> with
> > > > >>>>>>>>>>>>> to have a more high level design discussion alongside
> the
> > > > >>>>>> reference
> > > > >>>>>>> PR
> > > > >>>>>>>>>>>>> could alleviate some of these complaints?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> What are the complaints ?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> +1 for "DIP" heh, I think making
> > > > >>>>>>>>>>>>> them in the form of github issues is probably
> > appropriate,
> > > > >>>>> with a
> > > > >>>>>>> dev
> > > > >>>>>>>>>>>> list
> > > > >>>>>>>>>>>>> thread to announce them perhaps?
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I think  github issue with [Proposal] header like
> > > > >>>>>>>>>>>> https://github.com/apache/incubator-druid/issues/4349
> is
> > > > >> good
> > > > >>>>> to
> > > > >>>>>> me,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Thanks!
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <
> > > > >>>>> bslim@apache.org>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Thanks everyone for interacting with this thread.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> The fact that i, Roman, Jihoon  and others in the past
> > (FJ
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ
> > > > >>>>> )
> > > > >>>>>>>>>>>>>> raised this point indicates that PRs without a
> proposal
> > > > >> are
> > > > >>>>>> indeed
> > > > >>>>>>> an
> > > > >>>>>>>>>>>>> issue
> > > > >>>>>>>>>>>>>> and we need to solve it.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Something Similar to KIP maybe called DIPs is fine
> with
> > > > >> me.
> > > > >>>>>>>>>>>>>> What i strive to see is the following:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> [Step 1] formalize what is the kind of work that
> needs a
> > > > >>>>> formal
> > > > >>>>>>>>>>>>> Proposal, I
> > > > >>>>>>>>>>>>>> think Roman and Jihoon has already covered that pretty
> > > > >> well.
> > > > >>>>> am
> > > > >>>>>> +1
> > > > >>>>>>> on
> > > > >>>>>>>>>>>>> that.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> I am strongly in favor of the separation of Proposal
> > > > >> Review
> > > > >>>>> and
> > > > >>>>>>>>>>> (later)
> > > > >>>>>>>>>>>>>> Code review PRs. My  main reasons:
> > > > >>>>>>>>>>>>>> Most importantly code reviewing will introduce lot of
> > > > >> noise
> > > > >>>>> and
> > > > >>>>>>> will
> > > > >>>>>>>>>>>>>> ultimately make  the GitHub page unreadable.
> > > > >>>>>>>>>>>>>> Avoid overlapping of work.
> > > > >>>>>>>>>>>>>> Once code is written hard to think abstract.
> > > > >>>>>>>>>>>>>> Separate page for Design review later can always be
> used
> > > > >> it
> > > > >>>>> as a
> > > > >>>>>>>>>>> Design
> > > > >>>>>>>>>>>>>> document that is readable and code free-ish.
> > > > >>>>>>>>>>>>>> As i said the goal of this first round is to see if
> the
> > > > >>>>> community
> > > > >>>>>>>>>>> agree
> > > > >>>>>>>>>>>>>> about such change, then make the process of design
> more
> > > > >>>>> inclusive
> > > > >>>>>>>>>>> thus
> > > > >>>>>>>>>>>>>> other contributors can submit a counter proposals.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> [Step 2] IF everybody agree about that point Step 2 is
> > to
> > > > >>>>> define
> > > > >>>>>>>>>>> which
> > > > >>>>>>>>>>>>>> medium is used to Publish a primitive form of a CODE
> > FREE
> > > > >>>>>> Abstract
> > > > >>>>>>>>>>>>> Proposal
> > > > >>>>>>>>>>>>>> containing at least the following bullet points.
> > > > >>>>>>>>>>>>>> - The problem description and motivation
> > > > >>>>>>>>>>>>>> - Overview of the proposed change
> > > > >>>>>>>>>>>>>> - Operational impact (compatibility/ plans to
> upgrades)
> > > > >>> public
> > > > >>>>>> API
> > > > >>>>>>>>>>>>> changes,
> > > > >>>>>>>>>>>>>> configuration changes, algorithm, and so on
> > > > >>>>>>>>>>>>>> - Expected benefits and drawbacks
> > > > >>>>>>>>>>>>>> - Rationale and alternatives
> > > > >>>>>>>>>>>>>> - Estimate Time to Deliver if possible.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> The way i think this can be is a Github issue where
> > member
> > > > >>> of
> > > > >>>>> the
> > > > >>>>>>>>>>>>> community
> > > > >>>>>>>>>>>>>> will interact via comments and the author will be
> > updating
> > > > >>> the
> > > > >>>>>>>>>>>>> description
> > > > >>>>>>>>>>>>>> in the light of comments provided by the community.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> During and near the end of the design discussions the
> > > > >>> author/s
> > > > >>>>>> can
> > > > >>>>>>>>>>>> start
> > > > >>>>>>>>>>>>>> writing POCs to help guide the review process this
> > > > >> naturally
> > > > >>>>> will
> > > > >>>>>>> be
> > > > >>>>>>>>>>> a
> > > > >>>>>>>>>>>>> Pull
> > > > >>>>>>>>>>>>>> request with actual code.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> *Now the most important thing is that we need to agree
> > > > >> that
> > > > >>>>> any
> > > > >>>>>>> work
> > > > >>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>> does not align with this formal process will be
> ignored
> > > > >> and
> > > > >>>>> the
> > > > >>>>>>>>>>> author
> > > > >>>>>>>>>>>>> will
> > > > >>>>>>>>>>>>>> be asked to start with a DIP*
> > > > >>>>>>>>>>>>>> *That is what i meant with  “If it didn’t happen on
> the
> > > > >>>>> mailing
> > > > >>>>>>> list,
> > > > >>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>> didn’t happen.”*
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Thanks and happy coding!
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <
> > > > >>> gian@apache.org>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> One of the advantages I see with a more formal
> process
> > is
> > > > >>>>> (like
> > > > >>>>>>>>>>> Kafka
> > > > >>>>>>>>>>>>>> KIPs)
> > > > >>>>>>>>>>>>>>> is that it levels the playing field a bit and sets
> some
> > > > >>>>> ground
> > > > >>>>>>>>>>> rules
> > > > >>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>> working together. In a way it can help encourage
> > > > >>>>> contributions
> > > > >>>>>> by
> > > > >>>>>>>>>>>>> making
> > > > >>>>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>>> clear what is expected of potential contributors.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> We have a design review process today that is not as
> > > > >> formal
> > > > >>>>> as
> > > > >>>>>>>>>>> KIPs,
> > > > >>>>>>>>>>>>> but
> > > > >>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>> somewhat heavier than the one you describe. Maybe we
> > > > >> could
> > > > >>>>> tweak
> > > > >>>>>>>>>>> our
> > > > >>>>>>>>>>>>>>> current one by starting to do design reviews
> separately
> > > > >>> from
> > > > >>>>>> PRs.
> > > > >>>>>>>>>>>> i.e.,
> > > > >>>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>> anything that meets our 'design review' criteria, do
> > that
> > > > >>> on
> > > > >>>>> the
> > > > >>>>>>>>>>> dev
> > > > >>>>>>>>>>>>> list
> > > > >>>>>>>>>>>>>>> or in a separate issue, and keep the PR focused on
> > > > >>> code-level
> > > > >>>>>>>>>>> stuff.
> > > > >>>>>>>>>>>>> That
> > > > >>>>>>>>>>>>>>> way we don't end up trying to do both at once. And it
> > > > >> makes
> > > > >>>>> it
> > > > >>>>>>>>>>> easier
> > > > >>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>> start talking about design before the code is ready,
> > > > >> which
> > > > >>>>> would
> > > > >>>>>>> be
> > > > >>>>>>>>>>>>>> better.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <
> > > > >>> jhyde@apache.org
> > > > >>>>>>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> It’s really hard to say no to a contribution when
> > > > >> someone
> > > > >>>>> has
> > > > >>>>>> put
> > > > >>>>>>>>>>>> in
> > > > >>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>> significant amount of work.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> The following approach is simple and works really
> > well:
> > > > >>>>> Before
> > > > >>>>>>>>>>> you
> > > > >>>>>>>>>>>>>> start
> > > > >>>>>>>>>>>>>>>> work, log a case, describing the problem. When you
> > have
> > > > >>> some
> > > > >>>>>>>>>>> ideas
> > > > >>>>>>>>>>>>>> about
> > > > >>>>>>>>>>>>>>>> design, add those to the case. When you have a code
> > > > >>> branch,
> > > > >>>>> add
> > > > >>>>>>>>>>> its
> > > > >>>>>>>>>>>>> URL
> > > > >>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>> the case. And so forth. At any point in the
> > proceedings,
> > > > >>>>> people
> > > > >>>>>>>>>>> can
> > > > >>>>>>>>>>>>>> chime
> > > > >>>>>>>>>>>>>>>> in with their opinions.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> In my opinion, a formal “design review” process is
> not
> > > > >>>>>> necessary.
> > > > >>>>>>>>>>>>> Just
> > > > >>>>>>>>>>>>>>>> build consensus iteratively, by starting the
> > > > >> conversation
> > > > >>>>> early
> > > > >>>>>>>>>>> in
> > > > >>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>> process.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Julian
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <
> > > > >>> gian@apache.org
> > > > >>>>>>
> > > > >>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> In this particular case: please consider the PR as
> a
> > > > >>>>> proposal.
> > > > >>>>>>>>>>>>> Don't
> > > > >>>>>>>>>>>>>>> feel
> > > > >>>>>>>>>>>>>>>>> like just because there is code there that takes a
> > > > >>> certain
> > > > >>>>>>>>>>>>> approach,
> > > > >>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>> the approach is somehow sacred. I had to implement
> > > > >>>>> something
> > > > >>>>>> to
> > > > >>>>>>>>>>>>>>>> crystallize
> > > > >>>>>>>>>>>>>>>>> my own thinking about how the problem could be
> > > > >>> approached.
> > > > >>>>> I
> > > > >>>>>>>>>>>> won't
> > > > >>>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>> disappointed if, as a community, we decide a
> > different
> > > > >>>>>>>>>>> direction
> > > > >>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>> better
> > > > >>>>>>>>>>>>>>>>> and the code all gets thrown away. That's one of
> the
> > > > >>>>> reasons
> > > > >>>>>>>>>>>> that I
> > > > >>>>>>>>>>>>>>>> removed
> > > > >>>>>>>>>>>>>>>>> the 0.14.0 milestone that was added to the patch.
> (I
> > > > >>> don't
> > > > >>>>>> want
> > > > >>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>> rush
> > > > >>>>>>>>>>>>>>>> it,
> > > > >>>>>>>>>>>>>>>>> nor do I think that's a good idea.)
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> In general: Sounds like we could do with some more
> > > > >>>>>>>>>>> formalization
> > > > >>>>>>>>>>>>>> around
> > > > >>>>>>>>>>>>>>>>> what a proposal looks like, which sorts of changes
> > need
> > > > >>>>> one,
> > > > >>>>>>>>>>> and
> > > > >>>>>>>>>>>>> when
> > > > >>>>>>>>>>>>>>> in
> > > > >>>>>>>>>>>>>>>>> the dev cycle it is appropriate. FWIW I think
> Kafka's
> > > > >>>>> process
> > > > >>>>>>>>>>> is
> > > > >>>>>>>>>>>>> more
> > > > >>>>>>>>>>>>>>> or
> > > > >>>>>>>>>>>>>>>>> less fine, and would be okay with adopting it for
> > Druid
> > > > >>> if
> > > > >>>>>>>>>>> people
> > > > >>>>>>>>>>>>>> like
> > > > >>>>>>>>>>>>>>>> it.
> > > > >>>>>>>>>>>>>>>>> Right now our standards for what requires a "design
> > > > >>> review"
> > > > >>>>>> are
> > > > >>>>>>>>>>>>> very
> > > > >>>>>>>>>>>>>>>>> similar to the Kafka community standards for what
> > > > >>> requires
> > > > >>>>> a
> > > > >>>>>>>>>>> KIP,
> > > > >>>>>>>>>>>>> so
> > > > >>>>>>>>>>>>>> we
> > > > >>>>>>>>>>>>>>>>> have some familiarity with those concepts. However
> we
> > > > >>> don't
> > > > >>>>>>>>>>>>> separate
> > > > >>>>>>>>>>>>>> PR
> > > > >>>>>>>>>>>>>>>>> review and proposal discussion as strictly as they
> > do,
> > > > >>>>> which
> > > > >>>>>>>>>>>> seems
> > > > >>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>> the foundation for the feeling of exclusion that is
> > > > >> being
> > > > >>>>> felt
> > > > >>>>>>>>>>>>> here.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Separately: I just redid the description on
> > > > >>>>>>>>>>>>>>>>>
> https://github.com/apache/incubator-druid/pull/6794
> > to
> > > > >>> be
> > > > >>>>>> more
> > > > >>>>>>>>>>>>>>>> proposal-y.
> > > > >>>>>>>>>>>>>>>>> I followed the KIP style:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > >>>>>>>>>>>>>>>> .
> > > > >>>>>>>>>>>>>>>>> Please refresh the page and see if it looks more
> > > > >> useful.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Gian
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <
> > > > >>>>> jhyde@apache.org
> > > > >>>>>>>
> > > > >>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Slim,
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> I agree with your points that offline development
> is
> > > > >> bad
> > > > >>>>> for
> > > > >>>>>>>>>>>>>>> community.
> > > > >>>>>>>>>>>>>>>>>> But I don’t think you need much mentor help. You
> > have
> > > > >>>>> raised
> > > > >>>>>>>>>>>> valid
> > > > >>>>>>>>>>>>>>>> issues
> > > > >>>>>>>>>>>>>>>>>> and the Druid community needs to decide what its
> > > > >>>>> development
> > > > >>>>>>>>>>>>>> practices
> > > > >>>>>>>>>>>>>>>>>> should be.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Julian
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
> > > > >>>>>>>>>>> bslim@apache.org>
> > > > >>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Hello everyone and hope you all have very good
> > > > >>> holidays.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> First, this email is not directed on the author
> or
> > > > >> the
> > > > >>> PR
> > > > >>>>>>>>>>>>>>>>>>>
> > https://github.com/apache/incubator-druid/pull/6794
> > > > >>> it
> > > > >>>>>>>>>>> self,
> > > > >>>>>>>>>>>>> but
> > > > >>>>>>>>>>>>>> i
> > > > >>>>>>>>>>>>>>>> see
> > > > >>>>>>>>>>>>>>>>>>> this PR as a perfect example.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> One of the foundation of Apache Way or what i
> would
> > > > >>>>> simply
> > > > >>>>>>>>>>> call
> > > > >>>>>>>>>>>>>> open
> > > > >>>>>>>>>>>>>>>>>> source
> > > > >>>>>>>>>>>>>>>>>>> community driven development is that "Technical
> > > > >>> decisions
> > > > >>>>>> are
> > > > >>>>>>>>>>>>>>>> discussed,
> > > > >>>>>>>>>>>>>>>>>>> decided, and archived publicly.
> > > > >>>>>>>>>>>>>>>>>>> developpement"
> > > > >>>>>>>>>>>>>>>>>>> Which means that big technical  changes such as
> the
> > > > >> one
> > > > >>>>>>>>>>> brought
> > > > >>>>>>>>>>>>> by
> > > > >>>>>>>>>>>>>>>> #/6794
> > > > >>>>>>>>>>>>>>>>>>> should have started as a proposal and round of
> > > > >>>>> discussions
> > > > >>>>>>>>>>>> about
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> major
> > > > >>>>>>>>>>>>>>>>>>> changes designs not as 11K line of code.
> > > > >>>>>>>>>>>>>>>>>>> I believe such openness will promote a lot of
> good
> > > > >>>>> benefits
> > > > >>>>>>>>>>>> such
> > > > >>>>>>>>>>>>>> as:
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> - ensures community health and growth.
> > > > >>>>>>>>>>>>>>>>>>> - ensures everyone can participate not only the
> > > > >> authors
> > > > >>>>> and
> > > > >>>>>>>>>>> his
> > > > >>>>>>>>>>>>>>>>>> co-workers.
> > > > >>>>>>>>>>>>>>>>>>> - ensures that the project is driven by the
> > community
> > > > >>> and
> > > > >>>>>>>>>>> not a
> > > > >>>>>>>>>>>>>> given
> > > > >>>>>>>>>>>>>>>>>>> company or an individual.
> > > > >>>>>>>>>>>>>>>>>>> - ensures that there is consensus (not saying
> 100%
> > > > >>>>>>>>>>> agreement;)
> > > > >>>>>>>>>>>>>>> however
> > > > >>>>>>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>>>>>>> means that all individuals will accept the
> current
> > > > >>>>> progress
> > > > >>>>>>>>>>> on
> > > > >>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> project
> > > > >>>>>>>>>>>>>>>>>>> until some better proposal is put forth.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Personally such BIG offline PR makes me feel
> > excluded
> > > > >>> and
> > > > >>>>>>>>>>>> doesn't
> > > > >>>>>>>>>>>>>>> give
> > > > >>>>>>>>>>>>>>>>>> me a
> > > > >>>>>>>>>>>>>>>>>>> sense that i belong to  a community at all.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> To prevent such off list development i think as a
> > > > >> Druid
> > > > >>>>>>>>>>>> Community
> > > > >>>>>>>>>>>>>> we
> > > > >>>>>>>>>>>>>>>> need
> > > > >>>>>>>>>>>>>>>>>>> to stick to the apache way “If it didn’t happen
> on
> > > > >> the
> > > > >>>>>>>>>>> mailing
> > > > >>>>>>>>>>>>>> list,
> > > > >>>>>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>>>>>>> didn’t happen.”
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> I would appreciate if some of the Apache mentor
> > help
> > > > >>> with
> > > > >>>>>>>>>>> this.
> > > > >>>>>>>>>>>>>>>>>>> Thanks
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>
> > > > >>>
> > ---------------------------------------------------------------------
> > > > >>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > > > >>> dev-unsubscribe@druid.apache.org
> > > > >>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > >>>>> dev-help@druid.apache.org
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > > > >> dev-unsubscribe@druid.apache.org
> > > > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > >>> dev-help@druid.apache.org
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> --
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> B-Slim
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>
> > > > >>>
> > > >
> > _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>
> > > ---------------------------------------------------------------------
> > > > >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > >>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>
> > ---------------------------------------------------------------------
> > > > >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > >>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > For additional commands, e-mail: dev-help@druid.apache.org
> > > >
> > > >
> > >
> >
>

Re: Off list major development

[Jihoon Son <gh...@gmail.com>]

Roman,

> Jihoon in
https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
and later Gian in this thread suggested that _every_ piece of work that
should be labelled as "Design Review" according to the current rules should
be accompanied by an issue. I don't agree with this, there are some PRs as
small as a few dozens of lines of code, that add some configuration
parameter and therefore should be labelled "Design Review". I don't thing a
separate proposal issue is needed for them, and even for a little larger
PRs too.

What I'm concerned with is how people feel if their design is not accepted
even though they wrote code. Of course, as Clint said, sometimes code helps
better understanding of the proposal. But, I believe this is the case when
the proposal is quite complicated and not easy to understand without code.
Also the authors should be aware of that they might rewrite the entire code
if the design should be changed.

If writing code is simple, I don't see why the authors don't wait until the
review for their proposal is finished.

Jihoon

On Fri, Jan 11, 2019 at 9:51 AM Fangjin Yang <fa...@imply.io> wrote:

> I agree with Gian, as an Apache committer, your responsibility is for the
> betterment of the project. I agree it is in the best interest of the
> project to stop thinking about what orgs people belong to. We are all a
> part of the Apache software foundation, regardless of what our roles and
> titles are outside of it.
>
> On Fri, Jan 11, 2019 at 2:22 AM Roman Leventov <le...@gmail.com>
> wrote:
>
> > It's not that people from one org could abuse the project and push some
> > change, but that they have similar perspective (bubble effect) and some
> > important aspects of a large feature could escape their attention.
> >
> > I suggest it to be not a rigid rule, but a recommendation for authors of
> > large proposals to try to attract reviewers from other orgs.
> >
> > On Fri, 11 Jan 2019 at 02:51, Julian Hyde <jh...@apache.org> wrote:
> >
> > > I agree with Gian.
> > >
> > > As an Apache committer, you only have one affiliation: you are working
> in
> > > the best interests of the project.
> > >
> > > Obviously, in the real world there are other pressures. But we do our
> > best
> > > to compensate for them.
> > >
> > > Also, as a a community we try to design our process so as to avoid
> undue
> > > influences. For instance, when I advocate for logging cases early, I am
> > > trying to mitigate the effect of product managers and VPs of
> engineering,
> > > who like to have their say in meeting rooms rather than on public
> mailing
> > > lists. That’s just one example; if we see other influences at play,
> let’s
> > > evolve our process to try to level the playing field.
> > >
> > > Julian
> > >
> > >
> > > > On Jan 10, 2019, at 10:40 AM, Gian Merlino <gi...@apache.org> wrote:
> > > >
> > > >>> What do community members think about also making a requirement
> that
> > > >>> "Design Review" PRs and proposals are reviewed by at least two
> people
> > > > with
> > > >>> different affiliation?
> > > >> This seems pretty reasonable to me. I haven't found anything in
> Apache
> > > >> voting procedure docs (
> https://www.apache.org/foundation/voting.html)
> > > that
> > > >> seems to explicitly forbid something like this yet at least.
> > > >
> > > > On the other hand, my understanding of the Apache Way would mean that
> > > this
> > > > kind of rule doesn't make sense. In particular from
> > > > https://www.apache.org/foundation/how-it-works.html: "We firmly
> > believe
> > > in
> > > > hats. Your role at the ASF is one assigned to you personally, and is
> > > > bestowed on you by your peers. It is not tied to your job or current
> > > > employer or company." That sentiment seems incompatible with making
> > > > explicit rules about organizational diversity in voting. I have also
> > > heard
> > > > a few people say things like: people are supposed to represent
> > > themselves,
> > > > not their employers.
> > > >
> > > > Obviously, though, people's actions and opinions are influenced by
> > their
> > > > employer. IMO, a better way to approach a potential problem there is
> > that
> > > > if people from a particular organization end up behaving abusively,
> > then
> > > > PMC members from other organizations (or in extreme situations the
> > Apache
> > > > board itself) should tell them to knock it off. And then do more
> > serious
> > > > things if the inappropriate behavior continues. This kind
> > > >
> > > > The other thing you brought up, promoting more shared ownership and
> > > > awareness, I am hopeful that separating proposals from PRs will help
> > with
> > > > that. One reason is that it takes much less time to understand a well
> > > > written proposal than it takes to understand a PR. Code is very clear
> > but
> > > > it is also very verbose and takes a while to read and understand. So
> in
> > > > theory changing how we operate, in this way, should promote more
> > > > understanding of more people across the code base.
> > > >
> > > > On Thu, Jan 10, 2019 at 1:34 AM Clint Wylie <cl...@imply.io>
> > > wrote:
> > > >
> > > >>>
> > > >>> What do community members think about also making a requirement
> that
> > > >>> "Design Review" PRs and proposals are reviewed by at least two
> people
> > > >> with
> > > >>> different affiliation?
> > > >>
> > > >>
> > > >> This seems pretty reasonable to me. I haven't found anything in
> Apache
> > > >> voting procedure docs (
> https://www.apache.org/foundation/voting.html)
> > > that
> > > >> seems to explicitly forbid something like this yet at least.
> > > >>
> > > >> On Wed, Jan 9, 2019 at 8:52 PM Roman Leventov <
> leventov.ru@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> What do community members think about also making a requirement
> that
> > > >>> "Design Review" PRs and proposals are reviewed by at least two
> people
> > > >> with
> > > >>> different affiliation? IMO it's a good idea, because it ensures
> that
> > > >>> different interests are taken into account. Also it pushes people
> to
> > > >> engage
> > > >>> with work done in other parts of Druid, improving shared code
> owning
> > > and
> > > >>> awareness.
> > > >>>
> > > >>> Or it's against the Apache Way because it's assumed that there are
> no
> > > >>> company boundaries within the community?
> > > >>>
> > > >>> On Thu, 10 Jan 2019 at 11:45, Roman Leventov <
> leventov.ru@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> I see two important reasons why it makes sense to file an issue
> and
> > > >>>> probably announce it in the mailing list, before writing a lot of
> > > code,
> > > >>>> despite not having a clear picture of what it will be and any
> > > >> performance
> > > >>>> data:
> > > >>>> 1) somebody could already work on this problem privately in
> > parallel,
> > > >> it
> > > >>>> allows to avoid clash in people's works
> > > >>>> 2) some people could quickly think about the problem field and
> share
> > > >>>> high-level ideas that could wildly change the direction in which
> the
> > > >>> author
> > > >>>> (the person who is going to write code) will move in his work from
> > > >> early
> > > >>> on.
> > > >>>>
> > > >>>> Jihoon in
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> > > >>> and
> > > >>>> later Gian in this thread suggested that _every_ piece of work
> that
> > > >>> should
> > > >>>> be labelled as "Design Review" according to the current rules
> should
> > > be
> > > >>>> accompanied by an issue. I don't agree with this, there are some
> PRs
> > > as
> > > >>>> small as a few dozens of lines of code, that add some
> configuration
> > > >>>> parameter and therefore should be labelled "Design Review". I
> don't
> > > >>> thing a
> > > >>>> separate proposal issue is needed for them, and even for a little
> > > >> larger
> > > >>>> PRs too.
> > > >>>>
> > > >>>> For the same reason, I also don't see the point of renaming
> "Design
> > > >>>> Review" into "Proposal", as well as separating "Design Review"
> into
> > > >>>> "Proposal" and something like "API Review". I think a single
> "Design
> > > >>>> Review" tag handles it well.
> > > >>>>
> > > >>>> Gian mentioned an idea that PRs that follow a "Design Review"
> > proposal
> > > >>>> issue shouldn't be "Design Review" themselves. I don't agree with
> > > >> this, I
> > > >>>> think that actual code and performance data are important inputs
> > that
> > > >>>> should be re-evaluated at least by two people. I even think that
> > it's
> > > >>> very
> > > >>>> desirable that at least two people read _each line of production
> > code_
> > > >> in
> > > >>>> large PRs, although it's not what was done historically in Druid,
> > > >> because
> > > >>>> large bodies of newly added code, with whole new classes and
> > > subsystems
> > > >>>> added, are also coincidentally tested worse than already existing
> > > >> classes
> > > >>>> and subsystems, including in production. It seems to me that those
> > > huge
> > > >>>> code influxes is a major source of bugs, that could later take
> years
> > > to
> > > >>>> squeeze out from the codebase.
> > > >>>>
> > > >>>> On Wed, 9 Jan 2019 at 08:24, Clint Wylie <cl...@imply.io>
> > > wrote:
> > > >>>>
> > > >>>>> Apologies for the delayed response.
> > > >>>>>
> > > >>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > > >>> slim.bouguerra@gmail.com>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> I am wondering here what is the case where code first is better?
> > > >>>>>>
> > > >>>>>
> > > >>>>> I don't think it's wrong to share ideas as early as possible, and
> > > >> after
> > > >>>>> this discussion I think I am in favor of it too. I just meant
> that
> > I
> > > >>> don't
> > > >>>>> think it's always necessarily the most productive discussion
> until
> > > >> code
> > > >>>>> exists sometimes, with the types of thing I am thinking of are
> > almost
> > > >>>>> entirely limited to cases where things might sound good to anyone
> > on
> > > >>> paper
> > > >>>>> but in reality need a large amount of experiments conducted and
> > > >>>>> observations collected to determine that something is actually
> > worth
> > > >>>>> doing,
> > > >>>>> which I imagine is mainly things like reworking internals for
> > > >>> performance
> > > >>>>> improvements.
> > > >>>>>
> > > >>>>> In the case of my combined proposal PR, I needed to prove that
> the
> > > >>> thing I
> > > >>>>> was working on was a good idea... and it wasn't directly. But I
> > came
> > > >> up
> > > >>>>> with another idea during the course of experiment turned into
> > > >> something
> > > >>>>> compelling, so an initial proposal would have looked quite a lot
> > > >>> different
> > > >>>>> than what I ended up with. Once I had proven to myself that it
> was
> > a
> > > >>> good
> > > >>>>> idea, then I was comfortable sharing with the wider community.
> I'm
> > > not
> > > >>>>> certain how this would play out in an always proposal first
> model,
> > > >> maybe
> > > >>>>> the first proposal exists, I personally reject it after updating
> > with
> > > >>>>> experiment results show it's a bad idea, continue experimenting
> and
> > > >>> raise
> > > >>>>> a
> > > >>>>> new one after the experiments start looking promising?
> > > >>>>>
> > > >>>>>
> > > >>>>>> Let's not be naive this is very rare that a contributor will
> > accept
> > > >>> that
> > > >>>>>> his work is to be thrown, usually devs takes coding as personal
> > > >>> creation
> > > >>>>>> and they get attached to it.
> > > >>>>>>
> > > >>>>>
> > > >>>>> I agree, just because a handful of the committers have this
> > attitude,
> > > >> it
> > > >>>>> isn't fair to expect the wider community to also, that's why I am
> > in
> > > >>> favor
> > > >>>>> of formalizing the process.
> > > >>>>>
> > > >>>>> Can you please explain what is overbearing ? what can be changed
> to
> > > >> make
> > > >>>>> it
> > > >>>>>> easy ?
> > > >>>>>> Most of the points are kind of the actual questions that you
> want
> > to
> > > >>>>>> address before hand anyway isn't ?
> > > >>>>>>
> > > >>>>>
> > > >>>>> Sorry for the confusion, I said it's "not overbearing", I think
> > it's
> > > >>> fine.
> > > >>>>>
> > > >>>>> What are the complaints ?
> > > >>>>>
> > > >>>>>
> > > >>>>> Is this and other previous threads not a complaint about opening
> a
> > > >> large
> > > >>>>> PR
> > > >>>>> without a proposal? :) I just mean that formalizing the process,
> > even
> > > >>> if a
> > > >>>>> proposal has a reference PR opened with it near concurrently,
> could
> > > >>>>> prevent
> > > >>>>> these discussions from happening in the future because ground
> rules
> > > >> have
> > > >>>>> been set and we are all on the same page I guess.
> > > >>>>>
> > > >>>>> I think we should also probably consider renaming the "design
> > review"
> > > >>>>> label
> > > >>>>> to "proposal" or something to make it more clear that a PR is
> > > >> associated
> > > >>>>> with a proposal. It might also be worth considering using github
> > > >>> projects
> > > >>>>> for particularly large things that involve multiple follow up
> PRs,
> > > >> but I
> > > >>>>> haven't used them in much depth to know if they add anything.
> > > >>>>>
> > > >>>>> It seems like we are all converging on agreement to do a github
> > issue
> > > >>>>> proposal for larger changes (I would vote for announcing them on
> > the
> > > >> dev
> > > >>>>> list too for more visibility), so that design review is separated
> > > from
> > > >>>>> code
> > > >>>>> review. I guess my main concern was not wanting to discourage
> > > >>>>> experimentation by walling it off behind mandatory discussions,
> but
> > > >> the
> > > >>>>> more I think about it,  it doesn't technically change much about
> > this
> > > >>>>> process, it just requires a more formal proposal to accompany any
> > > >>>>> experiments that are shared as a PR.
> > > >>>>>
> > > >>>>>
> > > >>>>> On Tue, Jan 8, 2019 at 12:28 PM Gian Merlino <gi...@apache.org>
> > > wrote:
> > > >>>>>
> > > >>>>>> I think for us, choosing to use GitHub issues as discussion
> > threads
> > > >>> for
> > > >>>>>> potential 'major' contributions would be a good idea, especially
> > if
> > > >> we
> > > >>>>>> encourage people to start them before PRs show up. Definitely
> > agree
> > > >>> that
> > > >>>>>> all contributors should go through the same process -- I
> couldn't
> > > >> see
> > > >>> it
> > > >>>>>> working well any other way.
> > > >>>>>>
> > > >>>>>> On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <jh...@apache.org>
> > > >> wrote:
> > > >>>>>>
> > > >>>>>>> Statically, yes, GitHub PRs are the same as GitHub cases. But
> > > >>>>>> dynamically,
> > > >>>>>>> they are different, because you can only log a PR when you have
> > > >>>>> finished
> > > >>>>>>> work.
> > > >>>>>>>
> > > >>>>>>> A lot of other Apache projects use JIRA, so there is a clear
> > > >>>>> distinction
> > > >>>>>>> between cases and contributions. JIRA cases, especially when
> > > >> logged
> > > >>>>> early
> > > >>>>>>> in the lifecycle of a contribution, become long-running
> > > >> conversation
> > > >>>>>>> threads with a lot of community participation. If the Druid
> chose
> > > >> to
> > > >>>>> do
> > > >>>>>> so,
> > > >>>>>>> GitHub cases could be the same.
> > > >>>>>>>
> > > >>>>>>> Be careful that you do not treat “potential contributors” (by
> > > >> which
> > > >>> I
> > > >>>>>>> presume you mean non-committers) differently from committers
> and
> > > >> PMC
> > > >>>>>>> members. Anyone starting a major piece of work should follow
> the
> > > >>> same
> > > >>>>>>> process. (Experienced committers probably have a somewhat
> better
> > > >>> idea
> > > >>>>>> what
> > > >>>>>>> work will turn out to be “major”, so they get a little more
> > > >> leeway.)
> > > >>>>>>>
> > > >>>>>>> Julian
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>> On Jan 7, 2019, at 12:10 PM, Gian Merlino <gi...@apache.org>
> > > >>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> I don't think there's a need to raise issues for every
> change: a
> > > >>>>> small
> > > >>>>>>> bug
> > > >>>>>>>> fix or doc fix should just go straight to PR. (GitHub PRs show
> > > >> up
> > > >>> as
> > > >>>>>>> issues
> > > >>>>>>>> in the issue-search UI/API, so it's not like this means the
> > > >> patch
> > > >>>>> has
> > > >>>>>> no
> > > >>>>>>>> corresponding issue -- in a sense the PR _is_ the issue.)
> > > >>>>>>>>
> > > >>>>>>>> I do think it makes sense to encourage potential contributors
> to
> > > >>>>> write
> > > >>>>>> to
> > > >>>>>>>> the dev list or raise an issue if they aren't sure if
> something
> > > >>>>> would
> > > >>>>>>> need
> > > >>>>>>>> to go through a more heavy weight process.
> > > >>>>>>>>
> > > >>>>>>>> Fwiw we do have a set of 'design review' criteria already (we
> > > >> had
> > > >>> a
> > > >>>>>>>> discussion about this a couple years ago) at:
> > > >>>>>>>> http://druid.io/community/#getting-your-changes-accepted. So
> we
> > > >>>>>>> wouldn't be
> > > >>>>>>>> starting from zero on defining that. We set it up back when we
> > > >>> were
> > > >>>>>>> trying
> > > >>>>>>>> to _streamline_ our process -- we used to require two
> non-author
> > > >>> +1s
> > > >>>>>> for
> > > >>>>>>>> _every_ change, even minor ones. The introduction of design
> > > >> review
> > > >>>>>>> criteria
> > > >>>>>>>> was meant to classify which PRs need that level of review and
> > > >>> which
> > > >>>>>> ones
> > > >>>>>>>> are minor and can be merged with less review. I do think it
> > > >> helped
> > > >>>>> with
> > > >>>>>>>> getting minor PRs merged more quickly. The list of criteria
> is,
> > > >>>>>>>>
> > > >>>>>>>> - Major architectural changes or API changes
> > > >>>>>>>> - HTTP requests and responses (e. g. a new HTTP endpoint)
> > > >>>>>>>> - Interfaces for extensions
> > > >>>>>>>> - Server configuration (e. g. altering the behavior of a
> config
> > > >>>>>> property)
> > > >>>>>>>> - Emitted metrics
> > > >>>>>>>> - Other major changes, judged by the discretion of Druid
> > > >>> committers
> > > >>>>>>>>
> > > >>>>>>>> Some of it is subjective, but it has been in place for a
> while,
> > > >> so
> > > >>>>> it's
> > > >>>>>>> at
> > > >>>>>>>> least something we are relatively familiar with.
> > > >>>>>>>>
> > > >>>>>>>> On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <jhyde@apache.org
> >
> > > >>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Small contributions don’t need any design review, whereas
> large
> > > >>>>>>>>> contributions need significant review. I don’t think we
> should
> > > >>>>> require
> > > >>>>>>> an
> > > >>>>>>>>> additional step for those (many) small contributions. But who
> > > >>>>> decides
> > > >>>>>>>>> whether a contribution fits into the small or large category?
> > > >>>>>>>>>
> > > >>>>>>>>> I think the solution is for authors to log a case (or send an
> > > >>>>> email to
> > > >>>>>>>>> dev) before they start work on any contribution. Then
> > > >> committers
> > > >>>>> can
> > > >>>>>>>>> request a more heavy-weight process if they think it is
> needed.
> > > >>>>>>>>>
> > > >>>>>>>>> Julian
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <gi...@apache.org>
> > > >>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>> It sounds like splitting design from code review is a common
> > > >>> theme
> > > >>>>>> in a
> > > >>>>>>>>> few
> > > >>>>>>>>>> of the posts here. How does everyone feel about making a
> point
> > > >>> of
> > > >>>>>>>>>> encouraging design reviews to be done as issues, separate
> from
> > > >>> the
> > > >>>>>> pull
> > > >>>>>>>>>> request, with the expectations that (1) the design review
> > > >> issue
> > > >>>>>>>>>> ("proposal") should generally appear somewhat _before_ the
> > > >> pull
> > > >>>>>>> request;
> > > >>>>>>>>>> (2) pull requests should _not_ have design review happen on
> > > >>> them,
> > > >>>>>>> meaning
> > > >>>>>>>>>> there should no longer be PRs with design review tags, and
> we
> > > >>>>> should
> > > >>>>>>> move
> > > >>>>>>>>>> the design review approval process to the issue rather than
> > > >> the
> > > >>>>> PR.
> > > >>>>>>>>>>
> > > >>>>>>>>>> For (1), even if we encourage design review discussions to
> > > >> start
> > > >>>>>>> before a
> > > >>>>>>>>>> pull request appears, I don't see an issue with them running
> > > >>>>>>> concurrently
> > > >>>>>>>>>> for a while at some point.
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <
> > > >> jonwei@apache.org>
> > > >>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Thanks for raising these concerns!
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> My initial thoughts:
> > > >>>>>>>>>>> - I agree that separation of design review and code-level
> > > >>> review
> > > >>>>> for
> > > >>>>>>>>> major
> > > >>>>>>>>>>> changes would be more efficient
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - I agree that a clear, more formalized process for
> handling
> > > >>>>> major
> > > >>>>>>>>> changes
> > > >>>>>>>>>>> would be helpful for contributors:
> > > >>>>>>>>>>> - Define what is considered a major change
> > > >>>>>>>>>>> - Define a standard proposal structure, KIP-style proposal
> > > >>> format
> > > >>>>>>>>> sounds
> > > >>>>>>>>>>> good to me
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - I think it's too rigid to have a policy of "no code at
> all
> > > >>> with
> > > >>>>>> the
> > > >>>>>>>>>>> initial proposal"
> > > >>>>>>>>>>> - Code samples can be useful references for understanding
> > > >>> aspects
> > > >>>>>> of a
> > > >>>>>>>>>>> design
> > > >>>>>>>>>>> - In some cases it's necessary to run experiments to fully
> > > >>>>>> understand
> > > >>>>>>> a
> > > >>>>>>>>>>> problem and determine an appropriate design, or to
> determine
> > > >>>>> whether
> > > >>>>>>>>>>> something is even worth doing before committing to the work
> > > >> of
> > > >>>>>>> fleshing
> > > >>>>>>>>> out
> > > >>>>>>>>>>> a proposal, prototype code is a natural outcome of that and
> > > >> I'm
> > > >>>>> not
> > > >>>>>>>>> against
> > > >>>>>>>>>>> someone providing such code for reference
> > > >>>>>>>>>>> - I tend to view design/code as things that are often
> > > >> developed
> > > >>>>>>>>>>> simultaneously in an intertwined way
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Let's not be naive this is very rare that a contributor
> will
> > > >>>>> accept
> > > >>>>>>>>> that
> > > >>>>>>>>>>> his work is to be thrown, usually devs takes coding as
> > > >> personal
> > > >>>>>>> creation
> > > >>>>>>>>>>> and they get attached to it.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> If we have a clear review process that emphasizes the need
> > > >> for
> > > >>>>> early
> > > >>>>>>>>>>> consensus building, with separate design and code review,
> > > >> then
> > > >>> I
> > > >>>>>> feel
> > > >>>>>>>>> we've
> > > >>>>>>>>>>> done enough and don't need a hard rule against having some
> > > >> code
> > > >>>>>> linked
> > > >>>>>>>>> with
> > > >>>>>>>>>>> the initial proposal. If a potential contributor then still
> > > >>>>> wants to
> > > >>>>>>> go
> > > >>>>>>>>>>> ahead and write a lot of code that may be rejected or
> change
> > > >>>>>>>>> significantly,
> > > >>>>>>>>>>> the risks were made clear.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Once code is written hard to think abstract.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I can see the validity of the concern, but I personally
> don't
> > > >>>>> see it
> > > >>>>>>> as
> > > >>>>>>>>> a
> > > >>>>>>>>>>> huge risk. My impression from the Druid PR reviews I've
> seen
> > > >> is
> > > >>>>> that
> > > >>>>>>> our
> > > >>>>>>>>>>> reviewers are able to keep abstract design vs.
> implementation
> > > >>>>>> details
> > > >>>>>>>>>>> separate and consider alternate designs when reviewing.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> To summarize I think it's probably enough to have a policy
> > > >>> along
> > > >>>>> the
> > > >>>>>>>>> lines
> > > >>>>>>>>>>> of:
> > > >>>>>>>>>>> - Create more formalized guidelines for proposals and what
> > > >>>>> changes
> > > >>>>>>>>> require
> > > >>>>>>>>>>> proposals
> > > >>>>>>>>>>> - Separate design and code review for major changes, with
> > > >>> design
> > > >>>>>>> review
> > > >>>>>>>>>>> first, code-level review after reaching consensus on the
> > > >>> design.
> > > >>>>>>>>>>> - Code before the design review is completed is just for
> > > >>>>> reference,
> > > >>>>>>> not
> > > >>>>>>>>>>> regarded as a candidate for review/merging.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Jon
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > > >>>>>>>>> slim.bouguerra@gmail.com>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <
> > > >>>>> clint.wylie@imply.io>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> I am definitely biased in this matter as an owner of
> > > >> another
> > > >>>>> large
> > > >>>>>>> PR
> > > >>>>>>>>>>>> that
> > > >>>>>>>>>>>>> wasn't preceded by a direct proposal or dev list
> > > >> discussion,
> > > >>>>> and
> > > >>>>>> in
> > > >>>>>>>>>>>> general
> > > >>>>>>>>>>>>> I agree that proposal first is usually better, but I
> think
> > > >> in
> > > >>>>> some
> > > >>>>>>>>>>> rarer
> > > >>>>>>>>>>>>> cases approaching a problem code first *is* the most
> > > >>>>> appropriate
> > > >>>>>> way
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>> have a discussion.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I am wondering here what is the case where code first is
> > > >>> better?
> > > >>>>>>>>>>>> In general when you are writing code you have an idea
> about
> > > >>> what
> > > >>>>>> you
> > > >>>>>>>>> want
> > > >>>>>>>>>>>> to change, why you want to change and why you want to
> change
> > > >>> it.
> > > >>>>>>>>>>>> I do not see what is wrong with sharing this primitive
> ideas
> > > >>> and
> > > >>>>>>>>> thoughts
> > > >>>>>>>>>>>> as an abstract proposal (at least to avoid overlapping)
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I see nothing wrong with it so long as the author
> > > >>>>>>>>>>>>> accepts that the PR is treated as a combined proposal and
> > > >>>>> proof of
> > > >>>>>>>>>>>> concept,
> > > >>>>>>>>>>>>> and fair game to be radically changed via discussion or
> > > >> even
> > > >>>>>>> rejected,
> > > >>>>>>>>>>>>> which sounds like Gian's attitude on the matter and is
> mine
> > > >>> as
> > > >>>>>> well
> > > >>>>>>>>>>> with
> > > >>>>>>>>>>>> my
> > > >>>>>>>>>>>>> compression stuff.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Let's not be naive this is very rare that a contributor
> will
> > > >>>>> accept
> > > >>>>>>>>> that
> > > >>>>>>>>>>>> his work is to be thrown, usually devs takes coding as
> > > >>> personal
> > > >>>>>>>>> creation
> > > >>>>>>>>>>>> and they get attached to it.
> > > >>>>>>>>>>>> To my point you can take a look on some old issue in the
> > > >> Druid
> > > >>>>>> forum
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>
> > > >>
> > >
> >
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> > > >>>>>>>>>>>> and am sure other communities have similar problems.
> > > >>>>>>>>>>>> So leaving the door open to some side cases is not a good
> > > >> idea
> > > >>>>> in
> > > >>>>>> my
> > > >>>>>>>>>>>> opinion and will lead to similar issue in the future.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> This seems to me especially likely to happen in cases
> > > >>>>>>>>>>>>> where an approach still needs proven to be a viable idea
> > > >> *to
> > > >>>>> the
> > > >>>>>>>>>>> author*,
> > > >>>>>>>>>>>>> so that a much more productive discussion can be had in
> the
> > > >>>>> first
> > > >>>>>>>>>>> place.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I think there is a trade off, I don't think we want to
> > > >>>>> discourage
> > > >>>>>>>>>>>>> experimentation by walling it off behind mandatory
> > > >>> discussions
> > > >>>>>>> before
> > > >>>>>>>>>>> it
> > > >>>>>>>>>>>>> can even start, but I do think formalizing the process
> for
> > > >>>>> large
> > > >>>>>>>>>>> changes
> > > >>>>>>>>>>>> is
> > > >>>>>>>>>>>>> a good thing, especially since we probably can't expect
> the
> > > >>>>> wider
> > > >>>>>>>>>>>> community
> > > >>>>>>>>>>>>> to have the same attitude towards a large PR getting
> > > >>> discarded
> > > >>>>> as
> > > >>>>>> a
> > > >>>>>>>>>>>>> committer might. I think the Kafka approach is
> reasonable,
> > > >> a
> > > >>>>> bit
> > > >>>>>>> more
> > > >>>>>>>>>>>>> formal than our design review process but not
> overbearing.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Can you please explain what is overbearing ? what can be
> > > >>>>> changed to
> > > >>>>>>>>> make
> > > >>>>>>>>>>> it
> > > >>>>>>>>>>>> easy ?
> > > >>>>>>>>>>>> Most of the points are kind of the actual questions that
> you
> > > >>>>> want
> > > >>>>>> to
> > > >>>>>>>>>>>> address before hand anyway isn't ?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Going code first
> > > >>>>>>>>>>>>> should be in general discouraged, but when it does
> happen,
> > > >> it
> > > >>>>>> seems
> > > >>>>>>>>>>> like
> > > >>>>>>>>>>>>> opening DIP/an issue/starting a mailing list thread or
> > > >>>>> whatever we
> > > >>>>>>> go
> > > >>>>>>>>>>>> with
> > > >>>>>>>>>>>>> to have a more high level design discussion alongside the
> > > >>>>>> reference
> > > >>>>>>> PR
> > > >>>>>>>>>>>>> could alleviate some of these complaints?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> What are the complaints ?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> +1 for "DIP" heh, I think making
> > > >>>>>>>>>>>>> them in the form of github issues is probably
> appropriate,
> > > >>>>> with a
> > > >>>>>>> dev
> > > >>>>>>>>>>>> list
> > > >>>>>>>>>>>>> thread to announce them perhaps?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I think  github issue with [Proposal] header like
> > > >>>>>>>>>>>> https://github.com/apache/incubator-druid/issues/4349 is
> > > >> good
> > > >>>>> to
> > > >>>>>> me,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <
> > > >>>>> bslim@apache.org>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks everyone for interacting with this thread.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> The fact that i, Roman, Jihoon  and others in the past
> (FJ
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>
> > > >>
> > >
> >
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ
> > > >>>>> )
> > > >>>>>>>>>>>>>> raised this point indicates that PRs without a proposal
> > > >> are
> > > >>>>>> indeed
> > > >>>>>>> an
> > > >>>>>>>>>>>>> issue
> > > >>>>>>>>>>>>>> and we need to solve it.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Something Similar to KIP maybe called DIPs is fine with
> > > >> me.
> > > >>>>>>>>>>>>>> What i strive to see is the following:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> [Step 1] formalize what is the kind of work that needs a
> > > >>>>> formal
> > > >>>>>>>>>>>>> Proposal, I
> > > >>>>>>>>>>>>>> think Roman and Jihoon has already covered that pretty
> > > >> well.
> > > >>>>> am
> > > >>>>>> +1
> > > >>>>>>> on
> > > >>>>>>>>>>>>> that.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I am strongly in favor of the separation of Proposal
> > > >> Review
> > > >>>>> and
> > > >>>>>>>>>>> (later)
> > > >>>>>>>>>>>>>> Code review PRs. My  main reasons:
> > > >>>>>>>>>>>>>> Most importantly code reviewing will introduce lot of
> > > >> noise
> > > >>>>> and
> > > >>>>>>> will
> > > >>>>>>>>>>>>>> ultimately make  the GitHub page unreadable.
> > > >>>>>>>>>>>>>> Avoid overlapping of work.
> > > >>>>>>>>>>>>>> Once code is written hard to think abstract.
> > > >>>>>>>>>>>>>> Separate page for Design review later can always be used
> > > >> it
> > > >>>>> as a
> > > >>>>>>>>>>> Design
> > > >>>>>>>>>>>>>> document that is readable and code free-ish.
> > > >>>>>>>>>>>>>> As i said the goal of this first round is to see if the
> > > >>>>> community
> > > >>>>>>>>>>> agree
> > > >>>>>>>>>>>>>> about such change, then make the process of design more
> > > >>>>> inclusive
> > > >>>>>>>>>>> thus
> > > >>>>>>>>>>>>>> other contributors can submit a counter proposals.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> [Step 2] IF everybody agree about that point Step 2 is
> to
> > > >>>>> define
> > > >>>>>>>>>>> which
> > > >>>>>>>>>>>>>> medium is used to Publish a primitive form of a CODE
> FREE
> > > >>>>>> Abstract
> > > >>>>>>>>>>>>> Proposal
> > > >>>>>>>>>>>>>> containing at least the following bullet points.
> > > >>>>>>>>>>>>>> - The problem description and motivation
> > > >>>>>>>>>>>>>> - Overview of the proposed change
> > > >>>>>>>>>>>>>> - Operational impact (compatibility/ plans to upgrades)
> > > >>> public
> > > >>>>>> API
> > > >>>>>>>>>>>>> changes,
> > > >>>>>>>>>>>>>> configuration changes, algorithm, and so on
> > > >>>>>>>>>>>>>> - Expected benefits and drawbacks
> > > >>>>>>>>>>>>>> - Rationale and alternatives
> > > >>>>>>>>>>>>>> - Estimate Time to Deliver if possible.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> The way i think this can be is a Github issue where
> member
> > > >>> of
> > > >>>>> the
> > > >>>>>>>>>>>>> community
> > > >>>>>>>>>>>>>> will interact via comments and the author will be
> updating
> > > >>> the
> > > >>>>>>>>>>>>> description
> > > >>>>>>>>>>>>>> in the light of comments provided by the community.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> During and near the end of the design discussions the
> > > >>> author/s
> > > >>>>>> can
> > > >>>>>>>>>>>> start
> > > >>>>>>>>>>>>>> writing POCs to help guide the review process this
> > > >> naturally
> > > >>>>> will
> > > >>>>>>> be
> > > >>>>>>>>>>> a
> > > >>>>>>>>>>>>> Pull
> > > >>>>>>>>>>>>>> request with actual code.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> *Now the most important thing is that we need to agree
> > > >> that
> > > >>>>> any
> > > >>>>>>> work
> > > >>>>>>>>>>>> that
> > > >>>>>>>>>>>>>> does not align with this formal process will be ignored
> > > >> and
> > > >>>>> the
> > > >>>>>>>>>>> author
> > > >>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>> be asked to start with a DIP*
> > > >>>>>>>>>>>>>> *That is what i meant with  “If it didn’t happen on the
> > > >>>>> mailing
> > > >>>>>>> list,
> > > >>>>>>>>>>>> it
> > > >>>>>>>>>>>>>> didn’t happen.”*
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks and happy coding!
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <
> > > >>> gian@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> One of the advantages I see with a more formal process
> is
> > > >>>>> (like
> > > >>>>>>>>>>> Kafka
> > > >>>>>>>>>>>>>> KIPs)
> > > >>>>>>>>>>>>>>> is that it levels the playing field a bit and sets some
> > > >>>>> ground
> > > >>>>>>>>>>> rules
> > > >>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>> working together. In a way it can help encourage
> > > >>>>> contributions
> > > >>>>>> by
> > > >>>>>>>>>>>>> making
> > > >>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>> clear what is expected of potential contributors.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> We have a design review process today that is not as
> > > >> formal
> > > >>>>> as
> > > >>>>>>>>>>> KIPs,
> > > >>>>>>>>>>>>> but
> > > >>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>> somewhat heavier than the one you describe. Maybe we
> > > >> could
> > > >>>>> tweak
> > > >>>>>>>>>>> our
> > > >>>>>>>>>>>>>>> current one by starting to do design reviews separately
> > > >>> from
> > > >>>>>> PRs.
> > > >>>>>>>>>>>> i.e.,
> > > >>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>> anything that meets our 'design review' criteria, do
> that
> > > >>> on
> > > >>>>> the
> > > >>>>>>>>>>> dev
> > > >>>>>>>>>>>>> list
> > > >>>>>>>>>>>>>>> or in a separate issue, and keep the PR focused on
> > > >>> code-level
> > > >>>>>>>>>>> stuff.
> > > >>>>>>>>>>>>> That
> > > >>>>>>>>>>>>>>> way we don't end up trying to do both at once. And it
> > > >> makes
> > > >>>>> it
> > > >>>>>>>>>>> easier
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> start talking about design before the code is ready,
> > > >> which
> > > >>>>> would
> > > >>>>>>> be
> > > >>>>>>>>>>>>>> better.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <
> > > >>> jhyde@apache.org
> > > >>>>>>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> It’s really hard to say no to a contribution when
> > > >> someone
> > > >>>>> has
> > > >>>>>> put
> > > >>>>>>>>>>>> in
> > > >>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>> significant amount of work.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> The following approach is simple and works really
> well:
> > > >>>>> Before
> > > >>>>>>>>>>> you
> > > >>>>>>>>>>>>>> start
> > > >>>>>>>>>>>>>>>> work, log a case, describing the problem. When you
> have
> > > >>> some
> > > >>>>>>>>>>> ideas
> > > >>>>>>>>>>>>>> about
> > > >>>>>>>>>>>>>>>> design, add those to the case. When you have a code
> > > >>> branch,
> > > >>>>> add
> > > >>>>>>>>>>> its
> > > >>>>>>>>>>>>> URL
> > > >>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>> the case. And so forth. At any point in the
> proceedings,
> > > >>>>> people
> > > >>>>>>>>>>> can
> > > >>>>>>>>>>>>>> chime
> > > >>>>>>>>>>>>>>>> in with their opinions.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> In my opinion, a formal “design review” process is not
> > > >>>>>> necessary.
> > > >>>>>>>>>>>>> Just
> > > >>>>>>>>>>>>>>>> build consensus iteratively, by starting the
> > > >> conversation
> > > >>>>> early
> > > >>>>>>>>>>> in
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> process.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Julian
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <
> > > >>> gian@apache.org
> > > >>>>>>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> In this particular case: please consider the PR as a
> > > >>>>> proposal.
> > > >>>>>>>>>>>>> Don't
> > > >>>>>>>>>>>>>>> feel
> > > >>>>>>>>>>>>>>>>> like just because there is code there that takes a
> > > >>> certain
> > > >>>>>>>>>>>>> approach,
> > > >>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>> the approach is somehow sacred. I had to implement
> > > >>>>> something
> > > >>>>>> to
> > > >>>>>>>>>>>>>>>> crystallize
> > > >>>>>>>>>>>>>>>>> my own thinking about how the problem could be
> > > >>> approached.
> > > >>>>> I
> > > >>>>>>>>>>>> won't
> > > >>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>> disappointed if, as a community, we decide a
> different
> > > >>>>>>>>>>> direction
> > > >>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>> better
> > > >>>>>>>>>>>>>>>>> and the code all gets thrown away. That's one of the
> > > >>>>> reasons
> > > >>>>>>>>>>>> that I
> > > >>>>>>>>>>>>>>>> removed
> > > >>>>>>>>>>>>>>>>> the 0.14.0 milestone that was added to the patch. (I
> > > >>> don't
> > > >>>>>> want
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>> rush
> > > >>>>>>>>>>>>>>>> it,
> > > >>>>>>>>>>>>>>>>> nor do I think that's a good idea.)
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> In general: Sounds like we could do with some more
> > > >>>>>>>>>>> formalization
> > > >>>>>>>>>>>>>> around
> > > >>>>>>>>>>>>>>>>> what a proposal looks like, which sorts of changes
> need
> > > >>>>> one,
> > > >>>>>>>>>>> and
> > > >>>>>>>>>>>>> when
> > > >>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>> the dev cycle it is appropriate. FWIW I think Kafka's
> > > >>>>> process
> > > >>>>>>>>>>> is
> > > >>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>> less fine, and would be okay with adopting it for
> Druid
> > > >>> if
> > > >>>>>>>>>>> people
> > > >>>>>>>>>>>>>> like
> > > >>>>>>>>>>>>>>>> it.
> > > >>>>>>>>>>>>>>>>> Right now our standards for what requires a "design
> > > >>> review"
> > > >>>>>> are
> > > >>>>>>>>>>>>> very
> > > >>>>>>>>>>>>>>>>> similar to the Kafka community standards for what
> > > >>> requires
> > > >>>>> a
> > > >>>>>>>>>>> KIP,
> > > >>>>>>>>>>>>> so
> > > >>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>> have some familiarity with those concepts. However we
> > > >>> don't
> > > >>>>>>>>>>>>> separate
> > > >>>>>>>>>>>>>> PR
> > > >>>>>>>>>>>>>>>>> review and proposal discussion as strictly as they
> do,
> > > >>>>> which
> > > >>>>>>>>>>>> seems
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>> the foundation for the feeling of exclusion that is
> > > >> being
> > > >>>>> felt
> > > >>>>>>>>>>>>> here.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Separately: I just redid the description on
> > > >>>>>>>>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794
> to
> > > >>> be
> > > >>>>>> more
> > > >>>>>>>>>>>>>>>> proposal-y.
> > > >>>>>>>>>>>>>>>>> I followed the KIP style:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > >>>>>>>>>>>>>>>> .
> > > >>>>>>>>>>>>>>>>> Please refresh the page and see if it looks more
> > > >> useful.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Gian
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <
> > > >>>>> jhyde@apache.org
> > > >>>>>>>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Slim,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I agree with your points that offline development is
> > > >> bad
> > > >>>>> for
> > > >>>>>>>>>>>>>>> community.
> > > >>>>>>>>>>>>>>>>>> But I don’t think you need much mentor help. You
> have
> > > >>>>> raised
> > > >>>>>>>>>>>> valid
> > > >>>>>>>>>>>>>>>> issues
> > > >>>>>>>>>>>>>>>>>> and the Druid community needs to decide what its
> > > >>>>> development
> > > >>>>>>>>>>>>>> practices
> > > >>>>>>>>>>>>>>>>>> should be.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Julian
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
> > > >>>>>>>>>>> bslim@apache.org>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Hello everyone and hope you all have very good
> > > >>> holidays.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> First, this email is not directed on the author or
> > > >> the
> > > >>> PR
> > > >>>>>>>>>>>>>>>>>>>
> https://github.com/apache/incubator-druid/pull/6794
> > > >>> it
> > > >>>>>>>>>>> self,
> > > >>>>>>>>>>>>> but
> > > >>>>>>>>>>>>>> i
> > > >>>>>>>>>>>>>>>> see
> > > >>>>>>>>>>>>>>>>>>> this PR as a perfect example.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> One of the foundation of Apache Way or what i would
> > > >>>>> simply
> > > >>>>>>>>>>> call
> > > >>>>>>>>>>>>>> open
> > > >>>>>>>>>>>>>>>>>> source
> > > >>>>>>>>>>>>>>>>>>> community driven development is that "Technical
> > > >>> decisions
> > > >>>>>> are
> > > >>>>>>>>>>>>>>>> discussed,
> > > >>>>>>>>>>>>>>>>>>> decided, and archived publicly.
> > > >>>>>>>>>>>>>>>>>>> developpement"
> > > >>>>>>>>>>>>>>>>>>> Which means that big technical  changes such as the
> > > >> one
> > > >>>>>>>>>>> brought
> > > >>>>>>>>>>>>> by
> > > >>>>>>>>>>>>>>>> #/6794
> > > >>>>>>>>>>>>>>>>>>> should have started as a proposal and round of
> > > >>>>> discussions
> > > >>>>>>>>>>>> about
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> major
> > > >>>>>>>>>>>>>>>>>>> changes designs not as 11K line of code.
> > > >>>>>>>>>>>>>>>>>>> I believe such openness will promote a lot of good
> > > >>>>> benefits
> > > >>>>>>>>>>>> such
> > > >>>>>>>>>>>>>> as:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> - ensures community health and growth.
> > > >>>>>>>>>>>>>>>>>>> - ensures everyone can participate not only the
> > > >> authors
> > > >>>>> and
> > > >>>>>>>>>>> his
> > > >>>>>>>>>>>>>>>>>> co-workers.
> > > >>>>>>>>>>>>>>>>>>> - ensures that the project is driven by the
> community
> > > >>> and
> > > >>>>>>>>>>> not a
> > > >>>>>>>>>>>>>> given
> > > >>>>>>>>>>>>>>>>>>> company or an individual.
> > > >>>>>>>>>>>>>>>>>>> - ensures that there is consensus (not saying 100%
> > > >>>>>>>>>>> agreement;)
> > > >>>>>>>>>>>>>>> however
> > > >>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>> means that all individuals will accept the current
> > > >>>>> progress
> > > >>>>>>>>>>> on
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> project
> > > >>>>>>>>>>>>>>>>>>> until some better proposal is put forth.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Personally such BIG offline PR makes me feel
> excluded
> > > >>> and
> > > >>>>>>>>>>>> doesn't
> > > >>>>>>>>>>>>>>> give
> > > >>>>>>>>>>>>>>>>>> me a
> > > >>>>>>>>>>>>>>>>>>> sense that i belong to  a community at all.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> To prevent such off list development i think as a
> > > >> Druid
> > > >>>>>>>>>>>> Community
> > > >>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>> need
> > > >>>>>>>>>>>>>>>>>>> to stick to the apache way “If it didn’t happen on
> > > >> the
> > > >>>>>>>>>>> mailing
> > > >>>>>>>>>>>>>> list,
> > > >>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>> didn’t happen.”
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> I would appreciate if some of the Apache mentor
> help
> > > >>> with
> > > >>>>>>>>>>> this.
> > > >>>>>>>>>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>
> > > >>>
> ---------------------------------------------------------------------
> > > >>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > > >>> dev-unsubscribe@druid.apache.org
> > > >>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > >>>>> dev-help@druid.apache.org
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>
> > > >>
> ---------------------------------------------------------------------
> > > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > > >> dev-unsubscribe@druid.apache.org
> > > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > >>> dev-help@druid.apache.org
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> --
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> B-Slim
> > > >>>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>
> > >
> _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>
> > ---------------------------------------------------------------------
> > > >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > >>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>
> ---------------------------------------------------------------------
> > > >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > >>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > For additional commands, e-mail: dev-help@druid.apache.org
> > >
> > >
> >
>

Re: Off list major development

[Fangjin Yang <fa...@imply.io>]

I agree with Gian, as an Apache committer, your responsibility is for the
betterment of the project. I agree it is in the best interest of the
project to stop thinking about what orgs people belong to. We are all a
part of the Apache software foundation, regardless of what our roles and
titles are outside of it.

On Fri, Jan 11, 2019 at 2:22 AM Roman Leventov <le...@gmail.com>
wrote:

> It's not that people from one org could abuse the project and push some
> change, but that they have similar perspective (bubble effect) and some
> important aspects of a large feature could escape their attention.
>
> I suggest it to be not a rigid rule, but a recommendation for authors of
> large proposals to try to attract reviewers from other orgs.
>
> On Fri, 11 Jan 2019 at 02:51, Julian Hyde <jh...@apache.org> wrote:
>
> > I agree with Gian.
> >
> > As an Apache committer, you only have one affiliation: you are working in
> > the best interests of the project.
> >
> > Obviously, in the real world there are other pressures. But we do our
> best
> > to compensate for them.
> >
> > Also, as a a community we try to design our process so as to avoid undue
> > influences. For instance, when I advocate for logging cases early, I am
> > trying to mitigate the effect of product managers and VPs of engineering,
> > who like to have their say in meeting rooms rather than on public mailing
> > lists. That’s just one example; if we see other influences at play, let’s
> > evolve our process to try to level the playing field.
> >
> > Julian
> >
> >
> > > On Jan 10, 2019, at 10:40 AM, Gian Merlino <gi...@apache.org> wrote:
> > >
> > >>> What do community members think about also making a requirement that
> > >>> "Design Review" PRs and proposals are reviewed by at least two people
> > > with
> > >>> different affiliation?
> > >> This seems pretty reasonable to me. I haven't found anything in Apache
> > >> voting procedure docs (https://www.apache.org/foundation/voting.html)
> > that
> > >> seems to explicitly forbid something like this yet at least.
> > >
> > > On the other hand, my understanding of the Apache Way would mean that
> > this
> > > kind of rule doesn't make sense. In particular from
> > > https://www.apache.org/foundation/how-it-works.html: "We firmly
> believe
> > in
> > > hats. Your role at the ASF is one assigned to you personally, and is
> > > bestowed on you by your peers. It is not tied to your job or current
> > > employer or company." That sentiment seems incompatible with making
> > > explicit rules about organizational diversity in voting. I have also
> > heard
> > > a few people say things like: people are supposed to represent
> > themselves,
> > > not their employers.
> > >
> > > Obviously, though, people's actions and opinions are influenced by
> their
> > > employer. IMO, a better way to approach a potential problem there is
> that
> > > if people from a particular organization end up behaving abusively,
> then
> > > PMC members from other organizations (or in extreme situations the
> Apache
> > > board itself) should tell them to knock it off. And then do more
> serious
> > > things if the inappropriate behavior continues. This kind
> > >
> > > The other thing you brought up, promoting more shared ownership and
> > > awareness, I am hopeful that separating proposals from PRs will help
> with
> > > that. One reason is that it takes much less time to understand a well
> > > written proposal than it takes to understand a PR. Code is very clear
> but
> > > it is also very verbose and takes a while to read and understand. So in
> > > theory changing how we operate, in this way, should promote more
> > > understanding of more people across the code base.
> > >
> > > On Thu, Jan 10, 2019 at 1:34 AM Clint Wylie <cl...@imply.io>
> > wrote:
> > >
> > >>>
> > >>> What do community members think about also making a requirement that
> > >>> "Design Review" PRs and proposals are reviewed by at least two people
> > >> with
> > >>> different affiliation?
> > >>
> > >>
> > >> This seems pretty reasonable to me. I haven't found anything in Apache
> > >> voting procedure docs (https://www.apache.org/foundation/voting.html)
> > that
> > >> seems to explicitly forbid something like this yet at least.
> > >>
> > >> On Wed, Jan 9, 2019 at 8:52 PM Roman Leventov <le...@gmail.com>
> > >> wrote:
> > >>
> > >>> What do community members think about also making a requirement that
> > >>> "Design Review" PRs and proposals are reviewed by at least two people
> > >> with
> > >>> different affiliation? IMO it's a good idea, because it ensures that
> > >>> different interests are taken into account. Also it pushes people to
> > >> engage
> > >>> with work done in other parts of Druid, improving shared code owning
> > and
> > >>> awareness.
> > >>>
> > >>> Or it's against the Apache Way because it's assumed that there are no
> > >>> company boundaries within the community?
> > >>>
> > >>> On Thu, 10 Jan 2019 at 11:45, Roman Leventov <le...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> I see two important reasons why it makes sense to file an issue and
> > >>>> probably announce it in the mailing list, before writing a lot of
> > code,
> > >>>> despite not having a clear picture of what it will be and any
> > >> performance
> > >>>> data:
> > >>>> 1) somebody could already work on this problem privately in
> parallel,
> > >> it
> > >>>> allows to avoid clash in people's works
> > >>>> 2) some people could quickly think about the problem field and share
> > >>>> high-level ideas that could wildly change the direction in which the
> > >>> author
> > >>>> (the person who is going to write code) will move in his work from
> > >> early
> > >>> on.
> > >>>>
> > >>>> Jihoon in
> > >>>>
> > >>>
> > >>
> >
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> > >>> and
> > >>>> later Gian in this thread suggested that _every_ piece of work that
> > >>> should
> > >>>> be labelled as "Design Review" according to the current rules should
> > be
> > >>>> accompanied by an issue. I don't agree with this, there are some PRs
> > as
> > >>>> small as a few dozens of lines of code, that add some configuration
> > >>>> parameter and therefore should be labelled "Design Review". I don't
> > >>> thing a
> > >>>> separate proposal issue is needed for them, and even for a little
> > >> larger
> > >>>> PRs too.
> > >>>>
> > >>>> For the same reason, I also don't see the point of renaming "Design
> > >>>> Review" into "Proposal", as well as separating "Design Review" into
> > >>>> "Proposal" and something like "API Review". I think a single "Design
> > >>>> Review" tag handles it well.
> > >>>>
> > >>>> Gian mentioned an idea that PRs that follow a "Design Review"
> proposal
> > >>>> issue shouldn't be "Design Review" themselves. I don't agree with
> > >> this, I
> > >>>> think that actual code and performance data are important inputs
> that
> > >>>> should be re-evaluated at least by two people. I even think that
> it's
> > >>> very
> > >>>> desirable that at least two people read _each line of production
> code_
> > >> in
> > >>>> large PRs, although it's not what was done historically in Druid,
> > >> because
> > >>>> large bodies of newly added code, with whole new classes and
> > subsystems
> > >>>> added, are also coincidentally tested worse than already existing
> > >> classes
> > >>>> and subsystems, including in production. It seems to me that those
> > huge
> > >>>> code influxes is a major source of bugs, that could later take years
> > to
> > >>>> squeeze out from the codebase.
> > >>>>
> > >>>> On Wed, 9 Jan 2019 at 08:24, Clint Wylie <cl...@imply.io>
> > wrote:
> > >>>>
> > >>>>> Apologies for the delayed response.
> > >>>>>
> > >>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > >>> slim.bouguerra@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> I am wondering here what is the case where code first is better?
> > >>>>>>
> > >>>>>
> > >>>>> I don't think it's wrong to share ideas as early as possible, and
> > >> after
> > >>>>> this discussion I think I am in favor of it too. I just meant that
> I
> > >>> don't
> > >>>>> think it's always necessarily the most productive discussion until
> > >> code
> > >>>>> exists sometimes, with the types of thing I am thinking of are
> almost
> > >>>>> entirely limited to cases where things might sound good to anyone
> on
> > >>> paper
> > >>>>> but in reality need a large amount of experiments conducted and
> > >>>>> observations collected to determine that something is actually
> worth
> > >>>>> doing,
> > >>>>> which I imagine is mainly things like reworking internals for
> > >>> performance
> > >>>>> improvements.
> > >>>>>
> > >>>>> In the case of my combined proposal PR, I needed to prove that the
> > >>> thing I
> > >>>>> was working on was a good idea... and it wasn't directly. But I
> came
> > >> up
> > >>>>> with another idea during the course of experiment turned into
> > >> something
> > >>>>> compelling, so an initial proposal would have looked quite a lot
> > >>> different
> > >>>>> than what I ended up with. Once I had proven to myself that it was
> a
> > >>> good
> > >>>>> idea, then I was comfortable sharing with the wider community. I'm
> > not
> > >>>>> certain how this would play out in an always proposal first model,
> > >> maybe
> > >>>>> the first proposal exists, I personally reject it after updating
> with
> > >>>>> experiment results show it's a bad idea, continue experimenting and
> > >>> raise
> > >>>>> a
> > >>>>> new one after the experiments start looking promising?
> > >>>>>
> > >>>>>
> > >>>>>> Let's not be naive this is very rare that a contributor will
> accept
> > >>> that
> > >>>>>> his work is to be thrown, usually devs takes coding as personal
> > >>> creation
> > >>>>>> and they get attached to it.
> > >>>>>>
> > >>>>>
> > >>>>> I agree, just because a handful of the committers have this
> attitude,
> > >> it
> > >>>>> isn't fair to expect the wider community to also, that's why I am
> in
> > >>> favor
> > >>>>> of formalizing the process.
> > >>>>>
> > >>>>> Can you please explain what is overbearing ? what can be changed to
> > >> make
> > >>>>> it
> > >>>>>> easy ?
> > >>>>>> Most of the points are kind of the actual questions that you want
> to
> > >>>>>> address before hand anyway isn't ?
> > >>>>>>
> > >>>>>
> > >>>>> Sorry for the confusion, I said it's "not overbearing", I think
> it's
> > >>> fine.
> > >>>>>
> > >>>>> What are the complaints ?
> > >>>>>
> > >>>>>
> > >>>>> Is this and other previous threads not a complaint about opening a
> > >> large
> > >>>>> PR
> > >>>>> without a proposal? :) I just mean that formalizing the process,
> even
> > >>> if a
> > >>>>> proposal has a reference PR opened with it near concurrently, could
> > >>>>> prevent
> > >>>>> these discussions from happening in the future because ground rules
> > >> have
> > >>>>> been set and we are all on the same page I guess.
> > >>>>>
> > >>>>> I think we should also probably consider renaming the "design
> review"
> > >>>>> label
> > >>>>> to "proposal" or something to make it more clear that a PR is
> > >> associated
> > >>>>> with a proposal. It might also be worth considering using github
> > >>> projects
> > >>>>> for particularly large things that involve multiple follow up PRs,
> > >> but I
> > >>>>> haven't used them in much depth to know if they add anything.
> > >>>>>
> > >>>>> It seems like we are all converging on agreement to do a github
> issue
> > >>>>> proposal for larger changes (I would vote for announcing them on
> the
> > >> dev
> > >>>>> list too for more visibility), so that design review is separated
> > from
> > >>>>> code
> > >>>>> review. I guess my main concern was not wanting to discourage
> > >>>>> experimentation by walling it off behind mandatory discussions, but
> > >> the
> > >>>>> more I think about it,  it doesn't technically change much about
> this
> > >>>>> process, it just requires a more formal proposal to accompany any
> > >>>>> experiments that are shared as a PR.
> > >>>>>
> > >>>>>
> > >>>>> On Tue, Jan 8, 2019 at 12:28 PM Gian Merlino <gi...@apache.org>
> > wrote:
> > >>>>>
> > >>>>>> I think for us, choosing to use GitHub issues as discussion
> threads
> > >>> for
> > >>>>>> potential 'major' contributions would be a good idea, especially
> if
> > >> we
> > >>>>>> encourage people to start them before PRs show up. Definitely
> agree
> > >>> that
> > >>>>>> all contributors should go through the same process -- I couldn't
> > >> see
> > >>> it
> > >>>>>> working well any other way.
> > >>>>>>
> > >>>>>> On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <jh...@apache.org>
> > >> wrote:
> > >>>>>>
> > >>>>>>> Statically, yes, GitHub PRs are the same as GitHub cases. But
> > >>>>>> dynamically,
> > >>>>>>> they are different, because you can only log a PR when you have
> > >>>>> finished
> > >>>>>>> work.
> > >>>>>>>
> > >>>>>>> A lot of other Apache projects use JIRA, so there is a clear
> > >>>>> distinction
> > >>>>>>> between cases and contributions. JIRA cases, especially when
> > >> logged
> > >>>>> early
> > >>>>>>> in the lifecycle of a contribution, become long-running
> > >> conversation
> > >>>>>>> threads with a lot of community participation. If the Druid chose
> > >> to
> > >>>>> do
> > >>>>>> so,
> > >>>>>>> GitHub cases could be the same.
> > >>>>>>>
> > >>>>>>> Be careful that you do not treat “potential contributors” (by
> > >> which
> > >>> I
> > >>>>>>> presume you mean non-committers) differently from committers and
> > >> PMC
> > >>>>>>> members. Anyone starting a major piece of work should follow the
> > >>> same
> > >>>>>>> process. (Experienced committers probably have a somewhat better
> > >>> idea
> > >>>>>> what
> > >>>>>>> work will turn out to be “major”, so they get a little more
> > >> leeway.)
> > >>>>>>>
> > >>>>>>> Julian
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> On Jan 7, 2019, at 12:10 PM, Gian Merlino <gi...@apache.org>
> > >>> wrote:
> > >>>>>>>>
> > >>>>>>>> I don't think there's a need to raise issues for every change: a
> > >>>>> small
> > >>>>>>> bug
> > >>>>>>>> fix or doc fix should just go straight to PR. (GitHub PRs show
> > >> up
> > >>> as
> > >>>>>>> issues
> > >>>>>>>> in the issue-search UI/API, so it's not like this means the
> > >> patch
> > >>>>> has
> > >>>>>> no
> > >>>>>>>> corresponding issue -- in a sense the PR _is_ the issue.)
> > >>>>>>>>
> > >>>>>>>> I do think it makes sense to encourage potential contributors to
> > >>>>> write
> > >>>>>> to
> > >>>>>>>> the dev list or raise an issue if they aren't sure if something
> > >>>>> would
> > >>>>>>> need
> > >>>>>>>> to go through a more heavy weight process.
> > >>>>>>>>
> > >>>>>>>> Fwiw we do have a set of 'design review' criteria already (we
> > >> had
> > >>> a
> > >>>>>>>> discussion about this a couple years ago) at:
> > >>>>>>>> http://druid.io/community/#getting-your-changes-accepted. So we
> > >>>>>>> wouldn't be
> > >>>>>>>> starting from zero on defining that. We set it up back when we
> > >>> were
> > >>>>>>> trying
> > >>>>>>>> to _streamline_ our process -- we used to require two non-author
> > >>> +1s
> > >>>>>> for
> > >>>>>>>> _every_ change, even minor ones. The introduction of design
> > >> review
> > >>>>>>> criteria
> > >>>>>>>> was meant to classify which PRs need that level of review and
> > >>> which
> > >>>>>> ones
> > >>>>>>>> are minor and can be merged with less review. I do think it
> > >> helped
> > >>>>> with
> > >>>>>>>> getting minor PRs merged more quickly. The list of criteria is,
> > >>>>>>>>
> > >>>>>>>> - Major architectural changes or API changes
> > >>>>>>>> - HTTP requests and responses (e. g. a new HTTP endpoint)
> > >>>>>>>> - Interfaces for extensions
> > >>>>>>>> - Server configuration (e. g. altering the behavior of a config
> > >>>>>> property)
> > >>>>>>>> - Emitted metrics
> > >>>>>>>> - Other major changes, judged by the discretion of Druid
> > >>> committers
> > >>>>>>>>
> > >>>>>>>> Some of it is subjective, but it has been in place for a while,
> > >> so
> > >>>>> it's
> > >>>>>>> at
> > >>>>>>>> least something we are relatively familiar with.
> > >>>>>>>>
> > >>>>>>>> On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <jh...@apache.org>
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Small contributions don’t need any design review, whereas large
> > >>>>>>>>> contributions need significant review. I don’t think we should
> > >>>>> require
> > >>>>>>> an
> > >>>>>>>>> additional step for those (many) small contributions. But who
> > >>>>> decides
> > >>>>>>>>> whether a contribution fits into the small or large category?
> > >>>>>>>>>
> > >>>>>>>>> I think the solution is for authors to log a case (or send an
> > >>>>> email to
> > >>>>>>>>> dev) before they start work on any contribution. Then
> > >> committers
> > >>>>> can
> > >>>>>>>>> request a more heavy-weight process if they think it is needed.
> > >>>>>>>>>
> > >>>>>>>>> Julian
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <gi...@apache.org>
> > >>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> It sounds like splitting design from code review is a common
> > >>> theme
> > >>>>>> in a
> > >>>>>>>>> few
> > >>>>>>>>>> of the posts here. How does everyone feel about making a point
> > >>> of
> > >>>>>>>>>> encouraging design reviews to be done as issues, separate from
> > >>> the
> > >>>>>> pull
> > >>>>>>>>>> request, with the expectations that (1) the design review
> > >> issue
> > >>>>>>>>>> ("proposal") should generally appear somewhat _before_ the
> > >> pull
> > >>>>>>> request;
> > >>>>>>>>>> (2) pull requests should _not_ have design review happen on
> > >>> them,
> > >>>>>>> meaning
> > >>>>>>>>>> there should no longer be PRs with design review tags, and we
> > >>>>> should
> > >>>>>>> move
> > >>>>>>>>>> the design review approval process to the issue rather than
> > >> the
> > >>>>> PR.
> > >>>>>>>>>>
> > >>>>>>>>>> For (1), even if we encourage design review discussions to
> > >> start
> > >>>>>>> before a
> > >>>>>>>>>> pull request appears, I don't see an issue with them running
> > >>>>>>> concurrently
> > >>>>>>>>>> for a while at some point.
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <
> > >> jonwei@apache.org>
> > >>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Thanks for raising these concerns!
> > >>>>>>>>>>>
> > >>>>>>>>>>> My initial thoughts:
> > >>>>>>>>>>> - I agree that separation of design review and code-level
> > >>> review
> > >>>>> for
> > >>>>>>>>> major
> > >>>>>>>>>>> changes would be more efficient
> > >>>>>>>>>>>
> > >>>>>>>>>>> - I agree that a clear, more formalized process for handling
> > >>>>> major
> > >>>>>>>>> changes
> > >>>>>>>>>>> would be helpful for contributors:
> > >>>>>>>>>>> - Define what is considered a major change
> > >>>>>>>>>>> - Define a standard proposal structure, KIP-style proposal
> > >>> format
> > >>>>>>>>> sounds
> > >>>>>>>>>>> good to me
> > >>>>>>>>>>>
> > >>>>>>>>>>> - I think it's too rigid to have a policy of "no code at all
> > >>> with
> > >>>>>> the
> > >>>>>>>>>>> initial proposal"
> > >>>>>>>>>>> - Code samples can be useful references for understanding
> > >>> aspects
> > >>>>>> of a
> > >>>>>>>>>>> design
> > >>>>>>>>>>> - In some cases it's necessary to run experiments to fully
> > >>>>>> understand
> > >>>>>>> a
> > >>>>>>>>>>> problem and determine an appropriate design, or to determine
> > >>>>> whether
> > >>>>>>>>>>> something is even worth doing before committing to the work
> > >> of
> > >>>>>>> fleshing
> > >>>>>>>>> out
> > >>>>>>>>>>> a proposal, prototype code is a natural outcome of that and
> > >> I'm
> > >>>>> not
> > >>>>>>>>> against
> > >>>>>>>>>>> someone providing such code for reference
> > >>>>>>>>>>> - I tend to view design/code as things that are often
> > >> developed
> > >>>>>>>>>>> simultaneously in an intertwined way
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Let's not be naive this is very rare that a contributor will
> > >>>>> accept
> > >>>>>>>>> that
> > >>>>>>>>>>> his work is to be thrown, usually devs takes coding as
> > >> personal
> > >>>>>>> creation
> > >>>>>>>>>>> and they get attached to it.
> > >>>>>>>>>>>
> > >>>>>>>>>>> If we have a clear review process that emphasizes the need
> > >> for
> > >>>>> early
> > >>>>>>>>>>> consensus building, with separate design and code review,
> > >> then
> > >>> I
> > >>>>>> feel
> > >>>>>>>>> we've
> > >>>>>>>>>>> done enough and don't need a hard rule against having some
> > >> code
> > >>>>>> linked
> > >>>>>>>>> with
> > >>>>>>>>>>> the initial proposal. If a potential contributor then still
> > >>>>> wants to
> > >>>>>>> go
> > >>>>>>>>>>> ahead and write a lot of code that may be rejected or change
> > >>>>>>>>> significantly,
> > >>>>>>>>>>> the risks were made clear.
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Once code is written hard to think abstract.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I can see the validity of the concern, but I personally don't
> > >>>>> see it
> > >>>>>>> as
> > >>>>>>>>> a
> > >>>>>>>>>>> huge risk. My impression from the Druid PR reviews I've seen
> > >> is
> > >>>>> that
> > >>>>>>> our
> > >>>>>>>>>>> reviewers are able to keep abstract design vs. implementation
> > >>>>>> details
> > >>>>>>>>>>> separate and consider alternate designs when reviewing.
> > >>>>>>>>>>>
> > >>>>>>>>>>> To summarize I think it's probably enough to have a policy
> > >>> along
> > >>>>> the
> > >>>>>>>>> lines
> > >>>>>>>>>>> of:
> > >>>>>>>>>>> - Create more formalized guidelines for proposals and what
> > >>>>> changes
> > >>>>>>>>> require
> > >>>>>>>>>>> proposals
> > >>>>>>>>>>> - Separate design and code review for major changes, with
> > >>> design
> > >>>>>>> review
> > >>>>>>>>>>> first, code-level review after reaching consensus on the
> > >>> design.
> > >>>>>>>>>>> - Code before the design review is completed is just for
> > >>>>> reference,
> > >>>>>>> not
> > >>>>>>>>>>> regarded as a candidate for review/merging.
> > >>>>>>>>>>>
> > >>>>>>>>>>> - Jon
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > >>>>>>>>> slim.bouguerra@gmail.com>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <
> > >>>>> clint.wylie@imply.io>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> I am definitely biased in this matter as an owner of
> > >> another
> > >>>>> large
> > >>>>>>> PR
> > >>>>>>>>>>>> that
> > >>>>>>>>>>>>> wasn't preceded by a direct proposal or dev list
> > >> discussion,
> > >>>>> and
> > >>>>>> in
> > >>>>>>>>>>>> general
> > >>>>>>>>>>>>> I agree that proposal first is usually better, but I think
> > >> in
> > >>>>> some
> > >>>>>>>>>>> rarer
> > >>>>>>>>>>>>> cases approaching a problem code first *is* the most
> > >>>>> appropriate
> > >>>>>> way
> > >>>>>>>>> to
> > >>>>>>>>>>>>> have a discussion.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I am wondering here what is the case where code first is
> > >>> better?
> > >>>>>>>>>>>> In general when you are writing code you have an idea about
> > >>> what
> > >>>>>> you
> > >>>>>>>>> want
> > >>>>>>>>>>>> to change, why you want to change and why you want to change
> > >>> it.
> > >>>>>>>>>>>> I do not see what is wrong with sharing this primitive ideas
> > >>> and
> > >>>>>>>>> thoughts
> > >>>>>>>>>>>> as an abstract proposal (at least to avoid overlapping)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I see nothing wrong with it so long as the author
> > >>>>>>>>>>>>> accepts that the PR is treated as a combined proposal and
> > >>>>> proof of
> > >>>>>>>>>>>> concept,
> > >>>>>>>>>>>>> and fair game to be radically changed via discussion or
> > >> even
> > >>>>>>> rejected,
> > >>>>>>>>>>>>> which sounds like Gian's attitude on the matter and is mine
> > >>> as
> > >>>>>> well
> > >>>>>>>>>>> with
> > >>>>>>>>>>>> my
> > >>>>>>>>>>>>> compression stuff.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Let's not be naive this is very rare that a contributor will
> > >>>>> accept
> > >>>>>>>>> that
> > >>>>>>>>>>>> his work is to be thrown, usually devs takes coding as
> > >>> personal
> > >>>>>>>>> creation
> > >>>>>>>>>>>> and they get attached to it.
> > >>>>>>>>>>>> To my point you can take a look on some old issue in the
> > >> Druid
> > >>>>>> forum
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>
> >
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> > >>>>>>>>>>>> and am sure other communities have similar problems.
> > >>>>>>>>>>>> So leaving the door open to some side cases is not a good
> > >> idea
> > >>>>> in
> > >>>>>> my
> > >>>>>>>>>>>> opinion and will lead to similar issue in the future.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This seems to me especially likely to happen in cases
> > >>>>>>>>>>>>> where an approach still needs proven to be a viable idea
> > >> *to
> > >>>>> the
> > >>>>>>>>>>> author*,
> > >>>>>>>>>>>>> so that a much more productive discussion can be had in the
> > >>>>> first
> > >>>>>>>>>>> place.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I think there is a trade off, I don't think we want to
> > >>>>> discourage
> > >>>>>>>>>>>>> experimentation by walling it off behind mandatory
> > >>> discussions
> > >>>>>>> before
> > >>>>>>>>>>> it
> > >>>>>>>>>>>>> can even start, but I do think formalizing the process for
> > >>>>> large
> > >>>>>>>>>>> changes
> > >>>>>>>>>>>> is
> > >>>>>>>>>>>>> a good thing, especially since we probably can't expect the
> > >>>>> wider
> > >>>>>>>>>>>> community
> > >>>>>>>>>>>>> to have the same attitude towards a large PR getting
> > >>> discarded
> > >>>>> as
> > >>>>>> a
> > >>>>>>>>>>>>> committer might. I think the Kafka approach is reasonable,
> > >> a
> > >>>>> bit
> > >>>>>>> more
> > >>>>>>>>>>>>> formal than our design review process but not overbearing.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Can you please explain what is overbearing ? what can be
> > >>>>> changed to
> > >>>>>>>>> make
> > >>>>>>>>>>> it
> > >>>>>>>>>>>> easy ?
> > >>>>>>>>>>>> Most of the points are kind of the actual questions that you
> > >>>>> want
> > >>>>>> to
> > >>>>>>>>>>>> address before hand anyway isn't ?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Going code first
> > >>>>>>>>>>>>> should be in general discouraged, but when it does happen,
> > >> it
> > >>>>>> seems
> > >>>>>>>>>>> like
> > >>>>>>>>>>>>> opening DIP/an issue/starting a mailing list thread or
> > >>>>> whatever we
> > >>>>>>> go
> > >>>>>>>>>>>> with
> > >>>>>>>>>>>>> to have a more high level design discussion alongside the
> > >>>>>> reference
> > >>>>>>> PR
> > >>>>>>>>>>>>> could alleviate some of these complaints?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> What are the complaints ?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> +1 for "DIP" heh, I think making
> > >>>>>>>>>>>>> them in the form of github issues is probably appropriate,
> > >>>>> with a
> > >>>>>>> dev
> > >>>>>>>>>>>> list
> > >>>>>>>>>>>>> thread to announce them perhaps?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I think  github issue with [Proposal] header like
> > >>>>>>>>>>>> https://github.com/apache/incubator-druid/issues/4349 is
> > >> good
> > >>>>> to
> > >>>>>> me,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <
> > >>>>> bslim@apache.org>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks everyone for interacting with this thread.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> The fact that i, Roman, Jihoon  and others in the past (FJ
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>
> >
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ
> > >>>>> )
> > >>>>>>>>>>>>>> raised this point indicates that PRs without a proposal
> > >> are
> > >>>>>> indeed
> > >>>>>>> an
> > >>>>>>>>>>>>> issue
> > >>>>>>>>>>>>>> and we need to solve it.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Something Similar to KIP maybe called DIPs is fine with
> > >> me.
> > >>>>>>>>>>>>>> What i strive to see is the following:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> [Step 1] formalize what is the kind of work that needs a
> > >>>>> formal
> > >>>>>>>>>>>>> Proposal, I
> > >>>>>>>>>>>>>> think Roman and Jihoon has already covered that pretty
> > >> well.
> > >>>>> am
> > >>>>>> +1
> > >>>>>>> on
> > >>>>>>>>>>>>> that.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>
> >
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I am strongly in favor of the separation of Proposal
> > >> Review
> > >>>>> and
> > >>>>>>>>>>> (later)
> > >>>>>>>>>>>>>> Code review PRs. My  main reasons:
> > >>>>>>>>>>>>>> Most importantly code reviewing will introduce lot of
> > >> noise
> > >>>>> and
> > >>>>>>> will
> > >>>>>>>>>>>>>> ultimately make  the GitHub page unreadable.
> > >>>>>>>>>>>>>> Avoid overlapping of work.
> > >>>>>>>>>>>>>> Once code is written hard to think abstract.
> > >>>>>>>>>>>>>> Separate page for Design review later can always be used
> > >> it
> > >>>>> as a
> > >>>>>>>>>>> Design
> > >>>>>>>>>>>>>> document that is readable and code free-ish.
> > >>>>>>>>>>>>>> As i said the goal of this first round is to see if the
> > >>>>> community
> > >>>>>>>>>>> agree
> > >>>>>>>>>>>>>> about such change, then make the process of design more
> > >>>>> inclusive
> > >>>>>>>>>>> thus
> > >>>>>>>>>>>>>> other contributors can submit a counter proposals.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> [Step 2] IF everybody agree about that point Step 2 is to
> > >>>>> define
> > >>>>>>>>>>> which
> > >>>>>>>>>>>>>> medium is used to Publish a primitive form of a CODE FREE
> > >>>>>> Abstract
> > >>>>>>>>>>>>> Proposal
> > >>>>>>>>>>>>>> containing at least the following bullet points.
> > >>>>>>>>>>>>>> - The problem description and motivation
> > >>>>>>>>>>>>>> - Overview of the proposed change
> > >>>>>>>>>>>>>> - Operational impact (compatibility/ plans to upgrades)
> > >>> public
> > >>>>>> API
> > >>>>>>>>>>>>> changes,
> > >>>>>>>>>>>>>> configuration changes, algorithm, and so on
> > >>>>>>>>>>>>>> - Expected benefits and drawbacks
> > >>>>>>>>>>>>>> - Rationale and alternatives
> > >>>>>>>>>>>>>> - Estimate Time to Deliver if possible.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> The way i think this can be is a Github issue where member
> > >>> of
> > >>>>> the
> > >>>>>>>>>>>>> community
> > >>>>>>>>>>>>>> will interact via comments and the author will be updating
> > >>> the
> > >>>>>>>>>>>>> description
> > >>>>>>>>>>>>>> in the light of comments provided by the community.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> During and near the end of the design discussions the
> > >>> author/s
> > >>>>>> can
> > >>>>>>>>>>>> start
> > >>>>>>>>>>>>>> writing POCs to help guide the review process this
> > >> naturally
> > >>>>> will
> > >>>>>>> be
> > >>>>>>>>>>> a
> > >>>>>>>>>>>>> Pull
> > >>>>>>>>>>>>>> request with actual code.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> *Now the most important thing is that we need to agree
> > >> that
> > >>>>> any
> > >>>>>>> work
> > >>>>>>>>>>>> that
> > >>>>>>>>>>>>>> does not align with this formal process will be ignored
> > >> and
> > >>>>> the
> > >>>>>>>>>>> author
> > >>>>>>>>>>>>> will
> > >>>>>>>>>>>>>> be asked to start with a DIP*
> > >>>>>>>>>>>>>> *That is what i meant with  “If it didn’t happen on the
> > >>>>> mailing
> > >>>>>>> list,
> > >>>>>>>>>>>> it
> > >>>>>>>>>>>>>> didn’t happen.”*
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks and happy coding!
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <
> > >>> gian@apache.org>
> > >>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> One of the advantages I see with a more formal process is
> > >>>>> (like
> > >>>>>>>>>>> Kafka
> > >>>>>>>>>>>>>> KIPs)
> > >>>>>>>>>>>>>>> is that it levels the playing field a bit and sets some
> > >>>>> ground
> > >>>>>>>>>>> rules
> > >>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>> working together. In a way it can help encourage
> > >>>>> contributions
> > >>>>>> by
> > >>>>>>>>>>>>> making
> > >>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>> clear what is expected of potential contributors.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> We have a design review process today that is not as
> > >> formal
> > >>>>> as
> > >>>>>>>>>>> KIPs,
> > >>>>>>>>>>>>> but
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>> somewhat heavier than the one you describe. Maybe we
> > >> could
> > >>>>> tweak
> > >>>>>>>>>>> our
> > >>>>>>>>>>>>>>> current one by starting to do design reviews separately
> > >>> from
> > >>>>>> PRs.
> > >>>>>>>>>>>> i.e.,
> > >>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>> anything that meets our 'design review' criteria, do that
> > >>> on
> > >>>>> the
> > >>>>>>>>>>> dev
> > >>>>>>>>>>>>> list
> > >>>>>>>>>>>>>>> or in a separate issue, and keep the PR focused on
> > >>> code-level
> > >>>>>>>>>>> stuff.
> > >>>>>>>>>>>>> That
> > >>>>>>>>>>>>>>> way we don't end up trying to do both at once. And it
> > >> makes
> > >>>>> it
> > >>>>>>>>>>> easier
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> start talking about design before the code is ready,
> > >> which
> > >>>>> would
> > >>>>>>> be
> > >>>>>>>>>>>>>> better.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <
> > >>> jhyde@apache.org
> > >>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> It’s really hard to say no to a contribution when
> > >> someone
> > >>>>> has
> > >>>>>> put
> > >>>>>>>>>>>> in
> > >>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>> significant amount of work.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The following approach is simple and works really well:
> > >>>>> Before
> > >>>>>>>>>>> you
> > >>>>>>>>>>>>>> start
> > >>>>>>>>>>>>>>>> work, log a case, describing the problem. When you have
> > >>> some
> > >>>>>>>>>>> ideas
> > >>>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>> design, add those to the case. When you have a code
> > >>> branch,
> > >>>>> add
> > >>>>>>>>>>> its
> > >>>>>>>>>>>>> URL
> > >>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>> the case. And so forth. At any point in the proceedings,
> > >>>>> people
> > >>>>>>>>>>> can
> > >>>>>>>>>>>>>> chime
> > >>>>>>>>>>>>>>>> in with their opinions.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> In my opinion, a formal “design review” process is not
> > >>>>>> necessary.
> > >>>>>>>>>>>>> Just
> > >>>>>>>>>>>>>>>> build consensus iteratively, by starting the
> > >> conversation
> > >>>>> early
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> process.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Julian
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <
> > >>> gian@apache.org
> > >>>>>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> In this particular case: please consider the PR as a
> > >>>>> proposal.
> > >>>>>>>>>>>>> Don't
> > >>>>>>>>>>>>>>> feel
> > >>>>>>>>>>>>>>>>> like just because there is code there that takes a
> > >>> certain
> > >>>>>>>>>>>>> approach,
> > >>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>> the approach is somehow sacred. I had to implement
> > >>>>> something
> > >>>>>> to
> > >>>>>>>>>>>>>>>> crystallize
> > >>>>>>>>>>>>>>>>> my own thinking about how the problem could be
> > >>> approached.
> > >>>>> I
> > >>>>>>>>>>>> won't
> > >>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>> disappointed if, as a community, we decide a different
> > >>>>>>>>>>> direction
> > >>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>> better
> > >>>>>>>>>>>>>>>>> and the code all gets thrown away. That's one of the
> > >>>>> reasons
> > >>>>>>>>>>>> that I
> > >>>>>>>>>>>>>>>> removed
> > >>>>>>>>>>>>>>>>> the 0.14.0 milestone that was added to the patch. (I
> > >>> don't
> > >>>>>> want
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>> rush
> > >>>>>>>>>>>>>>>> it,
> > >>>>>>>>>>>>>>>>> nor do I think that's a good idea.)
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> In general: Sounds like we could do with some more
> > >>>>>>>>>>> formalization
> > >>>>>>>>>>>>>> around
> > >>>>>>>>>>>>>>>>> what a proposal looks like, which sorts of changes need
> > >>>>> one,
> > >>>>>>>>>>> and
> > >>>>>>>>>>>>> when
> > >>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>> the dev cycle it is appropriate. FWIW I think Kafka's
> > >>>>> process
> > >>>>>>>>>>> is
> > >>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>> less fine, and would be okay with adopting it for Druid
> > >>> if
> > >>>>>>>>>>> people
> > >>>>>>>>>>>>>> like
> > >>>>>>>>>>>>>>>> it.
> > >>>>>>>>>>>>>>>>> Right now our standards for what requires a "design
> > >>> review"
> > >>>>>> are
> > >>>>>>>>>>>>> very
> > >>>>>>>>>>>>>>>>> similar to the Kafka community standards for what
> > >>> requires
> > >>>>> a
> > >>>>>>>>>>> KIP,
> > >>>>>>>>>>>>> so
> > >>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>> have some familiarity with those concepts. However we
> > >>> don't
> > >>>>>>>>>>>>> separate
> > >>>>>>>>>>>>>> PR
> > >>>>>>>>>>>>>>>>> review and proposal discussion as strictly as they do,
> > >>>>> which
> > >>>>>>>>>>>> seems
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>> the foundation for the feeling of exclusion that is
> > >> being
> > >>>>> felt
> > >>>>>>>>>>>>> here.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Separately: I just redid the description on
> > >>>>>>>>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794 to
> > >>> be
> > >>>>>> more
> > >>>>>>>>>>>>>>>> proposal-y.
> > >>>>>>>>>>>>>>>>> I followed the KIP style:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >>>>>>>>>>>>>>>> .
> > >>>>>>>>>>>>>>>>> Please refresh the page and see if it looks more
> > >> useful.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Gian
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <
> > >>>>> jhyde@apache.org
> > >>>>>>>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Slim,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I agree with your points that offline development is
> > >> bad
> > >>>>> for
> > >>>>>>>>>>>>>>> community.
> > >>>>>>>>>>>>>>>>>> But I don’t think you need much mentor help. You have
> > >>>>> raised
> > >>>>>>>>>>>> valid
> > >>>>>>>>>>>>>>>> issues
> > >>>>>>>>>>>>>>>>>> and the Druid community needs to decide what its
> > >>>>> development
> > >>>>>>>>>>>>>> practices
> > >>>>>>>>>>>>>>>>>> should be.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Julian
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
> > >>>>>>>>>>> bslim@apache.org>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hello everyone and hope you all have very good
> > >>> holidays.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> First, this email is not directed on the author or
> > >> the
> > >>> PR
> > >>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794
> > >>> it
> > >>>>>>>>>>> self,
> > >>>>>>>>>>>>> but
> > >>>>>>>>>>>>>> i
> > >>>>>>>>>>>>>>>> see
> > >>>>>>>>>>>>>>>>>>> this PR as a perfect example.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> One of the foundation of Apache Way or what i would
> > >>>>> simply
> > >>>>>>>>>>> call
> > >>>>>>>>>>>>>> open
> > >>>>>>>>>>>>>>>>>> source
> > >>>>>>>>>>>>>>>>>>> community driven development is that "Technical
> > >>> decisions
> > >>>>>> are
> > >>>>>>>>>>>>>>>> discussed,
> > >>>>>>>>>>>>>>>>>>> decided, and archived publicly.
> > >>>>>>>>>>>>>>>>>>> developpement"
> > >>>>>>>>>>>>>>>>>>> Which means that big technical  changes such as the
> > >> one
> > >>>>>>>>>>> brought
> > >>>>>>>>>>>>> by
> > >>>>>>>>>>>>>>>> #/6794
> > >>>>>>>>>>>>>>>>>>> should have started as a proposal and round of
> > >>>>> discussions
> > >>>>>>>>>>>> about
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> major
> > >>>>>>>>>>>>>>>>>>> changes designs not as 11K line of code.
> > >>>>>>>>>>>>>>>>>>> I believe such openness will promote a lot of good
> > >>>>> benefits
> > >>>>>>>>>>>> such
> > >>>>>>>>>>>>>> as:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> - ensures community health and growth.
> > >>>>>>>>>>>>>>>>>>> - ensures everyone can participate not only the
> > >> authors
> > >>>>> and
> > >>>>>>>>>>> his
> > >>>>>>>>>>>>>>>>>> co-workers.
> > >>>>>>>>>>>>>>>>>>> - ensures that the project is driven by the community
> > >>> and
> > >>>>>>>>>>> not a
> > >>>>>>>>>>>>>> given
> > >>>>>>>>>>>>>>>>>>> company or an individual.
> > >>>>>>>>>>>>>>>>>>> - ensures that there is consensus (not saying 100%
> > >>>>>>>>>>> agreement;)
> > >>>>>>>>>>>>>>> however
> > >>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>> means that all individuals will accept the current
> > >>>>> progress
> > >>>>>>>>>>> on
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> project
> > >>>>>>>>>>>>>>>>>>> until some better proposal is put forth.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Personally such BIG offline PR makes me feel excluded
> > >>> and
> > >>>>>>>>>>>> doesn't
> > >>>>>>>>>>>>>>> give
> > >>>>>>>>>>>>>>>>>> me a
> > >>>>>>>>>>>>>>>>>>> sense that i belong to  a community at all.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> To prevent such off list development i think as a
> > >> Druid
> > >>>>>>>>>>>> Community
> > >>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>> need
> > >>>>>>>>>>>>>>>>>>> to stick to the apache way “If it didn’t happen on
> > >> the
> > >>>>>>>>>>> mailing
> > >>>>>>>>>>>>>> list,
> > >>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>> didn’t happen.”
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> I would appreciate if some of the Apache mentor help
> > >>> with
> > >>>>>>>>>>> this.
> > >>>>>>>>>>>>>>>>>>> Thanks
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>
> > >>> ---------------------------------------------------------------------
> > >>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > >>> dev-unsubscribe@druid.apache.org
> > >>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> > >>>>> dev-help@druid.apache.org
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>
> > >> ---------------------------------------------------------------------
> > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > >> dev-unsubscribe@druid.apache.org
> > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > >>> dev-help@druid.apache.org
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> B-Slim
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>
> > _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>
> ---------------------------------------------------------------------
> > >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > >>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>> ---------------------------------------------------------------------
> > >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > >>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > For additional commands, e-mail: dev-help@druid.apache.org
> >
> >
>

Re: Off list major development

[Roman Leventov <le...@gmail.com>]

It's not that people from one org could abuse the project and push some
change, but that they have similar perspective (bubble effect) and some
important aspects of a large feature could escape their attention.

I suggest it to be not a rigid rule, but a recommendation for authors of
large proposals to try to attract reviewers from other orgs.

On Fri, 11 Jan 2019 at 02:51, Julian Hyde <jh...@apache.org> wrote:

> I agree with Gian.
>
> As an Apache committer, you only have one affiliation: you are working in
> the best interests of the project.
>
> Obviously, in the real world there are other pressures. But we do our best
> to compensate for them.
>
> Also, as a a community we try to design our process so as to avoid undue
> influences. For instance, when I advocate for logging cases early, I am
> trying to mitigate the effect of product managers and VPs of engineering,
> who like to have their say in meeting rooms rather than on public mailing
> lists. That’s just one example; if we see other influences at play, let’s
> evolve our process to try to level the playing field.
>
> Julian
>
>
> > On Jan 10, 2019, at 10:40 AM, Gian Merlino <gi...@apache.org> wrote:
> >
> >>> What do community members think about also making a requirement that
> >>> "Design Review" PRs and proposals are reviewed by at least two people
> > with
> >>> different affiliation?
> >> This seems pretty reasonable to me. I haven't found anything in Apache
> >> voting procedure docs (https://www.apache.org/foundation/voting.html)
> that
> >> seems to explicitly forbid something like this yet at least.
> >
> > On the other hand, my understanding of the Apache Way would mean that
> this
> > kind of rule doesn't make sense. In particular from
> > https://www.apache.org/foundation/how-it-works.html: "We firmly believe
> in
> > hats. Your role at the ASF is one assigned to you personally, and is
> > bestowed on you by your peers. It is not tied to your job or current
> > employer or company." That sentiment seems incompatible with making
> > explicit rules about organizational diversity in voting. I have also
> heard
> > a few people say things like: people are supposed to represent
> themselves,
> > not their employers.
> >
> > Obviously, though, people's actions and opinions are influenced by their
> > employer. IMO, a better way to approach a potential problem there is that
> > if people from a particular organization end up behaving abusively, then
> > PMC members from other organizations (or in extreme situations the Apache
> > board itself) should tell them to knock it off. And then do more serious
> > things if the inappropriate behavior continues. This kind
> >
> > The other thing you brought up, promoting more shared ownership and
> > awareness, I am hopeful that separating proposals from PRs will help with
> > that. One reason is that it takes much less time to understand a well
> > written proposal than it takes to understand a PR. Code is very clear but
> > it is also very verbose and takes a while to read and understand. So in
> > theory changing how we operate, in this way, should promote more
> > understanding of more people across the code base.
> >
> > On Thu, Jan 10, 2019 at 1:34 AM Clint Wylie <cl...@imply.io>
> wrote:
> >
> >>>
> >>> What do community members think about also making a requirement that
> >>> "Design Review" PRs and proposals are reviewed by at least two people
> >> with
> >>> different affiliation?
> >>
> >>
> >> This seems pretty reasonable to me. I haven't found anything in Apache
> >> voting procedure docs (https://www.apache.org/foundation/voting.html)
> that
> >> seems to explicitly forbid something like this yet at least.
> >>
> >> On Wed, Jan 9, 2019 at 8:52 PM Roman Leventov <le...@gmail.com>
> >> wrote:
> >>
> >>> What do community members think about also making a requirement that
> >>> "Design Review" PRs and proposals are reviewed by at least two people
> >> with
> >>> different affiliation? IMO it's a good idea, because it ensures that
> >>> different interests are taken into account. Also it pushes people to
> >> engage
> >>> with work done in other parts of Druid, improving shared code owning
> and
> >>> awareness.
> >>>
> >>> Or it's against the Apache Way because it's assumed that there are no
> >>> company boundaries within the community?
> >>>
> >>> On Thu, 10 Jan 2019 at 11:45, Roman Leventov <le...@gmail.com>
> >>> wrote:
> >>>
> >>>> I see two important reasons why it makes sense to file an issue and
> >>>> probably announce it in the mailing list, before writing a lot of
> code,
> >>>> despite not having a clear picture of what it will be and any
> >> performance
> >>>> data:
> >>>> 1) somebody could already work on this problem privately in parallel,
> >> it
> >>>> allows to avoid clash in people's works
> >>>> 2) some people could quickly think about the problem field and share
> >>>> high-level ideas that could wildly change the direction in which the
> >>> author
> >>>> (the person who is going to write code) will move in his work from
> >> early
> >>> on.
> >>>>
> >>>> Jihoon in
> >>>>
> >>>
> >>
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> >>> and
> >>>> later Gian in this thread suggested that _every_ piece of work that
> >>> should
> >>>> be labelled as "Design Review" according to the current rules should
> be
> >>>> accompanied by an issue. I don't agree with this, there are some PRs
> as
> >>>> small as a few dozens of lines of code, that add some configuration
> >>>> parameter and therefore should be labelled "Design Review". I don't
> >>> thing a
> >>>> separate proposal issue is needed for them, and even for a little
> >> larger
> >>>> PRs too.
> >>>>
> >>>> For the same reason, I also don't see the point of renaming "Design
> >>>> Review" into "Proposal", as well as separating "Design Review" into
> >>>> "Proposal" and something like "API Review". I think a single "Design
> >>>> Review" tag handles it well.
> >>>>
> >>>> Gian mentioned an idea that PRs that follow a "Design Review" proposal
> >>>> issue shouldn't be "Design Review" themselves. I don't agree with
> >> this, I
> >>>> think that actual code and performance data are important inputs that
> >>>> should be re-evaluated at least by two people. I even think that it's
> >>> very
> >>>> desirable that at least two people read _each line of production code_
> >> in
> >>>> large PRs, although it's not what was done historically in Druid,
> >> because
> >>>> large bodies of newly added code, with whole new classes and
> subsystems
> >>>> added, are also coincidentally tested worse than already existing
> >> classes
> >>>> and subsystems, including in production. It seems to me that those
> huge
> >>>> code influxes is a major source of bugs, that could later take years
> to
> >>>> squeeze out from the codebase.
> >>>>
> >>>> On Wed, 9 Jan 2019 at 08:24, Clint Wylie <cl...@imply.io>
> wrote:
> >>>>
> >>>>> Apologies for the delayed response.
> >>>>>
> >>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> >>> slim.bouguerra@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> I am wondering here what is the case where code first is better?
> >>>>>>
> >>>>>
> >>>>> I don't think it's wrong to share ideas as early as possible, and
> >> after
> >>>>> this discussion I think I am in favor of it too. I just meant that I
> >>> don't
> >>>>> think it's always necessarily the most productive discussion until
> >> code
> >>>>> exists sometimes, with the types of thing I am thinking of are almost
> >>>>> entirely limited to cases where things might sound good to anyone on
> >>> paper
> >>>>> but in reality need a large amount of experiments conducted and
> >>>>> observations collected to determine that something is actually worth
> >>>>> doing,
> >>>>> which I imagine is mainly things like reworking internals for
> >>> performance
> >>>>> improvements.
> >>>>>
> >>>>> In the case of my combined proposal PR, I needed to prove that the
> >>> thing I
> >>>>> was working on was a good idea... and it wasn't directly. But I came
> >> up
> >>>>> with another idea during the course of experiment turned into
> >> something
> >>>>> compelling, so an initial proposal would have looked quite a lot
> >>> different
> >>>>> than what I ended up with. Once I had proven to myself that it was a
> >>> good
> >>>>> idea, then I was comfortable sharing with the wider community. I'm
> not
> >>>>> certain how this would play out in an always proposal first model,
> >> maybe
> >>>>> the first proposal exists, I personally reject it after updating with
> >>>>> experiment results show it's a bad idea, continue experimenting and
> >>> raise
> >>>>> a
> >>>>> new one after the experiments start looking promising?
> >>>>>
> >>>>>
> >>>>>> Let's not be naive this is very rare that a contributor will accept
> >>> that
> >>>>>> his work is to be thrown, usually devs takes coding as personal
> >>> creation
> >>>>>> and they get attached to it.
> >>>>>>
> >>>>>
> >>>>> I agree, just because a handful of the committers have this attitude,
> >> it
> >>>>> isn't fair to expect the wider community to also, that's why I am in
> >>> favor
> >>>>> of formalizing the process.
> >>>>>
> >>>>> Can you please explain what is overbearing ? what can be changed to
> >> make
> >>>>> it
> >>>>>> easy ?
> >>>>>> Most of the points are kind of the actual questions that you want to
> >>>>>> address before hand anyway isn't ?
> >>>>>>
> >>>>>
> >>>>> Sorry for the confusion, I said it's "not overbearing", I think it's
> >>> fine.
> >>>>>
> >>>>> What are the complaints ?
> >>>>>
> >>>>>
> >>>>> Is this and other previous threads not a complaint about opening a
> >> large
> >>>>> PR
> >>>>> without a proposal? :) I just mean that formalizing the process, even
> >>> if a
> >>>>> proposal has a reference PR opened with it near concurrently, could
> >>>>> prevent
> >>>>> these discussions from happening in the future because ground rules
> >> have
> >>>>> been set and we are all on the same page I guess.
> >>>>>
> >>>>> I think we should also probably consider renaming the "design review"
> >>>>> label
> >>>>> to "proposal" or something to make it more clear that a PR is
> >> associated
> >>>>> with a proposal. It might also be worth considering using github
> >>> projects
> >>>>> for particularly large things that involve multiple follow up PRs,
> >> but I
> >>>>> haven't used them in much depth to know if they add anything.
> >>>>>
> >>>>> It seems like we are all converging on agreement to do a github issue
> >>>>> proposal for larger changes (I would vote for announcing them on the
> >> dev
> >>>>> list too for more visibility), so that design review is separated
> from
> >>>>> code
> >>>>> review. I guess my main concern was not wanting to discourage
> >>>>> experimentation by walling it off behind mandatory discussions, but
> >> the
> >>>>> more I think about it,  it doesn't technically change much about this
> >>>>> process, it just requires a more formal proposal to accompany any
> >>>>> experiments that are shared as a PR.
> >>>>>
> >>>>>
> >>>>> On Tue, Jan 8, 2019 at 12:28 PM Gian Merlino <gi...@apache.org>
> wrote:
> >>>>>
> >>>>>> I think for us, choosing to use GitHub issues as discussion threads
> >>> for
> >>>>>> potential 'major' contributions would be a good idea, especially if
> >> we
> >>>>>> encourage people to start them before PRs show up. Definitely agree
> >>> that
> >>>>>> all contributors should go through the same process -- I couldn't
> >> see
> >>> it
> >>>>>> working well any other way.
> >>>>>>
> >>>>>> On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <jh...@apache.org>
> >> wrote:
> >>>>>>
> >>>>>>> Statically, yes, GitHub PRs are the same as GitHub cases. But
> >>>>>> dynamically,
> >>>>>>> they are different, because you can only log a PR when you have
> >>>>> finished
> >>>>>>> work.
> >>>>>>>
> >>>>>>> A lot of other Apache projects use JIRA, so there is a clear
> >>>>> distinction
> >>>>>>> between cases and contributions. JIRA cases, especially when
> >> logged
> >>>>> early
> >>>>>>> in the lifecycle of a contribution, become long-running
> >> conversation
> >>>>>>> threads with a lot of community participation. If the Druid chose
> >> to
> >>>>> do
> >>>>>> so,
> >>>>>>> GitHub cases could be the same.
> >>>>>>>
> >>>>>>> Be careful that you do not treat “potential contributors” (by
> >> which
> >>> I
> >>>>>>> presume you mean non-committers) differently from committers and
> >> PMC
> >>>>>>> members. Anyone starting a major piece of work should follow the
> >>> same
> >>>>>>> process. (Experienced committers probably have a somewhat better
> >>> idea
> >>>>>> what
> >>>>>>> work will turn out to be “major”, so they get a little more
> >> leeway.)
> >>>>>>>
> >>>>>>> Julian
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Jan 7, 2019, at 12:10 PM, Gian Merlino <gi...@apache.org>
> >>> wrote:
> >>>>>>>>
> >>>>>>>> I don't think there's a need to raise issues for every change: a
> >>>>> small
> >>>>>>> bug
> >>>>>>>> fix or doc fix should just go straight to PR. (GitHub PRs show
> >> up
> >>> as
> >>>>>>> issues
> >>>>>>>> in the issue-search UI/API, so it's not like this means the
> >> patch
> >>>>> has
> >>>>>> no
> >>>>>>>> corresponding issue -- in a sense the PR _is_ the issue.)
> >>>>>>>>
> >>>>>>>> I do think it makes sense to encourage potential contributors to
> >>>>> write
> >>>>>> to
> >>>>>>>> the dev list or raise an issue if they aren't sure if something
> >>>>> would
> >>>>>>> need
> >>>>>>>> to go through a more heavy weight process.
> >>>>>>>>
> >>>>>>>> Fwiw we do have a set of 'design review' criteria already (we
> >> had
> >>> a
> >>>>>>>> discussion about this a couple years ago) at:
> >>>>>>>> http://druid.io/community/#getting-your-changes-accepted. So we
> >>>>>>> wouldn't be
> >>>>>>>> starting from zero on defining that. We set it up back when we
> >>> were
> >>>>>>> trying
> >>>>>>>> to _streamline_ our process -- we used to require two non-author
> >>> +1s
> >>>>>> for
> >>>>>>>> _every_ change, even minor ones. The introduction of design
> >> review
> >>>>>>> criteria
> >>>>>>>> was meant to classify which PRs need that level of review and
> >>> which
> >>>>>> ones
> >>>>>>>> are minor and can be merged with less review. I do think it
> >> helped
> >>>>> with
> >>>>>>>> getting minor PRs merged more quickly. The list of criteria is,
> >>>>>>>>
> >>>>>>>> - Major architectural changes or API changes
> >>>>>>>> - HTTP requests and responses (e. g. a new HTTP endpoint)
> >>>>>>>> - Interfaces for extensions
> >>>>>>>> - Server configuration (e. g. altering the behavior of a config
> >>>>>> property)
> >>>>>>>> - Emitted metrics
> >>>>>>>> - Other major changes, judged by the discretion of Druid
> >>> committers
> >>>>>>>>
> >>>>>>>> Some of it is subjective, but it has been in place for a while,
> >> so
> >>>>> it's
> >>>>>>> at
> >>>>>>>> least something we are relatively familiar with.
> >>>>>>>>
> >>>>>>>> On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <jh...@apache.org>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Small contributions don’t need any design review, whereas large
> >>>>>>>>> contributions need significant review. I don’t think we should
> >>>>> require
> >>>>>>> an
> >>>>>>>>> additional step for those (many) small contributions. But who
> >>>>> decides
> >>>>>>>>> whether a contribution fits into the small or large category?
> >>>>>>>>>
> >>>>>>>>> I think the solution is for authors to log a case (or send an
> >>>>> email to
> >>>>>>>>> dev) before they start work on any contribution. Then
> >> committers
> >>>>> can
> >>>>>>>>> request a more heavy-weight process if they think it is needed.
> >>>>>>>>>
> >>>>>>>>> Julian
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <gi...@apache.org>
> >>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> It sounds like splitting design from code review is a common
> >>> theme
> >>>>>> in a
> >>>>>>>>> few
> >>>>>>>>>> of the posts here. How does everyone feel about making a point
> >>> of
> >>>>>>>>>> encouraging design reviews to be done as issues, separate from
> >>> the
> >>>>>> pull
> >>>>>>>>>> request, with the expectations that (1) the design review
> >> issue
> >>>>>>>>>> ("proposal") should generally appear somewhat _before_ the
> >> pull
> >>>>>>> request;
> >>>>>>>>>> (2) pull requests should _not_ have design review happen on
> >>> them,
> >>>>>>> meaning
> >>>>>>>>>> there should no longer be PRs with design review tags, and we
> >>>>> should
> >>>>>>> move
> >>>>>>>>>> the design review approval process to the issue rather than
> >> the
> >>>>> PR.
> >>>>>>>>>>
> >>>>>>>>>> For (1), even if we encourage design review discussions to
> >> start
> >>>>>>> before a
> >>>>>>>>>> pull request appears, I don't see an issue with them running
> >>>>>>> concurrently
> >>>>>>>>>> for a while at some point.
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <
> >> jonwei@apache.org>
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Thanks for raising these concerns!
> >>>>>>>>>>>
> >>>>>>>>>>> My initial thoughts:
> >>>>>>>>>>> - I agree that separation of design review and code-level
> >>> review
> >>>>> for
> >>>>>>>>> major
> >>>>>>>>>>> changes would be more efficient
> >>>>>>>>>>>
> >>>>>>>>>>> - I agree that a clear, more formalized process for handling
> >>>>> major
> >>>>>>>>> changes
> >>>>>>>>>>> would be helpful for contributors:
> >>>>>>>>>>> - Define what is considered a major change
> >>>>>>>>>>> - Define a standard proposal structure, KIP-style proposal
> >>> format
> >>>>>>>>> sounds
> >>>>>>>>>>> good to me
> >>>>>>>>>>>
> >>>>>>>>>>> - I think it's too rigid to have a policy of "no code at all
> >>> with
> >>>>>> the
> >>>>>>>>>>> initial proposal"
> >>>>>>>>>>> - Code samples can be useful references for understanding
> >>> aspects
> >>>>>> of a
> >>>>>>>>>>> design
> >>>>>>>>>>> - In some cases it's necessary to run experiments to fully
> >>>>>> understand
> >>>>>>> a
> >>>>>>>>>>> problem and determine an appropriate design, or to determine
> >>>>> whether
> >>>>>>>>>>> something is even worth doing before committing to the work
> >> of
> >>>>>>> fleshing
> >>>>>>>>> out
> >>>>>>>>>>> a proposal, prototype code is a natural outcome of that and
> >> I'm
> >>>>> not
> >>>>>>>>> against
> >>>>>>>>>>> someone providing such code for reference
> >>>>>>>>>>> - I tend to view design/code as things that are often
> >> developed
> >>>>>>>>>>> simultaneously in an intertwined way
> >>>>>>>>>>>
> >>>>>>>>>>>> Let's not be naive this is very rare that a contributor will
> >>>>> accept
> >>>>>>>>> that
> >>>>>>>>>>> his work is to be thrown, usually devs takes coding as
> >> personal
> >>>>>>> creation
> >>>>>>>>>>> and they get attached to it.
> >>>>>>>>>>>
> >>>>>>>>>>> If we have a clear review process that emphasizes the need
> >> for
> >>>>> early
> >>>>>>>>>>> consensus building, with separate design and code review,
> >> then
> >>> I
> >>>>>> feel
> >>>>>>>>> we've
> >>>>>>>>>>> done enough and don't need a hard rule against having some
> >> code
> >>>>>> linked
> >>>>>>>>> with
> >>>>>>>>>>> the initial proposal. If a potential contributor then still
> >>>>> wants to
> >>>>>>> go
> >>>>>>>>>>> ahead and write a lot of code that may be rejected or change
> >>>>>>>>> significantly,
> >>>>>>>>>>> the risks were made clear.
> >>>>>>>>>>>
> >>>>>>>>>>>> Once code is written hard to think abstract.
> >>>>>>>>>>>
> >>>>>>>>>>> I can see the validity of the concern, but I personally don't
> >>>>> see it
> >>>>>>> as
> >>>>>>>>> a
> >>>>>>>>>>> huge risk. My impression from the Druid PR reviews I've seen
> >> is
> >>>>> that
> >>>>>>> our
> >>>>>>>>>>> reviewers are able to keep abstract design vs. implementation
> >>>>>> details
> >>>>>>>>>>> separate and consider alternate designs when reviewing.
> >>>>>>>>>>>
> >>>>>>>>>>> To summarize I think it's probably enough to have a policy
> >>> along
> >>>>> the
> >>>>>>>>> lines
> >>>>>>>>>>> of:
> >>>>>>>>>>> - Create more formalized guidelines for proposals and what
> >>>>> changes
> >>>>>>>>> require
> >>>>>>>>>>> proposals
> >>>>>>>>>>> - Separate design and code review for major changes, with
> >>> design
> >>>>>>> review
> >>>>>>>>>>> first, code-level review after reaching consensus on the
> >>> design.
> >>>>>>>>>>> - Code before the design review is completed is just for
> >>>>> reference,
> >>>>>>> not
> >>>>>>>>>>> regarded as a candidate for review/merging.
> >>>>>>>>>>>
> >>>>>>>>>>> - Jon
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> >>>>>>>>> slim.bouguerra@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <
> >>>>> clint.wylie@imply.io>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I am definitely biased in this matter as an owner of
> >> another
> >>>>> large
> >>>>>>> PR
> >>>>>>>>>>>> that
> >>>>>>>>>>>>> wasn't preceded by a direct proposal or dev list
> >> discussion,
> >>>>> and
> >>>>>> in
> >>>>>>>>>>>> general
> >>>>>>>>>>>>> I agree that proposal first is usually better, but I think
> >> in
> >>>>> some
> >>>>>>>>>>> rarer
> >>>>>>>>>>>>> cases approaching a problem code first *is* the most
> >>>>> appropriate
> >>>>>> way
> >>>>>>>>> to
> >>>>>>>>>>>>> have a discussion.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I am wondering here what is the case where code first is
> >>> better?
> >>>>>>>>>>>> In general when you are writing code you have an idea about
> >>> what
> >>>>>> you
> >>>>>>>>> want
> >>>>>>>>>>>> to change, why you want to change and why you want to change
> >>> it.
> >>>>>>>>>>>> I do not see what is wrong with sharing this primitive ideas
> >>> and
> >>>>>>>>> thoughts
> >>>>>>>>>>>> as an abstract proposal (at least to avoid overlapping)
> >>>>>>>>>>>>
> >>>>>>>>>>>> I see nothing wrong with it so long as the author
> >>>>>>>>>>>>> accepts that the PR is treated as a combined proposal and
> >>>>> proof of
> >>>>>>>>>>>> concept,
> >>>>>>>>>>>>> and fair game to be radically changed via discussion or
> >> even
> >>>>>>> rejected,
> >>>>>>>>>>>>> which sounds like Gian's attitude on the matter and is mine
> >>> as
> >>>>>> well
> >>>>>>>>>>> with
> >>>>>>>>>>>> my
> >>>>>>>>>>>>> compression stuff.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Let's not be naive this is very rare that a contributor will
> >>>>> accept
> >>>>>>>>> that
> >>>>>>>>>>>> his work is to be thrown, usually devs takes coding as
> >>> personal
> >>>>>>>>> creation
> >>>>>>>>>>>> and they get attached to it.
> >>>>>>>>>>>> To my point you can take a look on some old issue in the
> >> Druid
> >>>>>> forum
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> >>>>>>>>>>>> and am sure other communities have similar problems.
> >>>>>>>>>>>> So leaving the door open to some side cases is not a good
> >> idea
> >>>>> in
> >>>>>> my
> >>>>>>>>>>>> opinion and will lead to similar issue in the future.
> >>>>>>>>>>>>
> >>>>>>>>>>>> This seems to me especially likely to happen in cases
> >>>>>>>>>>>>> where an approach still needs proven to be a viable idea
> >> *to
> >>>>> the
> >>>>>>>>>>> author*,
> >>>>>>>>>>>>> so that a much more productive discussion can be had in the
> >>>>> first
> >>>>>>>>>>> place.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I think there is a trade off, I don't think we want to
> >>>>> discourage
> >>>>>>>>>>>>> experimentation by walling it off behind mandatory
> >>> discussions
> >>>>>>> before
> >>>>>>>>>>> it
> >>>>>>>>>>>>> can even start, but I do think formalizing the process for
> >>>>> large
> >>>>>>>>>>> changes
> >>>>>>>>>>>> is
> >>>>>>>>>>>>> a good thing, especially since we probably can't expect the
> >>>>> wider
> >>>>>>>>>>>> community
> >>>>>>>>>>>>> to have the same attitude towards a large PR getting
> >>> discarded
> >>>>> as
> >>>>>> a
> >>>>>>>>>>>>> committer might. I think the Kafka approach is reasonable,
> >> a
> >>>>> bit
> >>>>>>> more
> >>>>>>>>>>>>> formal than our design review process but not overbearing.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Can you please explain what is overbearing ? what can be
> >>>>> changed to
> >>>>>>>>> make
> >>>>>>>>>>> it
> >>>>>>>>>>>> easy ?
> >>>>>>>>>>>> Most of the points are kind of the actual questions that you
> >>>>> want
> >>>>>> to
> >>>>>>>>>>>> address before hand anyway isn't ?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Going code first
> >>>>>>>>>>>>> should be in general discouraged, but when it does happen,
> >> it
> >>>>>> seems
> >>>>>>>>>>> like
> >>>>>>>>>>>>> opening DIP/an issue/starting a mailing list thread or
> >>>>> whatever we
> >>>>>>> go
> >>>>>>>>>>>> with
> >>>>>>>>>>>>> to have a more high level design discussion alongside the
> >>>>>> reference
> >>>>>>> PR
> >>>>>>>>>>>>> could alleviate some of these complaints?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> What are the complaints ?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> +1 for "DIP" heh, I think making
> >>>>>>>>>>>>> them in the form of github issues is probably appropriate,
> >>>>> with a
> >>>>>>> dev
> >>>>>>>>>>>> list
> >>>>>>>>>>>>> thread to announce them perhaps?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think  github issue with [Proposal] header like
> >>>>>>>>>>>> https://github.com/apache/incubator-druid/issues/4349 is
> >> good
> >>>>> to
> >>>>>> me,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <
> >>>>> bslim@apache.org>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks everyone for interacting with this thread.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The fact that i, Roman, Jihoon  and others in the past (FJ
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ
> >>>>> )
> >>>>>>>>>>>>>> raised this point indicates that PRs without a proposal
> >> are
> >>>>>> indeed
> >>>>>>> an
> >>>>>>>>>>>>> issue
> >>>>>>>>>>>>>> and we need to solve it.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Something Similar to KIP maybe called DIPs is fine with
> >> me.
> >>>>>>>>>>>>>> What i strive to see is the following:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [Step 1] formalize what is the kind of work that needs a
> >>>>> formal
> >>>>>>>>>>>>> Proposal, I
> >>>>>>>>>>>>>> think Roman and Jihoon has already covered that pretty
> >> well.
> >>>>> am
> >>>>>> +1
> >>>>>>> on
> >>>>>>>>>>>>> that.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I am strongly in favor of the separation of Proposal
> >> Review
> >>>>> and
> >>>>>>>>>>> (later)
> >>>>>>>>>>>>>> Code review PRs. My  main reasons:
> >>>>>>>>>>>>>> Most importantly code reviewing will introduce lot of
> >> noise
> >>>>> and
> >>>>>>> will
> >>>>>>>>>>>>>> ultimately make  the GitHub page unreadable.
> >>>>>>>>>>>>>> Avoid overlapping of work.
> >>>>>>>>>>>>>> Once code is written hard to think abstract.
> >>>>>>>>>>>>>> Separate page for Design review later can always be used
> >> it
> >>>>> as a
> >>>>>>>>>>> Design
> >>>>>>>>>>>>>> document that is readable and code free-ish.
> >>>>>>>>>>>>>> As i said the goal of this first round is to see if the
> >>>>> community
> >>>>>>>>>>> agree
> >>>>>>>>>>>>>> about such change, then make the process of design more
> >>>>> inclusive
> >>>>>>>>>>> thus
> >>>>>>>>>>>>>> other contributors can submit a counter proposals.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [Step 2] IF everybody agree about that point Step 2 is to
> >>>>> define
> >>>>>>>>>>> which
> >>>>>>>>>>>>>> medium is used to Publish a primitive form of a CODE FREE
> >>>>>> Abstract
> >>>>>>>>>>>>> Proposal
> >>>>>>>>>>>>>> containing at least the following bullet points.
> >>>>>>>>>>>>>> - The problem description and motivation
> >>>>>>>>>>>>>> - Overview of the proposed change
> >>>>>>>>>>>>>> - Operational impact (compatibility/ plans to upgrades)
> >>> public
> >>>>>> API
> >>>>>>>>>>>>> changes,
> >>>>>>>>>>>>>> configuration changes, algorithm, and so on
> >>>>>>>>>>>>>> - Expected benefits and drawbacks
> >>>>>>>>>>>>>> - Rationale and alternatives
> >>>>>>>>>>>>>> - Estimate Time to Deliver if possible.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The way i think this can be is a Github issue where member
> >>> of
> >>>>> the
> >>>>>>>>>>>>> community
> >>>>>>>>>>>>>> will interact via comments and the author will be updating
> >>> the
> >>>>>>>>>>>>> description
> >>>>>>>>>>>>>> in the light of comments provided by the community.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> During and near the end of the design discussions the
> >>> author/s
> >>>>>> can
> >>>>>>>>>>>> start
> >>>>>>>>>>>>>> writing POCs to help guide the review process this
> >> naturally
> >>>>> will
> >>>>>>> be
> >>>>>>>>>>> a
> >>>>>>>>>>>>> Pull
> >>>>>>>>>>>>>> request with actual code.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Now the most important thing is that we need to agree
> >> that
> >>>>> any
> >>>>>>> work
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>> does not align with this formal process will be ignored
> >> and
> >>>>> the
> >>>>>>>>>>> author
> >>>>>>>>>>>>> will
> >>>>>>>>>>>>>> be asked to start with a DIP*
> >>>>>>>>>>>>>> *That is what i meant with  “If it didn’t happen on the
> >>>>> mailing
> >>>>>>> list,
> >>>>>>>>>>>> it
> >>>>>>>>>>>>>> didn’t happen.”*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks and happy coding!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <
> >>> gian@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> One of the advantages I see with a more formal process is
> >>>>> (like
> >>>>>>>>>>> Kafka
> >>>>>>>>>>>>>> KIPs)
> >>>>>>>>>>>>>>> is that it levels the playing field a bit and sets some
> >>>>> ground
> >>>>>>>>>>> rules
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>>>> working together. In a way it can help encourage
> >>>>> contributions
> >>>>>> by
> >>>>>>>>>>>>> making
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>> clear what is expected of potential contributors.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> We have a design review process today that is not as
> >> formal
> >>>>> as
> >>>>>>>>>>> KIPs,
> >>>>>>>>>>>>> but
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> somewhat heavier than the one you describe. Maybe we
> >> could
> >>>>> tweak
> >>>>>>>>>>> our
> >>>>>>>>>>>>>>> current one by starting to do design reviews separately
> >>> from
> >>>>>> PRs.
> >>>>>>>>>>>> i.e.,
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>> anything that meets our 'design review' criteria, do that
> >>> on
> >>>>> the
> >>>>>>>>>>> dev
> >>>>>>>>>>>>> list
> >>>>>>>>>>>>>>> or in a separate issue, and keep the PR focused on
> >>> code-level
> >>>>>>>>>>> stuff.
> >>>>>>>>>>>>> That
> >>>>>>>>>>>>>>> way we don't end up trying to do both at once. And it
> >> makes
> >>>>> it
> >>>>>>>>>>> easier
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> start talking about design before the code is ready,
> >> which
> >>>>> would
> >>>>>>> be
> >>>>>>>>>>>>>> better.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <
> >>> jhyde@apache.org
> >>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> It’s really hard to say no to a contribution when
> >> someone
> >>>>> has
> >>>>>> put
> >>>>>>>>>>>> in
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>> significant amount of work.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The following approach is simple and works really well:
> >>>>> Before
> >>>>>>>>>>> you
> >>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>> work, log a case, describing the problem. When you have
> >>> some
> >>>>>>>>>>> ideas
> >>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>> design, add those to the case. When you have a code
> >>> branch,
> >>>>> add
> >>>>>>>>>>> its
> >>>>>>>>>>>>> URL
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> the case. And so forth. At any point in the proceedings,
> >>>>> people
> >>>>>>>>>>> can
> >>>>>>>>>>>>>> chime
> >>>>>>>>>>>>>>>> in with their opinions.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> In my opinion, a formal “design review” process is not
> >>>>>> necessary.
> >>>>>>>>>>>>> Just
> >>>>>>>>>>>>>>>> build consensus iteratively, by starting the
> >> conversation
> >>>>> early
> >>>>>>>>>>> in
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> process.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Julian
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <
> >>> gian@apache.org
> >>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> In this particular case: please consider the PR as a
> >>>>> proposal.
> >>>>>>>>>>>>> Don't
> >>>>>>>>>>>>>>> feel
> >>>>>>>>>>>>>>>>> like just because there is code there that takes a
> >>> certain
> >>>>>>>>>>>>> approach,
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> the approach is somehow sacred. I had to implement
> >>>>> something
> >>>>>> to
> >>>>>>>>>>>>>>>> crystallize
> >>>>>>>>>>>>>>>>> my own thinking about how the problem could be
> >>> approached.
> >>>>> I
> >>>>>>>>>>>> won't
> >>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> disappointed if, as a community, we decide a different
> >>>>>>>>>>> direction
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>>>> better
> >>>>>>>>>>>>>>>>> and the code all gets thrown away. That's one of the
> >>>>> reasons
> >>>>>>>>>>>> that I
> >>>>>>>>>>>>>>>> removed
> >>>>>>>>>>>>>>>>> the 0.14.0 milestone that was added to the patch. (I
> >>> don't
> >>>>>> want
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> rush
> >>>>>>>>>>>>>>>> it,
> >>>>>>>>>>>>>>>>> nor do I think that's a good idea.)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> In general: Sounds like we could do with some more
> >>>>>>>>>>> formalization
> >>>>>>>>>>>>>> around
> >>>>>>>>>>>>>>>>> what a proposal looks like, which sorts of changes need
> >>>>> one,
> >>>>>>>>>>> and
> >>>>>>>>>>>>> when
> >>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> the dev cycle it is appropriate. FWIW I think Kafka's
> >>>>> process
> >>>>>>>>>>> is
> >>>>>>>>>>>>> more
> >>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>> less fine, and would be okay with adopting it for Druid
> >>> if
> >>>>>>>>>>> people
> >>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>> it.
> >>>>>>>>>>>>>>>>> Right now our standards for what requires a "design
> >>> review"
> >>>>>> are
> >>>>>>>>>>>>> very
> >>>>>>>>>>>>>>>>> similar to the Kafka community standards for what
> >>> requires
> >>>>> a
> >>>>>>>>>>> KIP,
> >>>>>>>>>>>>> so
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> have some familiarity with those concepts. However we
> >>> don't
> >>>>>>>>>>>>> separate
> >>>>>>>>>>>>>> PR
> >>>>>>>>>>>>>>>>> review and proposal discussion as strictly as they do,
> >>>>> which
> >>>>>>>>>>>> seems
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> the foundation for the feeling of exclusion that is
> >> being
> >>>>> felt
> >>>>>>>>>>>>> here.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Separately: I just redid the description on
> >>>>>>>>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794 to
> >>> be
> >>>>>> more
> >>>>>>>>>>>>>>>> proposal-y.
> >>>>>>>>>>>>>>>>> I followed the KIP style:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >>>>>>>>>>>>>>>> .
> >>>>>>>>>>>>>>>>> Please refresh the page and see if it looks more
> >> useful.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Gian
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <
> >>>>> jhyde@apache.org
> >>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Slim,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I agree with your points that offline development is
> >> bad
> >>>>> for
> >>>>>>>>>>>>>>> community.
> >>>>>>>>>>>>>>>>>> But I don’t think you need much mentor help. You have
> >>>>> raised
> >>>>>>>>>>>> valid
> >>>>>>>>>>>>>>>> issues
> >>>>>>>>>>>>>>>>>> and the Druid community needs to decide what its
> >>>>> development
> >>>>>>>>>>>>>> practices
> >>>>>>>>>>>>>>>>>> should be.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Julian
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
> >>>>>>>>>>> bslim@apache.org>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hello everyone and hope you all have very good
> >>> holidays.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> First, this email is not directed on the author or
> >> the
> >>> PR
> >>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794
> >>> it
> >>>>>>>>>>> self,
> >>>>>>>>>>>>> but
> >>>>>>>>>>>>>> i
> >>>>>>>>>>>>>>>> see
> >>>>>>>>>>>>>>>>>>> this PR as a perfect example.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> One of the foundation of Apache Way or what i would
> >>>>> simply
> >>>>>>>>>>> call
> >>>>>>>>>>>>>> open
> >>>>>>>>>>>>>>>>>> source
> >>>>>>>>>>>>>>>>>>> community driven development is that "Technical
> >>> decisions
> >>>>>> are
> >>>>>>>>>>>>>>>> discussed,
> >>>>>>>>>>>>>>>>>>> decided, and archived publicly.
> >>>>>>>>>>>>>>>>>>> developpement"
> >>>>>>>>>>>>>>>>>>> Which means that big technical  changes such as the
> >> one
> >>>>>>>>>>> brought
> >>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>> #/6794
> >>>>>>>>>>>>>>>>>>> should have started as a proposal and round of
> >>>>> discussions
> >>>>>>>>>>>> about
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> major
> >>>>>>>>>>>>>>>>>>> changes designs not as 11K line of code.
> >>>>>>>>>>>>>>>>>>> I believe such openness will promote a lot of good
> >>>>> benefits
> >>>>>>>>>>>> such
> >>>>>>>>>>>>>> as:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> - ensures community health and growth.
> >>>>>>>>>>>>>>>>>>> - ensures everyone can participate not only the
> >> authors
> >>>>> and
> >>>>>>>>>>> his
> >>>>>>>>>>>>>>>>>> co-workers.
> >>>>>>>>>>>>>>>>>>> - ensures that the project is driven by the community
> >>> and
> >>>>>>>>>>> not a
> >>>>>>>>>>>>>> given
> >>>>>>>>>>>>>>>>>>> company or an individual.
> >>>>>>>>>>>>>>>>>>> - ensures that there is consensus (not saying 100%
> >>>>>>>>>>> agreement;)
> >>>>>>>>>>>>>>> however
> >>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>> means that all individuals will accept the current
> >>>>> progress
> >>>>>>>>>>> on
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> project
> >>>>>>>>>>>>>>>>>>> until some better proposal is put forth.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Personally such BIG offline PR makes me feel excluded
> >>> and
> >>>>>>>>>>>> doesn't
> >>>>>>>>>>>>>>> give
> >>>>>>>>>>>>>>>>>> me a
> >>>>>>>>>>>>>>>>>>> sense that i belong to  a community at all.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> To prevent such off list development i think as a
> >> Druid
> >>>>>>>>>>>> Community
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>> to stick to the apache way “If it didn’t happen on
> >> the
> >>>>>>>>>>> mailing
> >>>>>>>>>>>>>> list,
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>> didn’t happen.”
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I would appreciate if some of the Apache mentor help
> >>> with
> >>>>>>>>>>> this.
> >>>>>>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>
> >>> ---------------------------------------------------------------------
> >>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> >>> dev-unsubscribe@druid.apache.org
> >>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> >>>>> dev-help@druid.apache.org
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>
> >> ---------------------------------------------------------------------
> >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> >> dev-unsubscribe@druid.apache.org
> >>>>>>>>>>>>>>>> For additional commands, e-mail:
> >>> dev-help@druid.apache.org
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>>
> >>>>>>>>>>>> B-Slim
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> >>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> >>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> For additional commands, e-mail: dev-help@druid.apache.org
>
>

Re: Off list major development

[Julian Hyde <jh...@apache.org>]

I agree with Gian.

As an Apache committer, you only have one affiliation: you are working in the best interests of the project.

Obviously, in the real world there are other pressures. But we do our best to compensate for them.

Also, as a a community we try to design our process so as to avoid undue influences. For instance, when I advocate for logging cases early, I am trying to mitigate the effect of product managers and VPs of engineering, who like to have their say in meeting rooms rather than on public mailing lists. That’s just one example; if we see other influences at play, let’s evolve our process to try to level the playing field.

Julian


> On Jan 10, 2019, at 10:40 AM, Gian Merlino <gi...@apache.org> wrote:
> 
>>> What do community members think about also making a requirement that
>>> "Design Review" PRs and proposals are reviewed by at least two people
> with
>>> different affiliation?
>> This seems pretty reasonable to me. I haven't found anything in Apache
>> voting procedure docs (https://www.apache.org/foundation/voting.html) that
>> seems to explicitly forbid something like this yet at least.
> 
> On the other hand, my understanding of the Apache Way would mean that this
> kind of rule doesn't make sense. In particular from
> https://www.apache.org/foundation/how-it-works.html: "We firmly believe in
> hats. Your role at the ASF is one assigned to you personally, and is
> bestowed on you by your peers. It is not tied to your job or current
> employer or company." That sentiment seems incompatible with making
> explicit rules about organizational diversity in voting. I have also heard
> a few people say things like: people are supposed to represent themselves,
> not their employers.
> 
> Obviously, though, people's actions and opinions are influenced by their
> employer. IMO, a better way to approach a potential problem there is that
> if people from a particular organization end up behaving abusively, then
> PMC members from other organizations (or in extreme situations the Apache
> board itself) should tell them to knock it off. And then do more serious
> things if the inappropriate behavior continues. This kind
> 
> The other thing you brought up, promoting more shared ownership and
> awareness, I am hopeful that separating proposals from PRs will help with
> that. One reason is that it takes much less time to understand a well
> written proposal than it takes to understand a PR. Code is very clear but
> it is also very verbose and takes a while to read and understand. So in
> theory changing how we operate, in this way, should promote more
> understanding of more people across the code base.
> 
> On Thu, Jan 10, 2019 at 1:34 AM Clint Wylie <cl...@imply.io> wrote:
> 
>>> 
>>> What do community members think about also making a requirement that
>>> "Design Review" PRs and proposals are reviewed by at least two people
>> with
>>> different affiliation?
>> 
>> 
>> This seems pretty reasonable to me. I haven't found anything in Apache
>> voting procedure docs (https://www.apache.org/foundation/voting.html) that
>> seems to explicitly forbid something like this yet at least.
>> 
>> On Wed, Jan 9, 2019 at 8:52 PM Roman Leventov <le...@gmail.com>
>> wrote:
>> 
>>> What do community members think about also making a requirement that
>>> "Design Review" PRs and proposals are reviewed by at least two people
>> with
>>> different affiliation? IMO it's a good idea, because it ensures that
>>> different interests are taken into account. Also it pushes people to
>> engage
>>> with work done in other parts of Druid, improving shared code owning and
>>> awareness.
>>> 
>>> Or it's against the Apache Way because it's assumed that there are no
>>> company boundaries within the community?
>>> 
>>> On Thu, 10 Jan 2019 at 11:45, Roman Leventov <le...@gmail.com>
>>> wrote:
>>> 
>>>> I see two important reasons why it makes sense to file an issue and
>>>> probably announce it in the mailing list, before writing a lot of code,
>>>> despite not having a clear picture of what it will be and any
>> performance
>>>> data:
>>>> 1) somebody could already work on this problem privately in parallel,
>> it
>>>> allows to avoid clash in people's works
>>>> 2) some people could quickly think about the problem field and share
>>>> high-level ideas that could wildly change the direction in which the
>>> author
>>>> (the person who is going to write code) will move in his work from
>> early
>>> on.
>>>> 
>>>> Jihoon in
>>>> 
>>> 
>> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
>>> and
>>>> later Gian in this thread suggested that _every_ piece of work that
>>> should
>>>> be labelled as "Design Review" according to the current rules should be
>>>> accompanied by an issue. I don't agree with this, there are some PRs as
>>>> small as a few dozens of lines of code, that add some configuration
>>>> parameter and therefore should be labelled "Design Review". I don't
>>> thing a
>>>> separate proposal issue is needed for them, and even for a little
>> larger
>>>> PRs too.
>>>> 
>>>> For the same reason, I also don't see the point of renaming "Design
>>>> Review" into "Proposal", as well as separating "Design Review" into
>>>> "Proposal" and something like "API Review". I think a single "Design
>>>> Review" tag handles it well.
>>>> 
>>>> Gian mentioned an idea that PRs that follow a "Design Review" proposal
>>>> issue shouldn't be "Design Review" themselves. I don't agree with
>> this, I
>>>> think that actual code and performance data are important inputs that
>>>> should be re-evaluated at least by two people. I even think that it's
>>> very
>>>> desirable that at least two people read _each line of production code_
>> in
>>>> large PRs, although it's not what was done historically in Druid,
>> because
>>>> large bodies of newly added code, with whole new classes and subsystems
>>>> added, are also coincidentally tested worse than already existing
>> classes
>>>> and subsystems, including in production. It seems to me that those huge
>>>> code influxes is a major source of bugs, that could later take years to
>>>> squeeze out from the codebase.
>>>> 
>>>> On Wed, 9 Jan 2019 at 08:24, Clint Wylie <cl...@imply.io> wrote:
>>>> 
>>>>> Apologies for the delayed response.
>>>>> 
>>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
>>> slim.bouguerra@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> I am wondering here what is the case where code first is better?
>>>>>> 
>>>>> 
>>>>> I don't think it's wrong to share ideas as early as possible, and
>> after
>>>>> this discussion I think I am in favor of it too. I just meant that I
>>> don't
>>>>> think it's always necessarily the most productive discussion until
>> code
>>>>> exists sometimes, with the types of thing I am thinking of are almost
>>>>> entirely limited to cases where things might sound good to anyone on
>>> paper
>>>>> but in reality need a large amount of experiments conducted and
>>>>> observations collected to determine that something is actually worth
>>>>> doing,
>>>>> which I imagine is mainly things like reworking internals for
>>> performance
>>>>> improvements.
>>>>> 
>>>>> In the case of my combined proposal PR, I needed to prove that the
>>> thing I
>>>>> was working on was a good idea... and it wasn't directly. But I came
>> up
>>>>> with another idea during the course of experiment turned into
>> something
>>>>> compelling, so an initial proposal would have looked quite a lot
>>> different
>>>>> than what I ended up with. Once I had proven to myself that it was a
>>> good
>>>>> idea, then I was comfortable sharing with the wider community. I'm not
>>>>> certain how this would play out in an always proposal first model,
>> maybe
>>>>> the first proposal exists, I personally reject it after updating with
>>>>> experiment results show it's a bad idea, continue experimenting and
>>> raise
>>>>> a
>>>>> new one after the experiments start looking promising?
>>>>> 
>>>>> 
>>>>>> Let's not be naive this is very rare that a contributor will accept
>>> that
>>>>>> his work is to be thrown, usually devs takes coding as personal
>>> creation
>>>>>> and they get attached to it.
>>>>>> 
>>>>> 
>>>>> I agree, just because a handful of the committers have this attitude,
>> it
>>>>> isn't fair to expect the wider community to also, that's why I am in
>>> favor
>>>>> of formalizing the process.
>>>>> 
>>>>> Can you please explain what is overbearing ? what can be changed to
>> make
>>>>> it
>>>>>> easy ?
>>>>>> Most of the points are kind of the actual questions that you want to
>>>>>> address before hand anyway isn't ?
>>>>>> 
>>>>> 
>>>>> Sorry for the confusion, I said it's "not overbearing", I think it's
>>> fine.
>>>>> 
>>>>> What are the complaints ?
>>>>> 
>>>>> 
>>>>> Is this and other previous threads not a complaint about opening a
>> large
>>>>> PR
>>>>> without a proposal? :) I just mean that formalizing the process, even
>>> if a
>>>>> proposal has a reference PR opened with it near concurrently, could
>>>>> prevent
>>>>> these discussions from happening in the future because ground rules
>> have
>>>>> been set and we are all on the same page I guess.
>>>>> 
>>>>> I think we should also probably consider renaming the "design review"
>>>>> label
>>>>> to "proposal" or something to make it more clear that a PR is
>> associated
>>>>> with a proposal. It might also be worth considering using github
>>> projects
>>>>> for particularly large things that involve multiple follow up PRs,
>> but I
>>>>> haven't used them in much depth to know if they add anything.
>>>>> 
>>>>> It seems like we are all converging on agreement to do a github issue
>>>>> proposal for larger changes (I would vote for announcing them on the
>> dev
>>>>> list too for more visibility), so that design review is separated from
>>>>> code
>>>>> review. I guess my main concern was not wanting to discourage
>>>>> experimentation by walling it off behind mandatory discussions, but
>> the
>>>>> more I think about it,  it doesn't technically change much about this
>>>>> process, it just requires a more formal proposal to accompany any
>>>>> experiments that are shared as a PR.
>>>>> 
>>>>> 
>>>>> On Tue, Jan 8, 2019 at 12:28 PM Gian Merlino <gi...@apache.org> wrote:
>>>>> 
>>>>>> I think for us, choosing to use GitHub issues as discussion threads
>>> for
>>>>>> potential 'major' contributions would be a good idea, especially if
>> we
>>>>>> encourage people to start them before PRs show up. Definitely agree
>>> that
>>>>>> all contributors should go through the same process -- I couldn't
>> see
>>> it
>>>>>> working well any other way.
>>>>>> 
>>>>>> On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <jh...@apache.org>
>> wrote:
>>>>>> 
>>>>>>> Statically, yes, GitHub PRs are the same as GitHub cases. But
>>>>>> dynamically,
>>>>>>> they are different, because you can only log a PR when you have
>>>>> finished
>>>>>>> work.
>>>>>>> 
>>>>>>> A lot of other Apache projects use JIRA, so there is a clear
>>>>> distinction
>>>>>>> between cases and contributions. JIRA cases, especially when
>> logged
>>>>> early
>>>>>>> in the lifecycle of a contribution, become long-running
>> conversation
>>>>>>> threads with a lot of community participation. If the Druid chose
>> to
>>>>> do
>>>>>> so,
>>>>>>> GitHub cases could be the same.
>>>>>>> 
>>>>>>> Be careful that you do not treat “potential contributors” (by
>> which
>>> I
>>>>>>> presume you mean non-committers) differently from committers and
>> PMC
>>>>>>> members. Anyone starting a major piece of work should follow the
>>> same
>>>>>>> process. (Experienced committers probably have a somewhat better
>>> idea
>>>>>> what
>>>>>>> work will turn out to be “major”, so they get a little more
>> leeway.)
>>>>>>> 
>>>>>>> Julian
>>>>>>> 
>>>>>>> 
>>>>>>>> On Jan 7, 2019, at 12:10 PM, Gian Merlino <gi...@apache.org>
>>> wrote:
>>>>>>>> 
>>>>>>>> I don't think there's a need to raise issues for every change: a
>>>>> small
>>>>>>> bug
>>>>>>>> fix or doc fix should just go straight to PR. (GitHub PRs show
>> up
>>> as
>>>>>>> issues
>>>>>>>> in the issue-search UI/API, so it's not like this means the
>> patch
>>>>> has
>>>>>> no
>>>>>>>> corresponding issue -- in a sense the PR _is_ the issue.)
>>>>>>>> 
>>>>>>>> I do think it makes sense to encourage potential contributors to
>>>>> write
>>>>>> to
>>>>>>>> the dev list or raise an issue if they aren't sure if something
>>>>> would
>>>>>>> need
>>>>>>>> to go through a more heavy weight process.
>>>>>>>> 
>>>>>>>> Fwiw we do have a set of 'design review' criteria already (we
>> had
>>> a
>>>>>>>> discussion about this a couple years ago) at:
>>>>>>>> http://druid.io/community/#getting-your-changes-accepted. So we
>>>>>>> wouldn't be
>>>>>>>> starting from zero on defining that. We set it up back when we
>>> were
>>>>>>> trying
>>>>>>>> to _streamline_ our process -- we used to require two non-author
>>> +1s
>>>>>> for
>>>>>>>> _every_ change, even minor ones. The introduction of design
>> review
>>>>>>> criteria
>>>>>>>> was meant to classify which PRs need that level of review and
>>> which
>>>>>> ones
>>>>>>>> are minor and can be merged with less review. I do think it
>> helped
>>>>> with
>>>>>>>> getting minor PRs merged more quickly. The list of criteria is,
>>>>>>>> 
>>>>>>>> - Major architectural changes or API changes
>>>>>>>> - HTTP requests and responses (e. g. a new HTTP endpoint)
>>>>>>>> - Interfaces for extensions
>>>>>>>> - Server configuration (e. g. altering the behavior of a config
>>>>>> property)
>>>>>>>> - Emitted metrics
>>>>>>>> - Other major changes, judged by the discretion of Druid
>>> committers
>>>>>>>> 
>>>>>>>> Some of it is subjective, but it has been in place for a while,
>> so
>>>>> it's
>>>>>>> at
>>>>>>>> least something we are relatively familiar with.
>>>>>>>> 
>>>>>>>> On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <jh...@apache.org>
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Small contributions don’t need any design review, whereas large
>>>>>>>>> contributions need significant review. I don’t think we should
>>>>> require
>>>>>>> an
>>>>>>>>> additional step for those (many) small contributions. But who
>>>>> decides
>>>>>>>>> whether a contribution fits into the small or large category?
>>>>>>>>> 
>>>>>>>>> I think the solution is for authors to log a case (or send an
>>>>> email to
>>>>>>>>> dev) before they start work on any contribution. Then
>> committers
>>>>> can
>>>>>>>>> request a more heavy-weight process if they think it is needed.
>>>>>>>>> 
>>>>>>>>> Julian
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <gi...@apache.org>
>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> It sounds like splitting design from code review is a common
>>> theme
>>>>>> in a
>>>>>>>>> few
>>>>>>>>>> of the posts here. How does everyone feel about making a point
>>> of
>>>>>>>>>> encouraging design reviews to be done as issues, separate from
>>> the
>>>>>> pull
>>>>>>>>>> request, with the expectations that (1) the design review
>> issue
>>>>>>>>>> ("proposal") should generally appear somewhat _before_ the
>> pull
>>>>>>> request;
>>>>>>>>>> (2) pull requests should _not_ have design review happen on
>>> them,
>>>>>>> meaning
>>>>>>>>>> there should no longer be PRs with design review tags, and we
>>>>> should
>>>>>>> move
>>>>>>>>>> the design review approval process to the issue rather than
>> the
>>>>> PR.
>>>>>>>>>> 
>>>>>>>>>> For (1), even if we encourage design review discussions to
>> start
>>>>>>> before a
>>>>>>>>>> pull request appears, I don't see an issue with them running
>>>>>>> concurrently
>>>>>>>>>> for a while at some point.
>>>>>>>>>> 
>>>>>>>>>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <
>> jonwei@apache.org>
>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Thanks for raising these concerns!
>>>>>>>>>>> 
>>>>>>>>>>> My initial thoughts:
>>>>>>>>>>> - I agree that separation of design review and code-level
>>> review
>>>>> for
>>>>>>>>> major
>>>>>>>>>>> changes would be more efficient
>>>>>>>>>>> 
>>>>>>>>>>> - I agree that a clear, more formalized process for handling
>>>>> major
>>>>>>>>> changes
>>>>>>>>>>> would be helpful for contributors:
>>>>>>>>>>> - Define what is considered a major change
>>>>>>>>>>> - Define a standard proposal structure, KIP-style proposal
>>> format
>>>>>>>>> sounds
>>>>>>>>>>> good to me
>>>>>>>>>>> 
>>>>>>>>>>> - I think it's too rigid to have a policy of "no code at all
>>> with
>>>>>> the
>>>>>>>>>>> initial proposal"
>>>>>>>>>>> - Code samples can be useful references for understanding
>>> aspects
>>>>>> of a
>>>>>>>>>>> design
>>>>>>>>>>> - In some cases it's necessary to run experiments to fully
>>>>>> understand
>>>>>>> a
>>>>>>>>>>> problem and determine an appropriate design, or to determine
>>>>> whether
>>>>>>>>>>> something is even worth doing before committing to the work
>> of
>>>>>>> fleshing
>>>>>>>>> out
>>>>>>>>>>> a proposal, prototype code is a natural outcome of that and
>> I'm
>>>>> not
>>>>>>>>> against
>>>>>>>>>>> someone providing such code for reference
>>>>>>>>>>> - I tend to view design/code as things that are often
>> developed
>>>>>>>>>>> simultaneously in an intertwined way
>>>>>>>>>>> 
>>>>>>>>>>>> Let's not be naive this is very rare that a contributor will
>>>>> accept
>>>>>>>>> that
>>>>>>>>>>> his work is to be thrown, usually devs takes coding as
>> personal
>>>>>>> creation
>>>>>>>>>>> and they get attached to it.
>>>>>>>>>>> 
>>>>>>>>>>> If we have a clear review process that emphasizes the need
>> for
>>>>> early
>>>>>>>>>>> consensus building, with separate design and code review,
>> then
>>> I
>>>>>> feel
>>>>>>>>> we've
>>>>>>>>>>> done enough and don't need a hard rule against having some
>> code
>>>>>> linked
>>>>>>>>> with
>>>>>>>>>>> the initial proposal. If a potential contributor then still
>>>>> wants to
>>>>>>> go
>>>>>>>>>>> ahead and write a lot of code that may be rejected or change
>>>>>>>>> significantly,
>>>>>>>>>>> the risks were made clear.
>>>>>>>>>>> 
>>>>>>>>>>>> Once code is written hard to think abstract.
>>>>>>>>>>> 
>>>>>>>>>>> I can see the validity of the concern, but I personally don't
>>>>> see it
>>>>>>> as
>>>>>>>>> a
>>>>>>>>>>> huge risk. My impression from the Druid PR reviews I've seen
>> is
>>>>> that
>>>>>>> our
>>>>>>>>>>> reviewers are able to keep abstract design vs. implementation
>>>>>> details
>>>>>>>>>>> separate and consider alternate designs when reviewing.
>>>>>>>>>>> 
>>>>>>>>>>> To summarize I think it's probably enough to have a policy
>>> along
>>>>> the
>>>>>>>>> lines
>>>>>>>>>>> of:
>>>>>>>>>>> - Create more formalized guidelines for proposals and what
>>>>> changes
>>>>>>>>> require
>>>>>>>>>>> proposals
>>>>>>>>>>> - Separate design and code review for major changes, with
>>> design
>>>>>>> review
>>>>>>>>>>> first, code-level review after reaching consensus on the
>>> design.
>>>>>>>>>>> - Code before the design review is completed is just for
>>>>> reference,
>>>>>>> not
>>>>>>>>>>> regarded as a candidate for review/merging.
>>>>>>>>>>> 
>>>>>>>>>>> - Jon
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
>>>>>>>>> slim.bouguerra@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <
>>>>> clint.wylie@imply.io>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> I am definitely biased in this matter as an owner of
>> another
>>>>> large
>>>>>>> PR
>>>>>>>>>>>> that
>>>>>>>>>>>>> wasn't preceded by a direct proposal or dev list
>> discussion,
>>>>> and
>>>>>> in
>>>>>>>>>>>> general
>>>>>>>>>>>>> I agree that proposal first is usually better, but I think
>> in
>>>>> some
>>>>>>>>>>> rarer
>>>>>>>>>>>>> cases approaching a problem code first *is* the most
>>>>> appropriate
>>>>>> way
>>>>>>>>> to
>>>>>>>>>>>>> have a discussion.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> I am wondering here what is the case where code first is
>>> better?
>>>>>>>>>>>> In general when you are writing code you have an idea about
>>> what
>>>>>> you
>>>>>>>>> want
>>>>>>>>>>>> to change, why you want to change and why you want to change
>>> it.
>>>>>>>>>>>> I do not see what is wrong with sharing this primitive ideas
>>> and
>>>>>>>>> thoughts
>>>>>>>>>>>> as an abstract proposal (at least to avoid overlapping)
>>>>>>>>>>>> 
>>>>>>>>>>>> I see nothing wrong with it so long as the author
>>>>>>>>>>>>> accepts that the PR is treated as a combined proposal and
>>>>> proof of
>>>>>>>>>>>> concept,
>>>>>>>>>>>>> and fair game to be radically changed via discussion or
>> even
>>>>>>> rejected,
>>>>>>>>>>>>> which sounds like Gian's attitude on the matter and is mine
>>> as
>>>>>> well
>>>>>>>>>>> with
>>>>>>>>>>>> my
>>>>>>>>>>>>> compression stuff.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Let's not be naive this is very rare that a contributor will
>>>>> accept
>>>>>>>>> that
>>>>>>>>>>>> his work is to be thrown, usually devs takes coding as
>>> personal
>>>>>>>>> creation
>>>>>>>>>>>> and they get attached to it.
>>>>>>>>>>>> To my point you can take a look on some old issue in the
>> Druid
>>>>>> forum
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
>>>>>>>>>>>> and am sure other communities have similar problems.
>>>>>>>>>>>> So leaving the door open to some side cases is not a good
>> idea
>>>>> in
>>>>>> my
>>>>>>>>>>>> opinion and will lead to similar issue in the future.
>>>>>>>>>>>> 
>>>>>>>>>>>> This seems to me especially likely to happen in cases
>>>>>>>>>>>>> where an approach still needs proven to be a viable idea
>> *to
>>>>> the
>>>>>>>>>>> author*,
>>>>>>>>>>>>> so that a much more productive discussion can be had in the
>>>>> first
>>>>>>>>>>> place.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I think there is a trade off, I don't think we want to
>>>>> discourage
>>>>>>>>>>>>> experimentation by walling it off behind mandatory
>>> discussions
>>>>>>> before
>>>>>>>>>>> it
>>>>>>>>>>>>> can even start, but I do think formalizing the process for
>>>>> large
>>>>>>>>>>> changes
>>>>>>>>>>>> is
>>>>>>>>>>>>> a good thing, especially since we probably can't expect the
>>>>> wider
>>>>>>>>>>>> community
>>>>>>>>>>>>> to have the same attitude towards a large PR getting
>>> discarded
>>>>> as
>>>>>> a
>>>>>>>>>>>>> committer might. I think the Kafka approach is reasonable,
>> a
>>>>> bit
>>>>>>> more
>>>>>>>>>>>>> formal than our design review process but not overbearing.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Can you please explain what is overbearing ? what can be
>>>>> changed to
>>>>>>>>> make
>>>>>>>>>>> it
>>>>>>>>>>>> easy ?
>>>>>>>>>>>> Most of the points are kind of the actual questions that you
>>>>> want
>>>>>> to
>>>>>>>>>>>> address before hand anyway isn't ?
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> Going code first
>>>>>>>>>>>>> should be in general discouraged, but when it does happen,
>> it
>>>>>> seems
>>>>>>>>>>> like
>>>>>>>>>>>>> opening DIP/an issue/starting a mailing list thread or
>>>>> whatever we
>>>>>>> go
>>>>>>>>>>>> with
>>>>>>>>>>>>> to have a more high level design discussion alongside the
>>>>>> reference
>>>>>>> PR
>>>>>>>>>>>>> could alleviate some of these complaints?
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> What are the complaints ?
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> +1 for "DIP" heh, I think making
>>>>>>>>>>>>> them in the form of github issues is probably appropriate,
>>>>> with a
>>>>>>> dev
>>>>>>>>>>>> list
>>>>>>>>>>>>> thread to announce them perhaps?
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> I think  github issue with [Proposal] header like
>>>>>>>>>>>> https://github.com/apache/incubator-druid/issues/4349 is
>> good
>>>>> to
>>>>>> me,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <
>>>>> bslim@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks everyone for interacting with this thread.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The fact that i, Roman, Jihoon  and others in the past (FJ
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ
>>>>> )
>>>>>>>>>>>>>> raised this point indicates that PRs without a proposal
>> are
>>>>>> indeed
>>>>>>> an
>>>>>>>>>>>>> issue
>>>>>>>>>>>>>> and we need to solve it.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Something Similar to KIP maybe called DIPs is fine with
>> me.
>>>>>>>>>>>>>> What i strive to see is the following:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> [Step 1] formalize what is the kind of work that needs a
>>>>> formal
>>>>>>>>>>>>> Proposal, I
>>>>>>>>>>>>>> think Roman and Jihoon has already covered that pretty
>> well.
>>>>> am
>>>>>> +1
>>>>>>> on
>>>>>>>>>>>>> that.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I am strongly in favor of the separation of Proposal
>> Review
>>>>> and
>>>>>>>>>>> (later)
>>>>>>>>>>>>>> Code review PRs. My  main reasons:
>>>>>>>>>>>>>> Most importantly code reviewing will introduce lot of
>> noise
>>>>> and
>>>>>>> will
>>>>>>>>>>>>>> ultimately make  the GitHub page unreadable.
>>>>>>>>>>>>>> Avoid overlapping of work.
>>>>>>>>>>>>>> Once code is written hard to think abstract.
>>>>>>>>>>>>>> Separate page for Design review later can always be used
>> it
>>>>> as a
>>>>>>>>>>> Design
>>>>>>>>>>>>>> document that is readable and code free-ish.
>>>>>>>>>>>>>> As i said the goal of this first round is to see if the
>>>>> community
>>>>>>>>>>> agree
>>>>>>>>>>>>>> about such change, then make the process of design more
>>>>> inclusive
>>>>>>>>>>> thus
>>>>>>>>>>>>>> other contributors can submit a counter proposals.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> [Step 2] IF everybody agree about that point Step 2 is to
>>>>> define
>>>>>>>>>>> which
>>>>>>>>>>>>>> medium is used to Publish a primitive form of a CODE FREE
>>>>>> Abstract
>>>>>>>>>>>>> Proposal
>>>>>>>>>>>>>> containing at least the following bullet points.
>>>>>>>>>>>>>> - The problem description and motivation
>>>>>>>>>>>>>> - Overview of the proposed change
>>>>>>>>>>>>>> - Operational impact (compatibility/ plans to upgrades)
>>> public
>>>>>> API
>>>>>>>>>>>>> changes,
>>>>>>>>>>>>>> configuration changes, algorithm, and so on
>>>>>>>>>>>>>> - Expected benefits and drawbacks
>>>>>>>>>>>>>> - Rationale and alternatives
>>>>>>>>>>>>>> - Estimate Time to Deliver if possible.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The way i think this can be is a Github issue where member
>>> of
>>>>> the
>>>>>>>>>>>>> community
>>>>>>>>>>>>>> will interact via comments and the author will be updating
>>> the
>>>>>>>>>>>>> description
>>>>>>>>>>>>>> in the light of comments provided by the community.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> During and near the end of the design discussions the
>>> author/s
>>>>>> can
>>>>>>>>>>>> start
>>>>>>>>>>>>>> writing POCs to help guide the review process this
>> naturally
>>>>> will
>>>>>>> be
>>>>>>>>>>> a
>>>>>>>>>>>>> Pull
>>>>>>>>>>>>>> request with actual code.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> *Now the most important thing is that we need to agree
>> that
>>>>> any
>>>>>>> work
>>>>>>>>>>>> that
>>>>>>>>>>>>>> does not align with this formal process will be ignored
>> and
>>>>> the
>>>>>>>>>>> author
>>>>>>>>>>>>> will
>>>>>>>>>>>>>> be asked to start with a DIP*
>>>>>>>>>>>>>> *That is what i meant with  “If it didn’t happen on the
>>>>> mailing
>>>>>>> list,
>>>>>>>>>>>> it
>>>>>>>>>>>>>> didn’t happen.”*
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks and happy coding!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <
>>> gian@apache.org>
>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> One of the advantages I see with a more formal process is
>>>>> (like
>>>>>>>>>>> Kafka
>>>>>>>>>>>>>> KIPs)
>>>>>>>>>>>>>>> is that it levels the playing field a bit and sets some
>>>>> ground
>>>>>>>>>>> rules
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> working together. In a way it can help encourage
>>>>> contributions
>>>>>> by
>>>>>>>>>>>>> making
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> clear what is expected of potential contributors.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> We have a design review process today that is not as
>> formal
>>>>> as
>>>>>>>>>>> KIPs,
>>>>>>>>>>>>> but
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> somewhat heavier than the one you describe. Maybe we
>> could
>>>>> tweak
>>>>>>>>>>> our
>>>>>>>>>>>>>>> current one by starting to do design reviews separately
>>> from
>>>>>> PRs.
>>>>>>>>>>>> i.e.,
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> anything that meets our 'design review' criteria, do that
>>> on
>>>>> the
>>>>>>>>>>> dev
>>>>>>>>>>>>> list
>>>>>>>>>>>>>>> or in a separate issue, and keep the PR focused on
>>> code-level
>>>>>>>>>>> stuff.
>>>>>>>>>>>>> That
>>>>>>>>>>>>>>> way we don't end up trying to do both at once. And it
>> makes
>>>>> it
>>>>>>>>>>> easier
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> start talking about design before the code is ready,
>> which
>>>>> would
>>>>>>> be
>>>>>>>>>>>>>> better.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <
>>> jhyde@apache.org
>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> It’s really hard to say no to a contribution when
>> someone
>>>>> has
>>>>>> put
>>>>>>>>>>>> in
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> significant amount of work.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The following approach is simple and works really well:
>>>>> Before
>>>>>>>>>>> you
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>> work, log a case, describing the problem. When you have
>>> some
>>>>>>>>>>> ideas
>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>> design, add those to the case. When you have a code
>>> branch,
>>>>> add
>>>>>>>>>>> its
>>>>>>>>>>>>> URL
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> the case. And so forth. At any point in the proceedings,
>>>>> people
>>>>>>>>>>> can
>>>>>>>>>>>>>> chime
>>>>>>>>>>>>>>>> in with their opinions.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> In my opinion, a formal “design review” process is not
>>>>>> necessary.
>>>>>>>>>>>>> Just
>>>>>>>>>>>>>>>> build consensus iteratively, by starting the
>> conversation
>>>>> early
>>>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> process.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Julian
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <
>>> gian@apache.org
>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> In this particular case: please consider the PR as a
>>>>> proposal.
>>>>>>>>>>>>> Don't
>>>>>>>>>>>>>>> feel
>>>>>>>>>>>>>>>>> like just because there is code there that takes a
>>> certain
>>>>>>>>>>>>> approach,
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> the approach is somehow sacred. I had to implement
>>>>> something
>>>>>> to
>>>>>>>>>>>>>>>> crystallize
>>>>>>>>>>>>>>>>> my own thinking about how the problem could be
>>> approached.
>>>>> I
>>>>>>>>>>>> won't
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> disappointed if, as a community, we decide a different
>>>>>>>>>>> direction
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>> and the code all gets thrown away. That's one of the
>>>>> reasons
>>>>>>>>>>>> that I
>>>>>>>>>>>>>>>> removed
>>>>>>>>>>>>>>>>> the 0.14.0 milestone that was added to the patch. (I
>>> don't
>>>>>> want
>>>>>>>>>>>> to
>>>>>>>>>>>>>> rush
>>>>>>>>>>>>>>>> it,
>>>>>>>>>>>>>>>>> nor do I think that's a good idea.)
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> In general: Sounds like we could do with some more
>>>>>>>>>>> formalization
>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>> what a proposal looks like, which sorts of changes need
>>>>> one,
>>>>>>>>>>> and
>>>>>>>>>>>>> when
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the dev cycle it is appropriate. FWIW I think Kafka's
>>>>> process
>>>>>>>>>>> is
>>>>>>>>>>>>> more
>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> less fine, and would be okay with adopting it for Druid
>>> if
>>>>>>>>>>> people
>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>>>>> Right now our standards for what requires a "design
>>> review"
>>>>>> are
>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>> similar to the Kafka community standards for what
>>> requires
>>>>> a
>>>>>>>>>>> KIP,
>>>>>>>>>>>>> so
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> have some familiarity with those concepts. However we
>>> don't
>>>>>>>>>>>>> separate
>>>>>>>>>>>>>> PR
>>>>>>>>>>>>>>>>> review and proposal discussion as strictly as they do,
>>>>> which
>>>>>>>>>>>> seems
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> the foundation for the feeling of exclusion that is
>> being
>>>>> felt
>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Separately: I just redid the description on
>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794 to
>>> be
>>>>>> more
>>>>>>>>>>>>>>>> proposal-y.
>>>>>>>>>>>>>>>>> I followed the KIP style:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>> Please refresh the page and see if it looks more
>> useful.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Gian
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <
>>>>> jhyde@apache.org
>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Slim,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I agree with your points that offline development is
>> bad
>>>>> for
>>>>>>>>>>>>>>> community.
>>>>>>>>>>>>>>>>>> But I don’t think you need much mentor help. You have
>>>>> raised
>>>>>>>>>>>> valid
>>>>>>>>>>>>>>>> issues
>>>>>>>>>>>>>>>>>> and the Druid community needs to decide what its
>>>>> development
>>>>>>>>>>>>>> practices
>>>>>>>>>>>>>>>>>> should be.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Julian
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
>>>>>>>>>>> bslim@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hello everyone and hope you all have very good
>>> holidays.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> First, this email is not directed on the author or
>> the
>>> PR
>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794
>>> it
>>>>>>>>>>> self,
>>>>>>>>>>>>> but
>>>>>>>>>>>>>> i
>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>> this PR as a perfect example.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> One of the foundation of Apache Way or what i would
>>>>> simply
>>>>>>>>>>> call
>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>> community driven development is that "Technical
>>> decisions
>>>>>> are
>>>>>>>>>>>>>>>> discussed,
>>>>>>>>>>>>>>>>>>> decided, and archived publicly.
>>>>>>>>>>>>>>>>>>> developpement"
>>>>>>>>>>>>>>>>>>> Which means that big technical  changes such as the
>> one
>>>>>>>>>>> brought
>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>> #/6794
>>>>>>>>>>>>>>>>>>> should have started as a proposal and round of
>>>>> discussions
>>>>>>>>>>>> about
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> major
>>>>>>>>>>>>>>>>>>> changes designs not as 11K line of code.
>>>>>>>>>>>>>>>>>>> I believe such openness will promote a lot of good
>>>>> benefits
>>>>>>>>>>>> such
>>>>>>>>>>>>>> as:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> - ensures community health and growth.
>>>>>>>>>>>>>>>>>>> - ensures everyone can participate not only the
>> authors
>>>>> and
>>>>>>>>>>> his
>>>>>>>>>>>>>>>>>> co-workers.
>>>>>>>>>>>>>>>>>>> - ensures that the project is driven by the community
>>> and
>>>>>>>>>>> not a
>>>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>>> company or an individual.
>>>>>>>>>>>>>>>>>>> - ensures that there is consensus (not saying 100%
>>>>>>>>>>> agreement;)
>>>>>>>>>>>>>>> however
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> means that all individuals will accept the current
>>>>> progress
>>>>>>>>>>> on
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> project
>>>>>>>>>>>>>>>>>>> until some better proposal is put forth.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Personally such BIG offline PR makes me feel excluded
>>> and
>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>> give
>>>>>>>>>>>>>>>>>> me a
>>>>>>>>>>>>>>>>>>> sense that i belong to  a community at all.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> To prevent such off list development i think as a
>> Druid
>>>>>>>>>>>> Community
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>> to stick to the apache way “If it didn’t happen on
>> the
>>>>>>>>>>> mailing
>>>>>>>>>>>>>> list,
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> didn’t happen.”
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I would appreciate if some of the Apache mentor help
>>> with
>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>> 
>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>> dev-unsubscribe@druid.apache.org
>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>> dev-help@druid.apache.org
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>> 
>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>> dev-unsubscribe@druid.apache.org
>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>> dev-help@druid.apache.org
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> 
>>>>>>>>>>>> B-Slim
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>> _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org

Re: Off list major development

[Gian Merlino <gi...@apache.org>]

> > What do community members think about also making a requirement that
> > "Design Review" PRs and proposals are reviewed by at least two people
with
> > different affiliation?
> This seems pretty reasonable to me. I haven't found anything in Apache
> voting procedure docs (https://www.apache.org/foundation/voting.html) that
> seems to explicitly forbid something like this yet at least.

On the other hand, my understanding of the Apache Way would mean that this
kind of rule doesn't make sense. In particular from
https://www.apache.org/foundation/how-it-works.html: "We firmly believe in
hats. Your role at the ASF is one assigned to you personally, and is
bestowed on you by your peers. It is not tied to your job or current
employer or company." That sentiment seems incompatible with making
explicit rules about organizational diversity in voting. I have also heard
a few people say things like: people are supposed to represent themselves,
not their employers.

Obviously, though, people's actions and opinions are influenced by their
employer. IMO, a better way to approach a potential problem there is that
if people from a particular organization end up behaving abusively, then
PMC members from other organizations (or in extreme situations the Apache
board itself) should tell them to knock it off. And then do more serious
things if the inappropriate behavior continues. This kind

The other thing you brought up, promoting more shared ownership and
awareness, I am hopeful that separating proposals from PRs will help with
that. One reason is that it takes much less time to understand a well
written proposal than it takes to understand a PR. Code is very clear but
it is also very verbose and takes a while to read and understand. So in
theory changing how we operate, in this way, should promote more
understanding of more people across the code base.

On Thu, Jan 10, 2019 at 1:34 AM Clint Wylie <cl...@imply.io> wrote:

> >
> > What do community members think about also making a requirement that
> > "Design Review" PRs and proposals are reviewed by at least two people
> with
> > different affiliation?
>
>
> This seems pretty reasonable to me. I haven't found anything in Apache
> voting procedure docs (https://www.apache.org/foundation/voting.html) that
> seems to explicitly forbid something like this yet at least.
>
> On Wed, Jan 9, 2019 at 8:52 PM Roman Leventov <le...@gmail.com>
> wrote:
>
> > What do community members think about also making a requirement that
> > "Design Review" PRs and proposals are reviewed by at least two people
> with
> > different affiliation? IMO it's a good idea, because it ensures that
> > different interests are taken into account. Also it pushes people to
> engage
> > with work done in other parts of Druid, improving shared code owning and
> > awareness.
> >
> > Or it's against the Apache Way because it's assumed that there are no
> > company boundaries within the community?
> >
> > On Thu, 10 Jan 2019 at 11:45, Roman Leventov <le...@gmail.com>
> > wrote:
> >
> > > I see two important reasons why it makes sense to file an issue and
> > > probably announce it in the mailing list, before writing a lot of code,
> > > despite not having a clear picture of what it will be and any
> performance
> > > data:
> > >  1) somebody could already work on this problem privately in parallel,
> it
> > > allows to avoid clash in people's works
> > >  2) some people could quickly think about the problem field and share
> > > high-level ideas that could wildly change the direction in which the
> > author
> > > (the person who is going to write code) will move in his work from
> early
> > on.
> > >
> > > Jihoon in
> > >
> >
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> > and
> > > later Gian in this thread suggested that _every_ piece of work that
> > should
> > > be labelled as "Design Review" according to the current rules should be
> > > accompanied by an issue. I don't agree with this, there are some PRs as
> > > small as a few dozens of lines of code, that add some configuration
> > > parameter and therefore should be labelled "Design Review". I don't
> > thing a
> > > separate proposal issue is needed for them, and even for a little
> larger
> > > PRs too.
> > >
> > > For the same reason, I also don't see the point of renaming "Design
> > > Review" into "Proposal", as well as separating "Design Review" into
> > > "Proposal" and something like "API Review". I think a single "Design
> > > Review" tag handles it well.
> > >
> > > Gian mentioned an idea that PRs that follow a "Design Review" proposal
> > > issue shouldn't be "Design Review" themselves. I don't agree with
> this, I
> > > think that actual code and performance data are important inputs that
> > > should be re-evaluated at least by two people. I even think that it's
> > very
> > > desirable that at least two people read _each line of production code_
> in
> > > large PRs, although it's not what was done historically in Druid,
> because
> > > large bodies of newly added code, with whole new classes and subsystems
> > > added, are also coincidentally tested worse than already existing
> classes
> > > and subsystems, including in production. It seems to me that those huge
> > > code influxes is a major source of bugs, that could later take years to
> > > squeeze out from the codebase.
> > >
> > > On Wed, 9 Jan 2019 at 08:24, Clint Wylie <cl...@imply.io> wrote:
> > >
> > >> Apologies for the delayed response.
> > >>
> > >> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > slim.bouguerra@gmail.com>
> > >> wrote:
> > >>
> > >> > I am wondering here what is the case where code first is better?
> > >> >
> > >>
> > >> I don't think it's wrong to share ideas as early as possible, and
> after
> > >> this discussion I think I am in favor of it too. I just meant that I
> > don't
> > >> think it's always necessarily the most productive discussion until
> code
> > >> exists sometimes, with the types of thing I am thinking of are almost
> > >> entirely limited to cases where things might sound good to anyone on
> > paper
> > >> but in reality need a large amount of experiments conducted and
> > >> observations collected to determine that something is actually worth
> > >> doing,
> > >> which I imagine is mainly things like reworking internals for
> > performance
> > >> improvements.
> > >>
> > >> In the case of my combined proposal PR, I needed to prove that the
> > thing I
> > >> was working on was a good idea... and it wasn't directly. But I came
> up
> > >> with another idea during the course of experiment turned into
> something
> > >> compelling, so an initial proposal would have looked quite a lot
> > different
> > >> than what I ended up with. Once I had proven to myself that it was a
> > good
> > >> idea, then I was comfortable sharing with the wider community. I'm not
> > >> certain how this would play out in an always proposal first model,
> maybe
> > >> the first proposal exists, I personally reject it after updating with
> > >> experiment results show it's a bad idea, continue experimenting and
> > raise
> > >> a
> > >> new one after the experiments start looking promising?
> > >>
> > >>
> > >> > Let's not be naive this is very rare that a contributor will accept
> > that
> > >> > his work is to be thrown, usually devs takes coding as personal
> > creation
> > >> > and they get attached to it.
> > >> >
> > >>
> > >> I agree, just because a handful of the committers have this attitude,
> it
> > >> isn't fair to expect the wider community to also, that's why I am in
> > favor
> > >> of formalizing the process.
> > >>
> > >> Can you please explain what is overbearing ? what can be changed to
> make
> > >> it
> > >> > easy ?
> > >> > Most of the points are kind of the actual questions that you want to
> > >> > address before hand anyway isn't ?
> > >> >
> > >>
> > >> Sorry for the confusion, I said it's "not overbearing", I think it's
> > fine.
> > >>
> > >> What are the complaints ?
> > >>
> > >>
> > >> Is this and other previous threads not a complaint about opening a
> large
> > >> PR
> > >> without a proposal? :) I just mean that formalizing the process, even
> > if a
> > >> proposal has a reference PR opened with it near concurrently, could
> > >> prevent
> > >> these discussions from happening in the future because ground rules
> have
> > >> been set and we are all on the same page I guess.
> > >>
> > >> I think we should also probably consider renaming the "design review"
> > >> label
> > >> to "proposal" or something to make it more clear that a PR is
> associated
> > >> with a proposal. It might also be worth considering using github
> > projects
> > >> for particularly large things that involve multiple follow up PRs,
> but I
> > >> haven't used them in much depth to know if they add anything.
> > >>
> > >> It seems like we are all converging on agreement to do a github issue
> > >> proposal for larger changes (I would vote for announcing them on the
> dev
> > >> list too for more visibility), so that design review is separated from
> > >> code
> > >> review. I guess my main concern was not wanting to discourage
> > >> experimentation by walling it off behind mandatory discussions, but
> the
> > >> more I think about it,  it doesn't technically change much about this
> > >> process, it just requires a more formal proposal to accompany any
> > >> experiments that are shared as a PR.
> > >>
> > >>
> > >> On Tue, Jan 8, 2019 at 12:28 PM Gian Merlino <gi...@apache.org> wrote:
> > >>
> > >> > I think for us, choosing to use GitHub issues as discussion threads
> > for
> > >> > potential 'major' contributions would be a good idea, especially if
> we
> > >> > encourage people to start them before PRs show up. Definitely agree
> > that
> > >> > all contributors should go through the same process -- I couldn't
> see
> > it
> > >> > working well any other way.
> > >> >
> > >> > On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <jh...@apache.org>
> wrote:
> > >> >
> > >> > > Statically, yes, GitHub PRs are the same as GitHub cases. But
> > >> > dynamically,
> > >> > > they are different, because you can only log a PR when you have
> > >> finished
> > >> > > work.
> > >> > >
> > >> > > A lot of other Apache projects use JIRA, so there is a clear
> > >> distinction
> > >> > > between cases and contributions. JIRA cases, especially when
> logged
> > >> early
> > >> > > in the lifecycle of a contribution, become long-running
> conversation
> > >> > > threads with a lot of community participation. If the Druid chose
> to
> > >> do
> > >> > so,
> > >> > > GitHub cases could be the same.
> > >> > >
> > >> > > Be careful that you do not treat “potential contributors” (by
> which
> > I
> > >> > > presume you mean non-committers) differently from committers and
> PMC
> > >> > > members. Anyone starting a major piece of work should follow the
> > same
> > >> > > process. (Experienced committers probably have a somewhat better
> > idea
> > >> > what
> > >> > > work will turn out to be “major”, so they get a little more
> leeway.)
> > >> > >
> > >> > > Julian
> > >> > >
> > >> > >
> > >> > > > On Jan 7, 2019, at 12:10 PM, Gian Merlino <gi...@apache.org>
> > wrote:
> > >> > > >
> > >> > > > I don't think there's a need to raise issues for every change: a
> > >> small
> > >> > > bug
> > >> > > > fix or doc fix should just go straight to PR. (GitHub PRs show
> up
> > as
> > >> > > issues
> > >> > > > in the issue-search UI/API, so it's not like this means the
> patch
> > >> has
> > >> > no
> > >> > > > corresponding issue -- in a sense the PR _is_ the issue.)
> > >> > > >
> > >> > > > I do think it makes sense to encourage potential contributors to
> > >> write
> > >> > to
> > >> > > > the dev list or raise an issue if they aren't sure if something
> > >> would
> > >> > > need
> > >> > > > to go through a more heavy weight process.
> > >> > > >
> > >> > > > Fwiw we do have a set of 'design review' criteria already (we
> had
> > a
> > >> > > > discussion about this a couple years ago) at:
> > >> > > > http://druid.io/community/#getting-your-changes-accepted. So we
> > >> > > wouldn't be
> > >> > > > starting from zero on defining that. We set it up back when we
> > were
> > >> > > trying
> > >> > > > to _streamline_ our process -- we used to require two non-author
> > +1s
> > >> > for
> > >> > > > _every_ change, even minor ones. The introduction of design
> review
> > >> > > criteria
> > >> > > > was meant to classify which PRs need that level of review and
> > which
> > >> > ones
> > >> > > > are minor and can be merged with less review. I do think it
> helped
> > >> with
> > >> > > > getting minor PRs merged more quickly. The list of criteria is,
> > >> > > >
> > >> > > > - Major architectural changes or API changes
> > >> > > > - HTTP requests and responses (e. g. a new HTTP endpoint)
> > >> > > > - Interfaces for extensions
> > >> > > > - Server configuration (e. g. altering the behavior of a config
> > >> > property)
> > >> > > > - Emitted metrics
> > >> > > > - Other major changes, judged by the discretion of Druid
> > committers
> > >> > > >
> > >> > > > Some of it is subjective, but it has been in place for a while,
> so
> > >> it's
> > >> > > at
> > >> > > > least something we are relatively familiar with.
> > >> > > >
> > >> > > > On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <jh...@apache.org>
> > >> wrote:
> > >> > > >
> > >> > > >> Small contributions don’t need any design review, whereas large
> > >> > > >> contributions need significant review. I don’t think we should
> > >> require
> > >> > > an
> > >> > > >> additional step for those (many) small contributions. But who
> > >> decides
> > >> > > >> whether a contribution fits into the small or large category?
> > >> > > >>
> > >> > > >> I think the solution is for authors to log a case (or send an
> > >> email to
> > >> > > >> dev) before they start work on any contribution. Then
> committers
> > >> can
> > >> > > >> request a more heavy-weight process if they think it is needed.
> > >> > > >>
> > >> > > >> Julian
> > >> > > >>
> > >> > > >>
> > >> > > >>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <gi...@apache.org>
> > >> wrote:
> > >> > > >>>
> > >> > > >>> It sounds like splitting design from code review is a common
> > theme
> > >> > in a
> > >> > > >> few
> > >> > > >>> of the posts here. How does everyone feel about making a point
> > of
> > >> > > >>> encouraging design reviews to be done as issues, separate from
> > the
> > >> > pull
> > >> > > >>> request, with the expectations that (1) the design review
> issue
> > >> > > >>> ("proposal") should generally appear somewhat _before_ the
> pull
> > >> > > request;
> > >> > > >>> (2) pull requests should _not_ have design review happen on
> > them,
> > >> > > meaning
> > >> > > >>> there should no longer be PRs with design review tags, and we
> > >> should
> > >> > > move
> > >> > > >>> the design review approval process to the issue rather than
> the
> > >> PR.
> > >> > > >>>
> > >> > > >>> For (1), even if we encourage design review discussions to
> start
> > >> > > before a
> > >> > > >>> pull request appears, I don't see an issue with them running
> > >> > > concurrently
> > >> > > >>> for a while at some point.
> > >> > > >>>
> > >> > > >>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <
> jonwei@apache.org>
> > >> > wrote:
> > >> > > >>>
> > >> > > >>>> Thanks for raising these concerns!
> > >> > > >>>>
> > >> > > >>>> My initial thoughts:
> > >> > > >>>> - I agree that separation of design review and code-level
> > review
> > >> for
> > >> > > >> major
> > >> > > >>>> changes would be more efficient
> > >> > > >>>>
> > >> > > >>>> - I agree that a clear, more formalized process for handling
> > >> major
> > >> > > >> changes
> > >> > > >>>> would be helpful for contributors:
> > >> > > >>>> - Define what is considered a major change
> > >> > > >>>> - Define a standard proposal structure, KIP-style proposal
> > format
> > >> > > >> sounds
> > >> > > >>>> good to me
> > >> > > >>>>
> > >> > > >>>> - I think it's too rigid to have a policy of "no code at all
> > with
> > >> > the
> > >> > > >>>> initial proposal"
> > >> > > >>>> - Code samples can be useful references for understanding
> > aspects
> > >> > of a
> > >> > > >>>> design
> > >> > > >>>> - In some cases it's necessary to run experiments to fully
> > >> > understand
> > >> > > a
> > >> > > >>>> problem and determine an appropriate design, or to determine
> > >> whether
> > >> > > >>>> something is even worth doing before committing to the work
> of
> > >> > > fleshing
> > >> > > >> out
> > >> > > >>>> a proposal, prototype code is a natural outcome of that and
> I'm
> > >> not
> > >> > > >> against
> > >> > > >>>> someone providing such code for reference
> > >> > > >>>> - I tend to view design/code as things that are often
> developed
> > >> > > >>>> simultaneously in an intertwined way
> > >> > > >>>>
> > >> > > >>>>> Let's not be naive this is very rare that a contributor will
> > >> accept
> > >> > > >> that
> > >> > > >>>> his work is to be thrown, usually devs takes coding as
> personal
> > >> > > creation
> > >> > > >>>> and they get attached to it.
> > >> > > >>>>
> > >> > > >>>> If we have a clear review process that emphasizes the need
> for
> > >> early
> > >> > > >>>> consensus building, with separate design and code review,
> then
> > I
> > >> > feel
> > >> > > >> we've
> > >> > > >>>> done enough and don't need a hard rule against having some
> code
> > >> > linked
> > >> > > >> with
> > >> > > >>>> the initial proposal. If a potential contributor then still
> > >> wants to
> > >> > > go
> > >> > > >>>> ahead and write a lot of code that may be rejected or change
> > >> > > >> significantly,
> > >> > > >>>> the risks were made clear.
> > >> > > >>>>
> > >> > > >>>>> Once code is written hard to think abstract.
> > >> > > >>>>
> > >> > > >>>> I can see the validity of the concern, but I personally don't
> > >> see it
> > >> > > as
> > >> > > >> a
> > >> > > >>>> huge risk. My impression from the Druid PR reviews I've seen
> is
> > >> that
> > >> > > our
> > >> > > >>>> reviewers are able to keep abstract design vs. implementation
> > >> > details
> > >> > > >>>> separate and consider alternate designs when reviewing.
> > >> > > >>>>
> > >> > > >>>> To summarize I think it's probably enough to have a policy
> > along
> > >> the
> > >> > > >> lines
> > >> > > >>>> of:
> > >> > > >>>> - Create more formalized guidelines for proposals and what
> > >> changes
> > >> > > >> require
> > >> > > >>>> proposals
> > >> > > >>>> - Separate design and code review for major changes, with
> > design
> > >> > > review
> > >> > > >>>> first, code-level review after reaching consensus on the
> > design.
> > >> > > >>>> - Code before the design review is completed is just for
> > >> reference,
> > >> > > not
> > >> > > >>>> regarded as a candidate for review/merging.
> > >> > > >>>>
> > >> > > >>>> - Jon
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > >> > > >> slim.bouguerra@gmail.com>
> > >> > > >>>> wrote:
> > >> > > >>>>
> > >> > > >>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <
> > >> clint.wylie@imply.io>
> > >> > > >>>> wrote:
> > >> > > >>>>>
> > >> > > >>>>>> I am definitely biased in this matter as an owner of
> another
> > >> large
> > >> > > PR
> > >> > > >>>>> that
> > >> > > >>>>>> wasn't preceded by a direct proposal or dev list
> discussion,
> > >> and
> > >> > in
> > >> > > >>>>> general
> > >> > > >>>>>> I agree that proposal first is usually better, but I think
> in
> > >> some
> > >> > > >>>> rarer
> > >> > > >>>>>> cases approaching a problem code first *is* the most
> > >> appropriate
> > >> > way
> > >> > > >> to
> > >> > > >>>>>> have a discussion.
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > >>>>> I am wondering here what is the case where code first is
> > better?
> > >> > > >>>>> In general when you are writing code you have an idea about
> > what
> > >> > you
> > >> > > >> want
> > >> > > >>>>> to change, why you want to change and why you want to change
> > it.
> > >> > > >>>>> I do not see what is wrong with sharing this primitive ideas
> > and
> > >> > > >> thoughts
> > >> > > >>>>> as an abstract proposal (at least to avoid overlapping)
> > >> > > >>>>>
> > >> > > >>>>> I see nothing wrong with it so long as the author
> > >> > > >>>>>> accepts that the PR is treated as a combined proposal and
> > >> proof of
> > >> > > >>>>> concept,
> > >> > > >>>>>> and fair game to be radically changed via discussion or
> even
> > >> > > rejected,
> > >> > > >>>>>> which sounds like Gian's attitude on the matter and is mine
> > as
> > >> > well
> > >> > > >>>> with
> > >> > > >>>>> my
> > >> > > >>>>>> compression stuff.
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > >>>>> Let's not be naive this is very rare that a contributor will
> > >> accept
> > >> > > >> that
> > >> > > >>>>> his work is to be thrown, usually devs takes coding as
> > personal
> > >> > > >> creation
> > >> > > >>>>> and they get attached to it.
> > >> > > >>>>> To my point you can take a look on some old issue in the
> Druid
> > >> > forum
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> > >> > > >>>>> and am sure other communities have similar problems.
> > >> > > >>>>> So leaving the door open to some side cases is not a good
> idea
> > >> in
> > >> > my
> > >> > > >>>>> opinion and will lead to similar issue in the future.
> > >> > > >>>>>
> > >> > > >>>>> This seems to me especially likely to happen in cases
> > >> > > >>>>>> where an approach still needs proven to be a viable idea
> *to
> > >> the
> > >> > > >>>> author*,
> > >> > > >>>>>> so that a much more productive discussion can be had in the
> > >> first
> > >> > > >>>> place.
> > >> > > >>>>>>
> > >> > > >>>>>> I think there is a trade off, I don't think we want to
> > >> discourage
> > >> > > >>>>>> experimentation by walling it off behind mandatory
> > discussions
> > >> > > before
> > >> > > >>>> it
> > >> > > >>>>>> can even start, but I do think formalizing the process for
> > >> large
> > >> > > >>>> changes
> > >> > > >>>>> is
> > >> > > >>>>>> a good thing, especially since we probably can't expect the
> > >> wider
> > >> > > >>>>> community
> > >> > > >>>>>> to have the same attitude towards a large PR getting
> > discarded
> > >> as
> > >> > a
> > >> > > >>>>>> committer might. I think the Kafka approach is reasonable,
> a
> > >> bit
> > >> > > more
> > >> > > >>>>>> formal than our design review process but not overbearing.
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > >>>>> Can you please explain what is overbearing ? what can be
> > >> changed to
> > >> > > >> make
> > >> > > >>>> it
> > >> > > >>>>> easy ?
> > >> > > >>>>> Most of the points are kind of the actual questions that you
> > >> want
> > >> > to
> > >> > > >>>>> address before hand anyway isn't ?
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > >>>>>> Going code first
> > >> > > >>>>>> should be in general discouraged, but when it does happen,
> it
> > >> > seems
> > >> > > >>>> like
> > >> > > >>>>>> opening DIP/an issue/starting a mailing list thread or
> > >> whatever we
> > >> > > go
> > >> > > >>>>> with
> > >> > > >>>>>> to have a more high level design discussion alongside the
> > >> > reference
> > >> > > PR
> > >> > > >>>>>> could alleviate some of these complaints?
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > >>>>> What are the complaints ?
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > >>>>>> +1 for "DIP" heh, I think making
> > >> > > >>>>>> them in the form of github issues is probably appropriate,
> > >> with a
> > >> > > dev
> > >> > > >>>>> list
> > >> > > >>>>>> thread to announce them perhaps?
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>> I think  github issue with [Proposal] header like
> > >> > > >>>>> https://github.com/apache/incubator-druid/issues/4349 is
> good
> > >> to
> > >> > me,
> > >> > > >>>>>
> > >> > > >>>>> Thanks!
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > >>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <
> > >> bslim@apache.org>
> > >> > > >>>> wrote:
> > >> > > >>>>>>
> > >> > > >>>>>>> Thanks everyone for interacting with this thread.
> > >> > > >>>>>>>
> > >> > > >>>>>>> The fact that i, Roman, Jihoon  and others in the past (FJ
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ
> > >> )
> > >> > > >>>>>>> raised this point indicates that PRs without a proposal
> are
> > >> > indeed
> > >> > > an
> > >> > > >>>>>> issue
> > >> > > >>>>>>> and we need to solve it.
> > >> > > >>>>>>>
> > >> > > >>>>>>> Something Similar to KIP maybe called DIPs is fine with
> me.
> > >> > > >>>>>>> What i strive to see is the following:
> > >> > > >>>>>>>
> > >> > > >>>>>>> [Step 1] formalize what is the kind of work that needs a
> > >> formal
> > >> > > >>>>>> Proposal, I
> > >> > > >>>>>>> think Roman and Jihoon has already covered that pretty
> well.
> > >> am
> > >> > +1
> > >> > > on
> > >> > > >>>>>> that.
> > >> > > >>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> > >> > > >>>>>>>
> > >> > > >>>>>>> I am strongly in favor of the separation of Proposal
> Review
> > >> and
> > >> > > >>>> (later)
> > >> > > >>>>>>> Code review PRs. My  main reasons:
> > >> > > >>>>>>> Most importantly code reviewing will introduce lot of
> noise
> > >> and
> > >> > > will
> > >> > > >>>>>>> ultimately make  the GitHub page unreadable.
> > >> > > >>>>>>> Avoid overlapping of work.
> > >> > > >>>>>>> Once code is written hard to think abstract.
> > >> > > >>>>>>> Separate page for Design review later can always be used
> it
> > >> as a
> > >> > > >>>> Design
> > >> > > >>>>>>> document that is readable and code free-ish.
> > >> > > >>>>>>> As i said the goal of this first round is to see if the
> > >> community
> > >> > > >>>> agree
> > >> > > >>>>>>> about such change, then make the process of design more
> > >> inclusive
> > >> > > >>>> thus
> > >> > > >>>>>>> other contributors can submit a counter proposals.
> > >> > > >>>>>>>
> > >> > > >>>>>>> [Step 2] IF everybody agree about that point Step 2 is to
> > >> define
> > >> > > >>>> which
> > >> > > >>>>>>> medium is used to Publish a primitive form of a CODE FREE
> > >> > Abstract
> > >> > > >>>>>> Proposal
> > >> > > >>>>>>> containing at least the following bullet points.
> > >> > > >>>>>>> - The problem description and motivation
> > >> > > >>>>>>> - Overview of the proposed change
> > >> > > >>>>>>> - Operational impact (compatibility/ plans to upgrades)
> > public
> > >> > API
> > >> > > >>>>>> changes,
> > >> > > >>>>>>> configuration changes, algorithm, and so on
> > >> > > >>>>>>> - Expected benefits and drawbacks
> > >> > > >>>>>>> - Rationale and alternatives
> > >> > > >>>>>>> - Estimate Time to Deliver if possible.
> > >> > > >>>>>>>
> > >> > > >>>>>>> The way i think this can be is a Github issue where member
> > of
> > >> the
> > >> > > >>>>>> community
> > >> > > >>>>>>> will interact via comments and the author will be updating
> > the
> > >> > > >>>>>> description
> > >> > > >>>>>>> in the light of comments provided by the community.
> > >> > > >>>>>>>
> > >> > > >>>>>>> During and near the end of the design discussions the
> > author/s
> > >> > can
> > >> > > >>>>> start
> > >> > > >>>>>>> writing POCs to help guide the review process this
> naturally
> > >> will
> > >> > > be
> > >> > > >>>> a
> > >> > > >>>>>> Pull
> > >> > > >>>>>>> request with actual code.
> > >> > > >>>>>>>
> > >> > > >>>>>>> *Now the most important thing is that we need to agree
> that
> > >> any
> > >> > > work
> > >> > > >>>>> that
> > >> > > >>>>>>> does not align with this formal process will be ignored
> and
> > >> the
> > >> > > >>>> author
> > >> > > >>>>>> will
> > >> > > >>>>>>> be asked to start with a DIP*
> > >> > > >>>>>>> *That is what i meant with  “If it didn’t happen on the
> > >> mailing
> > >> > > list,
> > >> > > >>>>> it
> > >> > > >>>>>>> didn’t happen.”*
> > >> > > >>>>>>>
> > >> > > >>>>>>> Thanks and happy coding!
> > >> > > >>>>>>>
> > >> > > >>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <
> > gian@apache.org>
> > >> > > wrote:
> > >> > > >>>>>>>
> > >> > > >>>>>>>> One of the advantages I see with a more formal process is
> > >> (like
> > >> > > >>>> Kafka
> > >> > > >>>>>>> KIPs)
> > >> > > >>>>>>>> is that it levels the playing field a bit and sets some
> > >> ground
> > >> > > >>>> rules
> > >> > > >>>>>> for
> > >> > > >>>>>>>> working together. In a way it can help encourage
> > >> contributions
> > >> > by
> > >> > > >>>>>> making
> > >> > > >>>>>>> it
> > >> > > >>>>>>>> clear what is expected of potential contributors.
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> We have a design review process today that is not as
> formal
> > >> as
> > >> > > >>>> KIPs,
> > >> > > >>>>>> but
> > >> > > >>>>>>> is
> > >> > > >>>>>>>> somewhat heavier than the one you describe. Maybe we
> could
> > >> tweak
> > >> > > >>>> our
> > >> > > >>>>>>>> current one by starting to do design reviews separately
> > from
> > >> > PRs.
> > >> > > >>>>> i.e.,
> > >> > > >>>>>>> for
> > >> > > >>>>>>>> anything that meets our 'design review' criteria, do that
> > on
> > >> the
> > >> > > >>>> dev
> > >> > > >>>>>> list
> > >> > > >>>>>>>> or in a separate issue, and keep the PR focused on
> > code-level
> > >> > > >>>> stuff.
> > >> > > >>>>>> That
> > >> > > >>>>>>>> way we don't end up trying to do both at once. And it
> makes
> > >> it
> > >> > > >>>> easier
> > >> > > >>>>>> to
> > >> > > >>>>>>>> start talking about design before the code is ready,
> which
> > >> would
> > >> > > be
> > >> > > >>>>>>> better.
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <
> > jhyde@apache.org
> > >> >
> > >> > > >>>> wrote:
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>> It’s really hard to say no to a contribution when
> someone
> > >> has
> > >> > put
> > >> > > >>>>> in
> > >> > > >>>>>> a
> > >> > > >>>>>>>>> significant amount of work.
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>>> The following approach is simple and works really well:
> > >> Before
> > >> > > >>>> you
> > >> > > >>>>>>> start
> > >> > > >>>>>>>>> work, log a case, describing the problem. When you have
> > some
> > >> > > >>>> ideas
> > >> > > >>>>>>> about
> > >> > > >>>>>>>>> design, add those to the case. When you have a code
> > branch,
> > >> add
> > >> > > >>>> its
> > >> > > >>>>>> URL
> > >> > > >>>>>>>> to
> > >> > > >>>>>>>>> the case. And so forth. At any point in the proceedings,
> > >> people
> > >> > > >>>> can
> > >> > > >>>>>>> chime
> > >> > > >>>>>>>>> in with their opinions.
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>>> In my opinion, a formal “design review” process is not
> > >> > necessary.
> > >> > > >>>>>> Just
> > >> > > >>>>>>>>> build consensus iteratively, by starting the
> conversation
> > >> early
> > >> > > >>>> in
> > >> > > >>>>>> the
> > >> > > >>>>>>>>> process.
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>>> Julian
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <
> > gian@apache.org
> > >> >
> > >> > > >>>>> wrote:
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>> In this particular case: please consider the PR as a
> > >> proposal.
> > >> > > >>>>>> Don't
> > >> > > >>>>>>>> feel
> > >> > > >>>>>>>>>> like just because there is code there that takes a
> > certain
> > >> > > >>>>>> approach,
> > >> > > >>>>>>>> that
> > >> > > >>>>>>>>>> the approach is somehow sacred. I had to implement
> > >> something
> > >> > to
> > >> > > >>>>>>>>> crystallize
> > >> > > >>>>>>>>>> my own thinking about how the problem could be
> > approached.
> > >> I
> > >> > > >>>>> won't
> > >> > > >>>>>> be
> > >> > > >>>>>>>>>> disappointed if, as a community, we decide a different
> > >> > > >>>> direction
> > >> > > >>>>> is
> > >> > > >>>>>>>>> better
> > >> > > >>>>>>>>>> and the code all gets thrown away. That's one of the
> > >> reasons
> > >> > > >>>>> that I
> > >> > > >>>>>>>>> removed
> > >> > > >>>>>>>>>> the 0.14.0 milestone that was added to the patch. (I
> > don't
> > >> > want
> > >> > > >>>>> to
> > >> > > >>>>>>> rush
> > >> > > >>>>>>>>> it,
> > >> > > >>>>>>>>>> nor do I think that's a good idea.)
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>> In general: Sounds like we could do with some more
> > >> > > >>>> formalization
> > >> > > >>>>>>> around
> > >> > > >>>>>>>>>> what a proposal looks like, which sorts of changes need
> > >> one,
> > >> > > >>>> and
> > >> > > >>>>>> when
> > >> > > >>>>>>>> in
> > >> > > >>>>>>>>>> the dev cycle it is appropriate. FWIW I think Kafka's
> > >> process
> > >> > > >>>> is
> > >> > > >>>>>> more
> > >> > > >>>>>>>> or
> > >> > > >>>>>>>>>> less fine, and would be okay with adopting it for Druid
> > if
> > >> > > >>>> people
> > >> > > >>>>>>> like
> > >> > > >>>>>>>>> it.
> > >> > > >>>>>>>>>> Right now our standards for what requires a "design
> > review"
> > >> > are
> > >> > > >>>>>> very
> > >> > > >>>>>>>>>> similar to the Kafka community standards for what
> > requires
> > >> a
> > >> > > >>>> KIP,
> > >> > > >>>>>> so
> > >> > > >>>>>>> we
> > >> > > >>>>>>>>>> have some familiarity with those concepts. However we
> > don't
> > >> > > >>>>>> separate
> > >> > > >>>>>>> PR
> > >> > > >>>>>>>>>> review and proposal discussion as strictly as they do,
> > >> which
> > >> > > >>>>> seems
> > >> > > >>>>>> to
> > >> > > >>>>>>>> be
> > >> > > >>>>>>>>>> the foundation for the feeling of exclusion that is
> being
> > >> felt
> > >> > > >>>>>> here.
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>> Separately: I just redid the description on
> > >> > > >>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794 to
> > be
> > >> > more
> > >> > > >>>>>>>>> proposal-y.
> > >> > > >>>>>>>>>> I followed the KIP style:
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >> > > >>>>>>>>> .
> > >> > > >>>>>>>>>> Please refresh the page and see if it looks more
> useful.
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>> Gian
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <
> > >> jhyde@apache.org
> > >> > >
> > >> > > >>>>>>> wrote:
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>>> Slim,
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>> I agree with your points that offline development is
> bad
> > >> for
> > >> > > >>>>>>>> community.
> > >> > > >>>>>>>>>>> But I don’t think you need much mentor help. You have
> > >> raised
> > >> > > >>>>> valid
> > >> > > >>>>>>>>> issues
> > >> > > >>>>>>>>>>> and the Druid community needs to decide what its
> > >> development
> > >> > > >>>>>>> practices
> > >> > > >>>>>>>>>>> should be.
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>> Julian
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
> > >> > > >>>> bslim@apache.org>
> > >> > > >>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>> Hello everyone and hope you all have very good
> > holidays.
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>> First, this email is not directed on the author or
> the
> > PR
> > >> > > >>>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794
> > it
> > >> > > >>>> self,
> > >> > > >>>>>> but
> > >> > > >>>>>>> i
> > >> > > >>>>>>>>> see
> > >> > > >>>>>>>>>>>> this PR as a perfect example.
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>> One of the foundation of Apache Way or what i would
> > >> simply
> > >> > > >>>> call
> > >> > > >>>>>>> open
> > >> > > >>>>>>>>>>> source
> > >> > > >>>>>>>>>>>> community driven development is that "Technical
> > decisions
> > >> > are
> > >> > > >>>>>>>>> discussed,
> > >> > > >>>>>>>>>>>> decided, and archived publicly.
> > >> > > >>>>>>>>>>>> developpement"
> > >> > > >>>>>>>>>>>> Which means that big technical  changes such as the
> one
> > >> > > >>>> brought
> > >> > > >>>>>> by
> > >> > > >>>>>>>>> #/6794
> > >> > > >>>>>>>>>>>> should have started as a proposal and round of
> > >> discussions
> > >> > > >>>>> about
> > >> > > >>>>>>> the
> > >> > > >>>>>>>>>>> major
> > >> > > >>>>>>>>>>>> changes designs not as 11K line of code.
> > >> > > >>>>>>>>>>>> I believe such openness will promote a lot of good
> > >> benefits
> > >> > > >>>>> such
> > >> > > >>>>>>> as:
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>> - ensures community health and growth.
> > >> > > >>>>>>>>>>>> - ensures everyone can participate not only the
> authors
> > >> and
> > >> > > >>>> his
> > >> > > >>>>>>>>>>> co-workers.
> > >> > > >>>>>>>>>>>> - ensures that the project is driven by the community
> > and
> > >> > > >>>> not a
> > >> > > >>>>>>> given
> > >> > > >>>>>>>>>>>> company or an individual.
> > >> > > >>>>>>>>>>>> - ensures that there is consensus (not saying 100%
> > >> > > >>>> agreement;)
> > >> > > >>>>>>>> however
> > >> > > >>>>>>>>> it
> > >> > > >>>>>>>>>>>> means that all individuals will accept the current
> > >> progress
> > >> > > >>>> on
> > >> > > >>>>>> the
> > >> > > >>>>>>>>>>> project
> > >> > > >>>>>>>>>>>> until some better proposal is put forth.
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>> Personally such BIG offline PR makes me feel excluded
> > and
> > >> > > >>>>> doesn't
> > >> > > >>>>>>>> give
> > >> > > >>>>>>>>>>> me a
> > >> > > >>>>>>>>>>>> sense that i belong to  a community at all.
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>> To prevent such off list development i think as a
> Druid
> > >> > > >>>>> Community
> > >> > > >>>>>>> we
> > >> > > >>>>>>>>> need
> > >> > > >>>>>>>>>>>> to stick to the apache way “If it didn’t happen on
> the
> > >> > > >>>> mailing
> > >> > > >>>>>>> list,
> > >> > > >>>>>>>> it
> > >> > > >>>>>>>>>>>> didn’t happen.”
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>> I would appreciate if some of the Apache mentor help
> > with
> > >> > > >>>> this.
> > >> > > >>>>>>>>>>>> Thanks
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>
> > >> > >
> > ---------------------------------------------------------------------
> > >> > > >>>>>>>>>>> To unsubscribe, e-mail:
> > dev-unsubscribe@druid.apache.org
> > >> > > >>>>>>>>>>> For additional commands, e-mail:
> > >> dev-help@druid.apache.org
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>>>
> > >> > > >>>>>
> > >> >
> ---------------------------------------------------------------------
> > >> > > >>>>>>>>> To unsubscribe, e-mail:
> dev-unsubscribe@druid.apache.org
> > >> > > >>>>>>>>> For additional commands, e-mail:
> > dev-help@druid.apache.org
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > >>>>> --
> > >> > > >>>>>
> > >> > > >>>>> B-Slim
> > >> > > >>>>>
> > >> > > >>
> > >> >
> > _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> ---------------------------------------------------------------------
> > >> > > >> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > >> > > >> For additional commands, e-mail: dev-help@druid.apache.org
> > >> > > >>
> > >> > > >>
> > >> > >
> > >> > >
> > >> > >
> > ---------------------------------------------------------------------
> > >> > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > >> > > For additional commands, e-mail: dev-help@druid.apache.org
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: Off list major development

[Clint Wylie <cl...@imply.io>]

>
> What do community members think about also making a requirement that
> "Design Review" PRs and proposals are reviewed by at least two people with
> different affiliation?


This seems pretty reasonable to me. I haven't found anything in Apache
voting procedure docs (https://www.apache.org/foundation/voting.html) that
seems to explicitly forbid something like this yet at least.

On Wed, Jan 9, 2019 at 8:52 PM Roman Leventov <le...@gmail.com> wrote:

> What do community members think about also making a requirement that
> "Design Review" PRs and proposals are reviewed by at least two people with
> different affiliation? IMO it's a good idea, because it ensures that
> different interests are taken into account. Also it pushes people to engage
> with work done in other parts of Druid, improving shared code owning and
> awareness.
>
> Or it's against the Apache Way because it's assumed that there are no
> company boundaries within the community?
>
> On Thu, 10 Jan 2019 at 11:45, Roman Leventov <le...@gmail.com>
> wrote:
>
> > I see two important reasons why it makes sense to file an issue and
> > probably announce it in the mailing list, before writing a lot of code,
> > despite not having a clear picture of what it will be and any performance
> > data:
> >  1) somebody could already work on this problem privately in parallel, it
> > allows to avoid clash in people's works
> >  2) some people could quickly think about the problem field and share
> > high-level ideas that could wildly change the direction in which the
> author
> > (the person who is going to write code) will move in his work from early
> on.
> >
> > Jihoon in
> >
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
> and
> > later Gian in this thread suggested that _every_ piece of work that
> should
> > be labelled as "Design Review" according to the current rules should be
> > accompanied by an issue. I don't agree with this, there are some PRs as
> > small as a few dozens of lines of code, that add some configuration
> > parameter and therefore should be labelled "Design Review". I don't
> thing a
> > separate proposal issue is needed for them, and even for a little larger
> > PRs too.
> >
> > For the same reason, I also don't see the point of renaming "Design
> > Review" into "Proposal", as well as separating "Design Review" into
> > "Proposal" and something like "API Review". I think a single "Design
> > Review" tag handles it well.
> >
> > Gian mentioned an idea that PRs that follow a "Design Review" proposal
> > issue shouldn't be "Design Review" themselves. I don't agree with this, I
> > think that actual code and performance data are important inputs that
> > should be re-evaluated at least by two people. I even think that it's
> very
> > desirable that at least two people read _each line of production code_ in
> > large PRs, although it's not what was done historically in Druid, because
> > large bodies of newly added code, with whole new classes and subsystems
> > added, are also coincidentally tested worse than already existing classes
> > and subsystems, including in production. It seems to me that those huge
> > code influxes is a major source of bugs, that could later take years to
> > squeeze out from the codebase.
> >
> > On Wed, 9 Jan 2019 at 08:24, Clint Wylie <cl...@imply.io> wrote:
> >
> >> Apologies for the delayed response.
> >>
> >> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> slim.bouguerra@gmail.com>
> >> wrote:
> >>
> >> > I am wondering here what is the case where code first is better?
> >> >
> >>
> >> I don't think it's wrong to share ideas as early as possible, and after
> >> this discussion I think I am in favor of it too. I just meant that I
> don't
> >> think it's always necessarily the most productive discussion until code
> >> exists sometimes, with the types of thing I am thinking of are almost
> >> entirely limited to cases where things might sound good to anyone on
> paper
> >> but in reality need a large amount of experiments conducted and
> >> observations collected to determine that something is actually worth
> >> doing,
> >> which I imagine is mainly things like reworking internals for
> performance
> >> improvements.
> >>
> >> In the case of my combined proposal PR, I needed to prove that the
> thing I
> >> was working on was a good idea... and it wasn't directly. But I came up
> >> with another idea during the course of experiment turned into something
> >> compelling, so an initial proposal would have looked quite a lot
> different
> >> than what I ended up with. Once I had proven to myself that it was a
> good
> >> idea, then I was comfortable sharing with the wider community. I'm not
> >> certain how this would play out in an always proposal first model, maybe
> >> the first proposal exists, I personally reject it after updating with
> >> experiment results show it's a bad idea, continue experimenting and
> raise
> >> a
> >> new one after the experiments start looking promising?
> >>
> >>
> >> > Let's not be naive this is very rare that a contributor will accept
> that
> >> > his work is to be thrown, usually devs takes coding as personal
> creation
> >> > and they get attached to it.
> >> >
> >>
> >> I agree, just because a handful of the committers have this attitude, it
> >> isn't fair to expect the wider community to also, that's why I am in
> favor
> >> of formalizing the process.
> >>
> >> Can you please explain what is overbearing ? what can be changed to make
> >> it
> >> > easy ?
> >> > Most of the points are kind of the actual questions that you want to
> >> > address before hand anyway isn't ?
> >> >
> >>
> >> Sorry for the confusion, I said it's "not overbearing", I think it's
> fine.
> >>
> >> What are the complaints ?
> >>
> >>
> >> Is this and other previous threads not a complaint about opening a large
> >> PR
> >> without a proposal? :) I just mean that formalizing the process, even
> if a
> >> proposal has a reference PR opened with it near concurrently, could
> >> prevent
> >> these discussions from happening in the future because ground rules have
> >> been set and we are all on the same page I guess.
> >>
> >> I think we should also probably consider renaming the "design review"
> >> label
> >> to "proposal" or something to make it more clear that a PR is associated
> >> with a proposal. It might also be worth considering using github
> projects
> >> for particularly large things that involve multiple follow up PRs, but I
> >> haven't used them in much depth to know if they add anything.
> >>
> >> It seems like we are all converging on agreement to do a github issue
> >> proposal for larger changes (I would vote for announcing them on the dev
> >> list too for more visibility), so that design review is separated from
> >> code
> >> review. I guess my main concern was not wanting to discourage
> >> experimentation by walling it off behind mandatory discussions, but the
> >> more I think about it,  it doesn't technically change much about this
> >> process, it just requires a more formal proposal to accompany any
> >> experiments that are shared as a PR.
> >>
> >>
> >> On Tue, Jan 8, 2019 at 12:28 PM Gian Merlino <gi...@apache.org> wrote:
> >>
> >> > I think for us, choosing to use GitHub issues as discussion threads
> for
> >> > potential 'major' contributions would be a good idea, especially if we
> >> > encourage people to start them before PRs show up. Definitely agree
> that
> >> > all contributors should go through the same process -- I couldn't see
> it
> >> > working well any other way.
> >> >
> >> > On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <jh...@apache.org> wrote:
> >> >
> >> > > Statically, yes, GitHub PRs are the same as GitHub cases. But
> >> > dynamically,
> >> > > they are different, because you can only log a PR when you have
> >> finished
> >> > > work.
> >> > >
> >> > > A lot of other Apache projects use JIRA, so there is a clear
> >> distinction
> >> > > between cases and contributions. JIRA cases, especially when logged
> >> early
> >> > > in the lifecycle of a contribution, become long-running conversation
> >> > > threads with a lot of community participation. If the Druid chose to
> >> do
> >> > so,
> >> > > GitHub cases could be the same.
> >> > >
> >> > > Be careful that you do not treat “potential contributors” (by which
> I
> >> > > presume you mean non-committers) differently from committers and PMC
> >> > > members. Anyone starting a major piece of work should follow the
> same
> >> > > process. (Experienced committers probably have a somewhat better
> idea
> >> > what
> >> > > work will turn out to be “major”, so they get a little more leeway.)
> >> > >
> >> > > Julian
> >> > >
> >> > >
> >> > > > On Jan 7, 2019, at 12:10 PM, Gian Merlino <gi...@apache.org>
> wrote:
> >> > > >
> >> > > > I don't think there's a need to raise issues for every change: a
> >> small
> >> > > bug
> >> > > > fix or doc fix should just go straight to PR. (GitHub PRs show up
> as
> >> > > issues
> >> > > > in the issue-search UI/API, so it's not like this means the patch
> >> has
> >> > no
> >> > > > corresponding issue -- in a sense the PR _is_ the issue.)
> >> > > >
> >> > > > I do think it makes sense to encourage potential contributors to
> >> write
> >> > to
> >> > > > the dev list or raise an issue if they aren't sure if something
> >> would
> >> > > need
> >> > > > to go through a more heavy weight process.
> >> > > >
> >> > > > Fwiw we do have a set of 'design review' criteria already (we had
> a
> >> > > > discussion about this a couple years ago) at:
> >> > > > http://druid.io/community/#getting-your-changes-accepted. So we
> >> > > wouldn't be
> >> > > > starting from zero on defining that. We set it up back when we
> were
> >> > > trying
> >> > > > to _streamline_ our process -- we used to require two non-author
> +1s
> >> > for
> >> > > > _every_ change, even minor ones. The introduction of design review
> >> > > criteria
> >> > > > was meant to classify which PRs need that level of review and
> which
> >> > ones
> >> > > > are minor and can be merged with less review. I do think it helped
> >> with
> >> > > > getting minor PRs merged more quickly. The list of criteria is,
> >> > > >
> >> > > > - Major architectural changes or API changes
> >> > > > - HTTP requests and responses (e. g. a new HTTP endpoint)
> >> > > > - Interfaces for extensions
> >> > > > - Server configuration (e. g. altering the behavior of a config
> >> > property)
> >> > > > - Emitted metrics
> >> > > > - Other major changes, judged by the discretion of Druid
> committers
> >> > > >
> >> > > > Some of it is subjective, but it has been in place for a while, so
> >> it's
> >> > > at
> >> > > > least something we are relatively familiar with.
> >> > > >
> >> > > > On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <jh...@apache.org>
> >> wrote:
> >> > > >
> >> > > >> Small contributions don’t need any design review, whereas large
> >> > > >> contributions need significant review. I don’t think we should
> >> require
> >> > > an
> >> > > >> additional step for those (many) small contributions. But who
> >> decides
> >> > > >> whether a contribution fits into the small or large category?
> >> > > >>
> >> > > >> I think the solution is for authors to log a case (or send an
> >> email to
> >> > > >> dev) before they start work on any contribution. Then committers
> >> can
> >> > > >> request a more heavy-weight process if they think it is needed.
> >> > > >>
> >> > > >> Julian
> >> > > >>
> >> > > >>
> >> > > >>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <gi...@apache.org>
> >> wrote:
> >> > > >>>
> >> > > >>> It sounds like splitting design from code review is a common
> theme
> >> > in a
> >> > > >> few
> >> > > >>> of the posts here. How does everyone feel about making a point
> of
> >> > > >>> encouraging design reviews to be done as issues, separate from
> the
> >> > pull
> >> > > >>> request, with the expectations that (1) the design review issue
> >> > > >>> ("proposal") should generally appear somewhat _before_ the pull
> >> > > request;
> >> > > >>> (2) pull requests should _not_ have design review happen on
> them,
> >> > > meaning
> >> > > >>> there should no longer be PRs with design review tags, and we
> >> should
> >> > > move
> >> > > >>> the design review approval process to the issue rather than the
> >> PR.
> >> > > >>>
> >> > > >>> For (1), even if we encourage design review discussions to start
> >> > > before a
> >> > > >>> pull request appears, I don't see an issue with them running
> >> > > concurrently
> >> > > >>> for a while at some point.
> >> > > >>>
> >> > > >>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <jo...@apache.org>
> >> > wrote:
> >> > > >>>
> >> > > >>>> Thanks for raising these concerns!
> >> > > >>>>
> >> > > >>>> My initial thoughts:
> >> > > >>>> - I agree that separation of design review and code-level
> review
> >> for
> >> > > >> major
> >> > > >>>> changes would be more efficient
> >> > > >>>>
> >> > > >>>> - I agree that a clear, more formalized process for handling
> >> major
> >> > > >> changes
> >> > > >>>> would be helpful for contributors:
> >> > > >>>> - Define what is considered a major change
> >> > > >>>> - Define a standard proposal structure, KIP-style proposal
> format
> >> > > >> sounds
> >> > > >>>> good to me
> >> > > >>>>
> >> > > >>>> - I think it's too rigid to have a policy of "no code at all
> with
> >> > the
> >> > > >>>> initial proposal"
> >> > > >>>> - Code samples can be useful references for understanding
> aspects
> >> > of a
> >> > > >>>> design
> >> > > >>>> - In some cases it's necessary to run experiments to fully
> >> > understand
> >> > > a
> >> > > >>>> problem and determine an appropriate design, or to determine
> >> whether
> >> > > >>>> something is even worth doing before committing to the work of
> >> > > fleshing
> >> > > >> out
> >> > > >>>> a proposal, prototype code is a natural outcome of that and I'm
> >> not
> >> > > >> against
> >> > > >>>> someone providing such code for reference
> >> > > >>>> - I tend to view design/code as things that are often developed
> >> > > >>>> simultaneously in an intertwined way
> >> > > >>>>
> >> > > >>>>> Let's not be naive this is very rare that a contributor will
> >> accept
> >> > > >> that
> >> > > >>>> his work is to be thrown, usually devs takes coding as personal
> >> > > creation
> >> > > >>>> and they get attached to it.
> >> > > >>>>
> >> > > >>>> If we have a clear review process that emphasizes the need for
> >> early
> >> > > >>>> consensus building, with separate design and code review, then
> I
> >> > feel
> >> > > >> we've
> >> > > >>>> done enough and don't need a hard rule against having some code
> >> > linked
> >> > > >> with
> >> > > >>>> the initial proposal. If a potential contributor then still
> >> wants to
> >> > > go
> >> > > >>>> ahead and write a lot of code that may be rejected or change
> >> > > >> significantly,
> >> > > >>>> the risks were made clear.
> >> > > >>>>
> >> > > >>>>> Once code is written hard to think abstract.
> >> > > >>>>
> >> > > >>>> I can see the validity of the concern, but I personally don't
> >> see it
> >> > > as
> >> > > >> a
> >> > > >>>> huge risk. My impression from the Druid PR reviews I've seen is
> >> that
> >> > > our
> >> > > >>>> reviewers are able to keep abstract design vs. implementation
> >> > details
> >> > > >>>> separate and consider alternate designs when reviewing.
> >> > > >>>>
> >> > > >>>> To summarize I think it's probably enough to have a policy
> along
> >> the
> >> > > >> lines
> >> > > >>>> of:
> >> > > >>>> - Create more formalized guidelines for proposals and what
> >> changes
> >> > > >> require
> >> > > >>>> proposals
> >> > > >>>> - Separate design and code review for major changes, with
> design
> >> > > review
> >> > > >>>> first, code-level review after reaching consensus on the
> design.
> >> > > >>>> - Code before the design review is completed is just for
> >> reference,
> >> > > not
> >> > > >>>> regarded as a candidate for review/merging.
> >> > > >>>>
> >> > > >>>> - Jon
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> >> > > >> slim.bouguerra@gmail.com>
> >> > > >>>> wrote:
> >> > > >>>>
> >> > > >>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <
> >> clint.wylie@imply.io>
> >> > > >>>> wrote:
> >> > > >>>>>
> >> > > >>>>>> I am definitely biased in this matter as an owner of another
> >> large
> >> > > PR
> >> > > >>>>> that
> >> > > >>>>>> wasn't preceded by a direct proposal or dev list discussion,
> >> and
> >> > in
> >> > > >>>>> general
> >> > > >>>>>> I agree that proposal first is usually better, but I think in
> >> some
> >> > > >>>> rarer
> >> > > >>>>>> cases approaching a problem code first *is* the most
> >> appropriate
> >> > way
> >> > > >> to
> >> > > >>>>>> have a discussion.
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>> I am wondering here what is the case where code first is
> better?
> >> > > >>>>> In general when you are writing code you have an idea about
> what
> >> > you
> >> > > >> want
> >> > > >>>>> to change, why you want to change and why you want to change
> it.
> >> > > >>>>> I do not see what is wrong with sharing this primitive ideas
> and
> >> > > >> thoughts
> >> > > >>>>> as an abstract proposal (at least to avoid overlapping)
> >> > > >>>>>
> >> > > >>>>> I see nothing wrong with it so long as the author
> >> > > >>>>>> accepts that the PR is treated as a combined proposal and
> >> proof of
> >> > > >>>>> concept,
> >> > > >>>>>> and fair game to be radically changed via discussion or even
> >> > > rejected,
> >> > > >>>>>> which sounds like Gian's attitude on the matter and is mine
> as
> >> > well
> >> > > >>>> with
> >> > > >>>>> my
> >> > > >>>>>> compression stuff.
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>> Let's not be naive this is very rare that a contributor will
> >> accept
> >> > > >> that
> >> > > >>>>> his work is to be thrown, usually devs takes coding as
> personal
> >> > > >> creation
> >> > > >>>>> and they get attached to it.
> >> > > >>>>> To my point you can take a look on some old issue in the Druid
> >> > forum
> >> > > >>>>>
> >> > > >>>>
> >> > > >>
> >> > >
> >> >
> >>
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> >> > > >>>>> and am sure other communities have similar problems.
> >> > > >>>>> So leaving the door open to some side cases is not a good idea
> >> in
> >> > my
> >> > > >>>>> opinion and will lead to similar issue in the future.
> >> > > >>>>>
> >> > > >>>>> This seems to me especially likely to happen in cases
> >> > > >>>>>> where an approach still needs proven to be a viable idea *to
> >> the
> >> > > >>>> author*,
> >> > > >>>>>> so that a much more productive discussion can be had in the
> >> first
> >> > > >>>> place.
> >> > > >>>>>>
> >> > > >>>>>> I think there is a trade off, I don't think we want to
> >> discourage
> >> > > >>>>>> experimentation by walling it off behind mandatory
> discussions
> >> > > before
> >> > > >>>> it
> >> > > >>>>>> can even start, but I do think formalizing the process for
> >> large
> >> > > >>>> changes
> >> > > >>>>> is
> >> > > >>>>>> a good thing, especially since we probably can't expect the
> >> wider
> >> > > >>>>> community
> >> > > >>>>>> to have the same attitude towards a large PR getting
> discarded
> >> as
> >> > a
> >> > > >>>>>> committer might. I think the Kafka approach is reasonable, a
> >> bit
> >> > > more
> >> > > >>>>>> formal than our design review process but not overbearing.
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>> Can you please explain what is overbearing ? what can be
> >> changed to
> >> > > >> make
> >> > > >>>> it
> >> > > >>>>> easy ?
> >> > > >>>>> Most of the points are kind of the actual questions that you
> >> want
> >> > to
> >> > > >>>>> address before hand anyway isn't ?
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>>> Going code first
> >> > > >>>>>> should be in general discouraged, but when it does happen, it
> >> > seems
> >> > > >>>> like
> >> > > >>>>>> opening DIP/an issue/starting a mailing list thread or
> >> whatever we
> >> > > go
> >> > > >>>>> with
> >> > > >>>>>> to have a more high level design discussion alongside the
> >> > reference
> >> > > PR
> >> > > >>>>>> could alleviate some of these complaints?
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>> What are the complaints ?
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>>> +1 for "DIP" heh, I think making
> >> > > >>>>>> them in the form of github issues is probably appropriate,
> >> with a
> >> > > dev
> >> > > >>>>> list
> >> > > >>>>>> thread to announce them perhaps?
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>> I think  github issue with [Proposal] header like
> >> > > >>>>> https://github.com/apache/incubator-druid/issues/4349 is good
> >> to
> >> > me,
> >> > > >>>>>
> >> > > >>>>> Thanks!
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <
> >> bslim@apache.org>
> >> > > >>>> wrote:
> >> > > >>>>>>
> >> > > >>>>>>> Thanks everyone for interacting with this thread.
> >> > > >>>>>>>
> >> > > >>>>>>> The fact that i, Roman, Jihoon  and others in the past (FJ
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>
> >> > >
> >> >
> >>
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ
> >> )
> >> > > >>>>>>> raised this point indicates that PRs without a proposal are
> >> > indeed
> >> > > an
> >> > > >>>>>> issue
> >> > > >>>>>>> and we need to solve it.
> >> > > >>>>>>>
> >> > > >>>>>>> Something Similar to KIP maybe called DIPs is fine with me.
> >> > > >>>>>>> What i strive to see is the following:
> >> > > >>>>>>>
> >> > > >>>>>>> [Step 1] formalize what is the kind of work that needs a
> >> formal
> >> > > >>>>>> Proposal, I
> >> > > >>>>>>> think Roman and Jihoon has already covered that pretty well.
> >> am
> >> > +1
> >> > > on
> >> > > >>>>>> that.
> >> > > >>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>
> >> > >
> >> >
> >>
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> >> > > >>>>>>>
> >> > > >>>>>>> I am strongly in favor of the separation of Proposal Review
> >> and
> >> > > >>>> (later)
> >> > > >>>>>>> Code review PRs. My  main reasons:
> >> > > >>>>>>> Most importantly code reviewing will introduce lot of noise
> >> and
> >> > > will
> >> > > >>>>>>> ultimately make  the GitHub page unreadable.
> >> > > >>>>>>> Avoid overlapping of work.
> >> > > >>>>>>> Once code is written hard to think abstract.
> >> > > >>>>>>> Separate page for Design review later can always be used it
> >> as a
> >> > > >>>> Design
> >> > > >>>>>>> document that is readable and code free-ish.
> >> > > >>>>>>> As i said the goal of this first round is to see if the
> >> community
> >> > > >>>> agree
> >> > > >>>>>>> about such change, then make the process of design more
> >> inclusive
> >> > > >>>> thus
> >> > > >>>>>>> other contributors can submit a counter proposals.
> >> > > >>>>>>>
> >> > > >>>>>>> [Step 2] IF everybody agree about that point Step 2 is to
> >> define
> >> > > >>>> which
> >> > > >>>>>>> medium is used to Publish a primitive form of a CODE FREE
> >> > Abstract
> >> > > >>>>>> Proposal
> >> > > >>>>>>> containing at least the following bullet points.
> >> > > >>>>>>> - The problem description and motivation
> >> > > >>>>>>> - Overview of the proposed change
> >> > > >>>>>>> - Operational impact (compatibility/ plans to upgrades)
> public
> >> > API
> >> > > >>>>>> changes,
> >> > > >>>>>>> configuration changes, algorithm, and so on
> >> > > >>>>>>> - Expected benefits and drawbacks
> >> > > >>>>>>> - Rationale and alternatives
> >> > > >>>>>>> - Estimate Time to Deliver if possible.
> >> > > >>>>>>>
> >> > > >>>>>>> The way i think this can be is a Github issue where member
> of
> >> the
> >> > > >>>>>> community
> >> > > >>>>>>> will interact via comments and the author will be updating
> the
> >> > > >>>>>> description
> >> > > >>>>>>> in the light of comments provided by the community.
> >> > > >>>>>>>
> >> > > >>>>>>> During and near the end of the design discussions the
> author/s
> >> > can
> >> > > >>>>> start
> >> > > >>>>>>> writing POCs to help guide the review process this naturally
> >> will
> >> > > be
> >> > > >>>> a
> >> > > >>>>>> Pull
> >> > > >>>>>>> request with actual code.
> >> > > >>>>>>>
> >> > > >>>>>>> *Now the most important thing is that we need to agree that
> >> any
> >> > > work
> >> > > >>>>> that
> >> > > >>>>>>> does not align with this formal process will be ignored and
> >> the
> >> > > >>>> author
> >> > > >>>>>> will
> >> > > >>>>>>> be asked to start with a DIP*
> >> > > >>>>>>> *That is what i meant with  “If it didn’t happen on the
> >> mailing
> >> > > list,
> >> > > >>>>> it
> >> > > >>>>>>> didn’t happen.”*
> >> > > >>>>>>>
> >> > > >>>>>>> Thanks and happy coding!
> >> > > >>>>>>>
> >> > > >>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <
> gian@apache.org>
> >> > > wrote:
> >> > > >>>>>>>
> >> > > >>>>>>>> One of the advantages I see with a more formal process is
> >> (like
> >> > > >>>> Kafka
> >> > > >>>>>>> KIPs)
> >> > > >>>>>>>> is that it levels the playing field a bit and sets some
> >> ground
> >> > > >>>> rules
> >> > > >>>>>> for
> >> > > >>>>>>>> working together. In a way it can help encourage
> >> contributions
> >> > by
> >> > > >>>>>> making
> >> > > >>>>>>> it
> >> > > >>>>>>>> clear what is expected of potential contributors.
> >> > > >>>>>>>>
> >> > > >>>>>>>> We have a design review process today that is not as formal
> >> as
> >> > > >>>> KIPs,
> >> > > >>>>>> but
> >> > > >>>>>>> is
> >> > > >>>>>>>> somewhat heavier than the one you describe. Maybe we could
> >> tweak
> >> > > >>>> our
> >> > > >>>>>>>> current one by starting to do design reviews separately
> from
> >> > PRs.
> >> > > >>>>> i.e.,
> >> > > >>>>>>> for
> >> > > >>>>>>>> anything that meets our 'design review' criteria, do that
> on
> >> the
> >> > > >>>> dev
> >> > > >>>>>> list
> >> > > >>>>>>>> or in a separate issue, and keep the PR focused on
> code-level
> >> > > >>>> stuff.
> >> > > >>>>>> That
> >> > > >>>>>>>> way we don't end up trying to do both at once. And it makes
> >> it
> >> > > >>>> easier
> >> > > >>>>>> to
> >> > > >>>>>>>> start talking about design before the code is ready, which
> >> would
> >> > > be
> >> > > >>>>>>> better.
> >> > > >>>>>>>>
> >> > > >>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <
> jhyde@apache.org
> >> >
> >> > > >>>> wrote:
> >> > > >>>>>>>>
> >> > > >>>>>>>>> It’s really hard to say no to a contribution when someone
> >> has
> >> > put
> >> > > >>>>> in
> >> > > >>>>>> a
> >> > > >>>>>>>>> significant amount of work.
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> The following approach is simple and works really well:
> >> Before
> >> > > >>>> you
> >> > > >>>>>>> start
> >> > > >>>>>>>>> work, log a case, describing the problem. When you have
> some
> >> > > >>>> ideas
> >> > > >>>>>>> about
> >> > > >>>>>>>>> design, add those to the case. When you have a code
> branch,
> >> add
> >> > > >>>> its
> >> > > >>>>>> URL
> >> > > >>>>>>>> to
> >> > > >>>>>>>>> the case. And so forth. At any point in the proceedings,
> >> people
> >> > > >>>> can
> >> > > >>>>>>> chime
> >> > > >>>>>>>>> in with their opinions.
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> In my opinion, a formal “design review” process is not
> >> > necessary.
> >> > > >>>>>> Just
> >> > > >>>>>>>>> build consensus iteratively, by starting the conversation
> >> early
> >> > > >>>> in
> >> > > >>>>>> the
> >> > > >>>>>>>>> process.
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> Julian
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <
> gian@apache.org
> >> >
> >> > > >>>>> wrote:
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>> In this particular case: please consider the PR as a
> >> proposal.
> >> > > >>>>>> Don't
> >> > > >>>>>>>> feel
> >> > > >>>>>>>>>> like just because there is code there that takes a
> certain
> >> > > >>>>>> approach,
> >> > > >>>>>>>> that
> >> > > >>>>>>>>>> the approach is somehow sacred. I had to implement
> >> something
> >> > to
> >> > > >>>>>>>>> crystallize
> >> > > >>>>>>>>>> my own thinking about how the problem could be
> approached.
> >> I
> >> > > >>>>> won't
> >> > > >>>>>> be
> >> > > >>>>>>>>>> disappointed if, as a community, we decide a different
> >> > > >>>> direction
> >> > > >>>>> is
> >> > > >>>>>>>>> better
> >> > > >>>>>>>>>> and the code all gets thrown away. That's one of the
> >> reasons
> >> > > >>>>> that I
> >> > > >>>>>>>>> removed
> >> > > >>>>>>>>>> the 0.14.0 milestone that was added to the patch. (I
> don't
> >> > want
> >> > > >>>>> to
> >> > > >>>>>>> rush
> >> > > >>>>>>>>> it,
> >> > > >>>>>>>>>> nor do I think that's a good idea.)
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>> In general: Sounds like we could do with some more
> >> > > >>>> formalization
> >> > > >>>>>>> around
> >> > > >>>>>>>>>> what a proposal looks like, which sorts of changes need
> >> one,
> >> > > >>>> and
> >> > > >>>>>> when
> >> > > >>>>>>>> in
> >> > > >>>>>>>>>> the dev cycle it is appropriate. FWIW I think Kafka's
> >> process
> >> > > >>>> is
> >> > > >>>>>> more
> >> > > >>>>>>>> or
> >> > > >>>>>>>>>> less fine, and would be okay with adopting it for Druid
> if
> >> > > >>>> people
> >> > > >>>>>>> like
> >> > > >>>>>>>>> it.
> >> > > >>>>>>>>>> Right now our standards for what requires a "design
> review"
> >> > are
> >> > > >>>>>> very
> >> > > >>>>>>>>>> similar to the Kafka community standards for what
> requires
> >> a
> >> > > >>>> KIP,
> >> > > >>>>>> so
> >> > > >>>>>>> we
> >> > > >>>>>>>>>> have some familiarity with those concepts. However we
> don't
> >> > > >>>>>> separate
> >> > > >>>>>>> PR
> >> > > >>>>>>>>>> review and proposal discussion as strictly as they do,
> >> which
> >> > > >>>>> seems
> >> > > >>>>>> to
> >> > > >>>>>>>> be
> >> > > >>>>>>>>>> the foundation for the feeling of exclusion that is being
> >> felt
> >> > > >>>>>> here.
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>> Separately: I just redid the description on
> >> > > >>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794 to
> be
> >> > more
> >> > > >>>>>>>>> proposal-y.
> >> > > >>>>>>>>>> I followed the KIP style:
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >> > > >>>>>>>>> .
> >> > > >>>>>>>>>> Please refresh the page and see if it looks more useful.
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>> Gian
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <
> >> jhyde@apache.org
> >> > >
> >> > > >>>>>>> wrote:
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>>> Slim,
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>> I agree with your points that offline development is bad
> >> for
> >> > > >>>>>>>> community.
> >> > > >>>>>>>>>>> But I don’t think you need much mentor help. You have
> >> raised
> >> > > >>>>> valid
> >> > > >>>>>>>>> issues
> >> > > >>>>>>>>>>> and the Druid community needs to decide what its
> >> development
> >> > > >>>>>>> practices
> >> > > >>>>>>>>>>> should be.
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>> Julian
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
> >> > > >>>> bslim@apache.org>
> >> > > >>>>>>>> wrote:
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>> Hello everyone and hope you all have very good
> holidays.
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>> First, this email is not directed on the author or the
> PR
> >> > > >>>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794
> it
> >> > > >>>> self,
> >> > > >>>>>> but
> >> > > >>>>>>> i
> >> > > >>>>>>>>> see
> >> > > >>>>>>>>>>>> this PR as a perfect example.
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>> One of the foundation of Apache Way or what i would
> >> simply
> >> > > >>>> call
> >> > > >>>>>>> open
> >> > > >>>>>>>>>>> source
> >> > > >>>>>>>>>>>> community driven development is that "Technical
> decisions
> >> > are
> >> > > >>>>>>>>> discussed,
> >> > > >>>>>>>>>>>> decided, and archived publicly.
> >> > > >>>>>>>>>>>> developpement"
> >> > > >>>>>>>>>>>> Which means that big technical  changes such as the one
> >> > > >>>> brought
> >> > > >>>>>> by
> >> > > >>>>>>>>> #/6794
> >> > > >>>>>>>>>>>> should have started as a proposal and round of
> >> discussions
> >> > > >>>>> about
> >> > > >>>>>>> the
> >> > > >>>>>>>>>>> major
> >> > > >>>>>>>>>>>> changes designs not as 11K line of code.
> >> > > >>>>>>>>>>>> I believe such openness will promote a lot of good
> >> benefits
> >> > > >>>>> such
> >> > > >>>>>>> as:
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>> - ensures community health and growth.
> >> > > >>>>>>>>>>>> - ensures everyone can participate not only the authors
> >> and
> >> > > >>>> his
> >> > > >>>>>>>>>>> co-workers.
> >> > > >>>>>>>>>>>> - ensures that the project is driven by the community
> and
> >> > > >>>> not a
> >> > > >>>>>>> given
> >> > > >>>>>>>>>>>> company or an individual.
> >> > > >>>>>>>>>>>> - ensures that there is consensus (not saying 100%
> >> > > >>>> agreement;)
> >> > > >>>>>>>> however
> >> > > >>>>>>>>> it
> >> > > >>>>>>>>>>>> means that all individuals will accept the current
> >> progress
> >> > > >>>> on
> >> > > >>>>>> the
> >> > > >>>>>>>>>>> project
> >> > > >>>>>>>>>>>> until some better proposal is put forth.
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>> Personally such BIG offline PR makes me feel excluded
> and
> >> > > >>>>> doesn't
> >> > > >>>>>>>> give
> >> > > >>>>>>>>>>> me a
> >> > > >>>>>>>>>>>> sense that i belong to  a community at all.
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>> To prevent such off list development i think as a Druid
> >> > > >>>>> Community
> >> > > >>>>>>> we
> >> > > >>>>>>>>> need
> >> > > >>>>>>>>>>>> to stick to the apache way “If it didn’t happen on the
> >> > > >>>> mailing
> >> > > >>>>>>> list,
> >> > > >>>>>>>> it
> >> > > >>>>>>>>>>>> didn’t happen.”
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>> I would appreciate if some of the Apache mentor help
> with
> >> > > >>>> this.
> >> > > >>>>>>>>>>>> Thanks
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>
> >> > >
> ---------------------------------------------------------------------
> >> > > >>>>>>>>>>> To unsubscribe, e-mail:
> dev-unsubscribe@druid.apache.org
> >> > > >>>>>>>>>>> For additional commands, e-mail:
> >> dev-help@druid.apache.org
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>
> >> > ---------------------------------------------------------------------
> >> > > >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> >> > > >>>>>>>>> For additional commands, e-mail:
> dev-help@druid.apache.org
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>> --
> >> > > >>>>>
> >> > > >>>>> B-Slim
> >> > > >>>>>
> >> > > >>
> >> >
> _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> >> > > >>>>>
> >> > > >>>>
> >> > > >>
> >> > > >>
> >> > > >>
> >> ---------------------------------------------------------------------
> >> > > >> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> >> > > >> For additional commands, e-mail: dev-help@druid.apache.org
> >> > > >>
> >> > > >>
> >> > >
> >> > >
> >> > >
> ---------------------------------------------------------------------
> >> > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> >> > > For additional commands, e-mail: dev-help@druid.apache.org
> >> > >
> >> > >
> >> >
> >>
> >
>

Re: Off list major development

[Roman Leventov <le...@gmail.com>]

What do community members think about also making a requirement that
"Design Review" PRs and proposals are reviewed by at least two people with
different affiliation? IMO it's a good idea, because it ensures that
different interests are taken into account. Also it pushes people to engage
with work done in other parts of Druid, improving shared code owning and
awareness.

Or it's against the Apache Way because it's assumed that there are no
company boundaries within the community?

On Thu, 10 Jan 2019 at 11:45, Roman Leventov <le...@gmail.com> wrote:

> I see two important reasons why it makes sense to file an issue and
> probably announce it in the mailing list, before writing a lot of code,
> despite not having a clear picture of what it will be and any performance
> data:
>  1) somebody could already work on this problem privately in parallel, it
> allows to avoid clash in people's works
>  2) some people could quickly think about the problem field and share
> high-level ideas that could wildly change the direction in which the author
> (the person who is going to write code) will move in his work from early on.
>
> Jihoon in
> https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E and
> later Gian in this thread suggested that _every_ piece of work that should
> be labelled as "Design Review" according to the current rules should be
> accompanied by an issue. I don't agree with this, there are some PRs as
> small as a few dozens of lines of code, that add some configuration
> parameter and therefore should be labelled "Design Review". I don't thing a
> separate proposal issue is needed for them, and even for a little larger
> PRs too.
>
> For the same reason, I also don't see the point of renaming "Design
> Review" into "Proposal", as well as separating "Design Review" into
> "Proposal" and something like "API Review". I think a single "Design
> Review" tag handles it well.
>
> Gian mentioned an idea that PRs that follow a "Design Review" proposal
> issue shouldn't be "Design Review" themselves. I don't agree with this, I
> think that actual code and performance data are important inputs that
> should be re-evaluated at least by two people. I even think that it's very
> desirable that at least two people read _each line of production code_ in
> large PRs, although it's not what was done historically in Druid, because
> large bodies of newly added code, with whole new classes and subsystems
> added, are also coincidentally tested worse than already existing classes
> and subsystems, including in production. It seems to me that those huge
> code influxes is a major source of bugs, that could later take years to
> squeeze out from the codebase.
>
> On Wed, 9 Jan 2019 at 08:24, Clint Wylie <cl...@imply.io> wrote:
>
>> Apologies for the delayed response.
>>
>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <sl...@gmail.com>
>> wrote:
>>
>> > I am wondering here what is the case where code first is better?
>> >
>>
>> I don't think it's wrong to share ideas as early as possible, and after
>> this discussion I think I am in favor of it too. I just meant that I don't
>> think it's always necessarily the most productive discussion until code
>> exists sometimes, with the types of thing I am thinking of are almost
>> entirely limited to cases where things might sound good to anyone on paper
>> but in reality need a large amount of experiments conducted and
>> observations collected to determine that something is actually worth
>> doing,
>> which I imagine is mainly things like reworking internals for performance
>> improvements.
>>
>> In the case of my combined proposal PR, I needed to prove that the thing I
>> was working on was a good idea... and it wasn't directly. But I came up
>> with another idea during the course of experiment turned into something
>> compelling, so an initial proposal would have looked quite a lot different
>> than what I ended up with. Once I had proven to myself that it was a good
>> idea, then I was comfortable sharing with the wider community. I'm not
>> certain how this would play out in an always proposal first model, maybe
>> the first proposal exists, I personally reject it after updating with
>> experiment results show it's a bad idea, continue experimenting and raise
>> a
>> new one after the experiments start looking promising?
>>
>>
>> > Let's not be naive this is very rare that a contributor will accept that
>> > his work is to be thrown, usually devs takes coding as personal creation
>> > and they get attached to it.
>> >
>>
>> I agree, just because a handful of the committers have this attitude, it
>> isn't fair to expect the wider community to also, that's why I am in favor
>> of formalizing the process.
>>
>> Can you please explain what is overbearing ? what can be changed to make
>> it
>> > easy ?
>> > Most of the points are kind of the actual questions that you want to
>> > address before hand anyway isn't ?
>> >
>>
>> Sorry for the confusion, I said it's "not overbearing", I think it's fine.
>>
>> What are the complaints ?
>>
>>
>> Is this and other previous threads not a complaint about opening a large
>> PR
>> without a proposal? :) I just mean that formalizing the process, even if a
>> proposal has a reference PR opened with it near concurrently, could
>> prevent
>> these discussions from happening in the future because ground rules have
>> been set and we are all on the same page I guess.
>>
>> I think we should also probably consider renaming the "design review"
>> label
>> to "proposal" or something to make it more clear that a PR is associated
>> with a proposal. It might also be worth considering using github projects
>> for particularly large things that involve multiple follow up PRs, but I
>> haven't used them in much depth to know if they add anything.
>>
>> It seems like we are all converging on agreement to do a github issue
>> proposal for larger changes (I would vote for announcing them on the dev
>> list too for more visibility), so that design review is separated from
>> code
>> review. I guess my main concern was not wanting to discourage
>> experimentation by walling it off behind mandatory discussions, but the
>> more I think about it,  it doesn't technically change much about this
>> process, it just requires a more formal proposal to accompany any
>> experiments that are shared as a PR.
>>
>>
>> On Tue, Jan 8, 2019 at 12:28 PM Gian Merlino <gi...@apache.org> wrote:
>>
>> > I think for us, choosing to use GitHub issues as discussion threads for
>> > potential 'major' contributions would be a good idea, especially if we
>> > encourage people to start them before PRs show up. Definitely agree that
>> > all contributors should go through the same process -- I couldn't see it
>> > working well any other way.
>> >
>> > On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <jh...@apache.org> wrote:
>> >
>> > > Statically, yes, GitHub PRs are the same as GitHub cases. But
>> > dynamically,
>> > > they are different, because you can only log a PR when you have
>> finished
>> > > work.
>> > >
>> > > A lot of other Apache projects use JIRA, so there is a clear
>> distinction
>> > > between cases and contributions. JIRA cases, especially when logged
>> early
>> > > in the lifecycle of a contribution, become long-running conversation
>> > > threads with a lot of community participation. If the Druid chose to
>> do
>> > so,
>> > > GitHub cases could be the same.
>> > >
>> > > Be careful that you do not treat “potential contributors” (by which I
>> > > presume you mean non-committers) differently from committers and PMC
>> > > members. Anyone starting a major piece of work should follow the same
>> > > process. (Experienced committers probably have a somewhat better idea
>> > what
>> > > work will turn out to be “major”, so they get a little more leeway.)
>> > >
>> > > Julian
>> > >
>> > >
>> > > > On Jan 7, 2019, at 12:10 PM, Gian Merlino <gi...@apache.org> wrote:
>> > > >
>> > > > I don't think there's a need to raise issues for every change: a
>> small
>> > > bug
>> > > > fix or doc fix should just go straight to PR. (GitHub PRs show up as
>> > > issues
>> > > > in the issue-search UI/API, so it's not like this means the patch
>> has
>> > no
>> > > > corresponding issue -- in a sense the PR _is_ the issue.)
>> > > >
>> > > > I do think it makes sense to encourage potential contributors to
>> write
>> > to
>> > > > the dev list or raise an issue if they aren't sure if something
>> would
>> > > need
>> > > > to go through a more heavy weight process.
>> > > >
>> > > > Fwiw we do have a set of 'design review' criteria already (we had a
>> > > > discussion about this a couple years ago) at:
>> > > > http://druid.io/community/#getting-your-changes-accepted. So we
>> > > wouldn't be
>> > > > starting from zero on defining that. We set it up back when we were
>> > > trying
>> > > > to _streamline_ our process -- we used to require two non-author +1s
>> > for
>> > > > _every_ change, even minor ones. The introduction of design review
>> > > criteria
>> > > > was meant to classify which PRs need that level of review and which
>> > ones
>> > > > are minor and can be merged with less review. I do think it helped
>> with
>> > > > getting minor PRs merged more quickly. The list of criteria is,
>> > > >
>> > > > - Major architectural changes or API changes
>> > > > - HTTP requests and responses (e. g. a new HTTP endpoint)
>> > > > - Interfaces for extensions
>> > > > - Server configuration (e. g. altering the behavior of a config
>> > property)
>> > > > - Emitted metrics
>> > > > - Other major changes, judged by the discretion of Druid committers
>> > > >
>> > > > Some of it is subjective, but it has been in place for a while, so
>> it's
>> > > at
>> > > > least something we are relatively familiar with.
>> > > >
>> > > > On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <jh...@apache.org>
>> wrote:
>> > > >
>> > > >> Small contributions don’t need any design review, whereas large
>> > > >> contributions need significant review. I don’t think we should
>> require
>> > > an
>> > > >> additional step for those (many) small contributions. But who
>> decides
>> > > >> whether a contribution fits into the small or large category?
>> > > >>
>> > > >> I think the solution is for authors to log a case (or send an
>> email to
>> > > >> dev) before they start work on any contribution. Then committers
>> can
>> > > >> request a more heavy-weight process if they think it is needed.
>> > > >>
>> > > >> Julian
>> > > >>
>> > > >>
>> > > >>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <gi...@apache.org>
>> wrote:
>> > > >>>
>> > > >>> It sounds like splitting design from code review is a common theme
>> > in a
>> > > >> few
>> > > >>> of the posts here. How does everyone feel about making a point of
>> > > >>> encouraging design reviews to be done as issues, separate from the
>> > pull
>> > > >>> request, with the expectations that (1) the design review issue
>> > > >>> ("proposal") should generally appear somewhat _before_ the pull
>> > > request;
>> > > >>> (2) pull requests should _not_ have design review happen on them,
>> > > meaning
>> > > >>> there should no longer be PRs with design review tags, and we
>> should
>> > > move
>> > > >>> the design review approval process to the issue rather than the
>> PR.
>> > > >>>
>> > > >>> For (1), even if we encourage design review discussions to start
>> > > before a
>> > > >>> pull request appears, I don't see an issue with them running
>> > > concurrently
>> > > >>> for a while at some point.
>> > > >>>
>> > > >>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <jo...@apache.org>
>> > wrote:
>> > > >>>
>> > > >>>> Thanks for raising these concerns!
>> > > >>>>
>> > > >>>> My initial thoughts:
>> > > >>>> - I agree that separation of design review and code-level review
>> for
>> > > >> major
>> > > >>>> changes would be more efficient
>> > > >>>>
>> > > >>>> - I agree that a clear, more formalized process for handling
>> major
>> > > >> changes
>> > > >>>> would be helpful for contributors:
>> > > >>>> - Define what is considered a major change
>> > > >>>> - Define a standard proposal structure, KIP-style proposal format
>> > > >> sounds
>> > > >>>> good to me
>> > > >>>>
>> > > >>>> - I think it's too rigid to have a policy of "no code at all with
>> > the
>> > > >>>> initial proposal"
>> > > >>>> - Code samples can be useful references for understanding aspects
>> > of a
>> > > >>>> design
>> > > >>>> - In some cases it's necessary to run experiments to fully
>> > understand
>> > > a
>> > > >>>> problem and determine an appropriate design, or to determine
>> whether
>> > > >>>> something is even worth doing before committing to the work of
>> > > fleshing
>> > > >> out
>> > > >>>> a proposal, prototype code is a natural outcome of that and I'm
>> not
>> > > >> against
>> > > >>>> someone providing such code for reference
>> > > >>>> - I tend to view design/code as things that are often developed
>> > > >>>> simultaneously in an intertwined way
>> > > >>>>
>> > > >>>>> Let's not be naive this is very rare that a contributor will
>> accept
>> > > >> that
>> > > >>>> his work is to be thrown, usually devs takes coding as personal
>> > > creation
>> > > >>>> and they get attached to it.
>> > > >>>>
>> > > >>>> If we have a clear review process that emphasizes the need for
>> early
>> > > >>>> consensus building, with separate design and code review, then I
>> > feel
>> > > >> we've
>> > > >>>> done enough and don't need a hard rule against having some code
>> > linked
>> > > >> with
>> > > >>>> the initial proposal. If a potential contributor then still
>> wants to
>> > > go
>> > > >>>> ahead and write a lot of code that may be rejected or change
>> > > >> significantly,
>> > > >>>> the risks were made clear.
>> > > >>>>
>> > > >>>>> Once code is written hard to think abstract.
>> > > >>>>
>> > > >>>> I can see the validity of the concern, but I personally don't
>> see it
>> > > as
>> > > >> a
>> > > >>>> huge risk. My impression from the Druid PR reviews I've seen is
>> that
>> > > our
>> > > >>>> reviewers are able to keep abstract design vs. implementation
>> > details
>> > > >>>> separate and consider alternate designs when reviewing.
>> > > >>>>
>> > > >>>> To summarize I think it's probably enough to have a policy along
>> the
>> > > >> lines
>> > > >>>> of:
>> > > >>>> - Create more formalized guidelines for proposals and what
>> changes
>> > > >> require
>> > > >>>> proposals
>> > > >>>> - Separate design and code review for major changes, with design
>> > > review
>> > > >>>> first, code-level review after reaching consensus on the design.
>> > > >>>> - Code before the design review is completed is just for
>> reference,
>> > > not
>> > > >>>> regarded as a candidate for review/merging.
>> > > >>>>
>> > > >>>> - Jon
>> > > >>>>
>> > > >>>>
>> > > >>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
>> > > >> slim.bouguerra@gmail.com>
>> > > >>>> wrote:
>> > > >>>>
>> > > >>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <
>> clint.wylie@imply.io>
>> > > >>>> wrote:
>> > > >>>>>
>> > > >>>>>> I am definitely biased in this matter as an owner of another
>> large
>> > > PR
>> > > >>>>> that
>> > > >>>>>> wasn't preceded by a direct proposal or dev list discussion,
>> and
>> > in
>> > > >>>>> general
>> > > >>>>>> I agree that proposal first is usually better, but I think in
>> some
>> > > >>>> rarer
>> > > >>>>>> cases approaching a problem code first *is* the most
>> appropriate
>> > way
>> > > >> to
>> > > >>>>>> have a discussion.
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> I am wondering here what is the case where code first is better?
>> > > >>>>> In general when you are writing code you have an idea about what
>> > you
>> > > >> want
>> > > >>>>> to change, why you want to change and why you want to change it.
>> > > >>>>> I do not see what is wrong with sharing this primitive ideas and
>> > > >> thoughts
>> > > >>>>> as an abstract proposal (at least to avoid overlapping)
>> > > >>>>>
>> > > >>>>> I see nothing wrong with it so long as the author
>> > > >>>>>> accepts that the PR is treated as a combined proposal and
>> proof of
>> > > >>>>> concept,
>> > > >>>>>> and fair game to be radically changed via discussion or even
>> > > rejected,
>> > > >>>>>> which sounds like Gian's attitude on the matter and is mine as
>> > well
>> > > >>>> with
>> > > >>>>> my
>> > > >>>>>> compression stuff.
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> Let's not be naive this is very rare that a contributor will
>> accept
>> > > >> that
>> > > >>>>> his work is to be thrown, usually devs takes coding as personal
>> > > >> creation
>> > > >>>>> and they get attached to it.
>> > > >>>>> To my point you can take a look on some old issue in the Druid
>> > forum
>> > > >>>>>
>> > > >>>>
>> > > >>
>> > >
>> >
>> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
>> > > >>>>> and am sure other communities have similar problems.
>> > > >>>>> So leaving the door open to some side cases is not a good idea
>> in
>> > my
>> > > >>>>> opinion and will lead to similar issue in the future.
>> > > >>>>>
>> > > >>>>> This seems to me especially likely to happen in cases
>> > > >>>>>> where an approach still needs proven to be a viable idea *to
>> the
>> > > >>>> author*,
>> > > >>>>>> so that a much more productive discussion can be had in the
>> first
>> > > >>>> place.
>> > > >>>>>>
>> > > >>>>>> I think there is a trade off, I don't think we want to
>> discourage
>> > > >>>>>> experimentation by walling it off behind mandatory discussions
>> > > before
>> > > >>>> it
>> > > >>>>>> can even start, but I do think formalizing the process for
>> large
>> > > >>>> changes
>> > > >>>>> is
>> > > >>>>>> a good thing, especially since we probably can't expect the
>> wider
>> > > >>>>> community
>> > > >>>>>> to have the same attitude towards a large PR getting discarded
>> as
>> > a
>> > > >>>>>> committer might. I think the Kafka approach is reasonable, a
>> bit
>> > > more
>> > > >>>>>> formal than our design review process but not overbearing.
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> Can you please explain what is overbearing ? what can be
>> changed to
>> > > >> make
>> > > >>>> it
>> > > >>>>> easy ?
>> > > >>>>> Most of the points are kind of the actual questions that you
>> want
>> > to
>> > > >>>>> address before hand anyway isn't ?
>> > > >>>>>
>> > > >>>>>
>> > > >>>>>> Going code first
>> > > >>>>>> should be in general discouraged, but when it does happen, it
>> > seems
>> > > >>>> like
>> > > >>>>>> opening DIP/an issue/starting a mailing list thread or
>> whatever we
>> > > go
>> > > >>>>> with
>> > > >>>>>> to have a more high level design discussion alongside the
>> > reference
>> > > PR
>> > > >>>>>> could alleviate some of these complaints?
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> What are the complaints ?
>> > > >>>>>
>> > > >>>>>
>> > > >>>>>> +1 for "DIP" heh, I think making
>> > > >>>>>> them in the form of github issues is probably appropriate,
>> with a
>> > > dev
>> > > >>>>> list
>> > > >>>>>> thread to announce them perhaps?
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>> I think  github issue with [Proposal] header like
>> > > >>>>> https://github.com/apache/incubator-druid/issues/4349 is good
>> to
>> > me,
>> > > >>>>>
>> > > >>>>> Thanks!
>> > > >>>>>
>> > > >>>>>
>> > > >>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <
>> bslim@apache.org>
>> > > >>>> wrote:
>> > > >>>>>>
>> > > >>>>>>> Thanks everyone for interacting with this thread.
>> > > >>>>>>>
>> > > >>>>>>> The fact that i, Roman, Jihoon  and others in the past (FJ
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>
>> > >
>> >
>> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ
>> )
>> > > >>>>>>> raised this point indicates that PRs without a proposal are
>> > indeed
>> > > an
>> > > >>>>>> issue
>> > > >>>>>>> and we need to solve it.
>> > > >>>>>>>
>> > > >>>>>>> Something Similar to KIP maybe called DIPs is fine with me.
>> > > >>>>>>> What i strive to see is the following:
>> > > >>>>>>>
>> > > >>>>>>> [Step 1] formalize what is the kind of work that needs a
>> formal
>> > > >>>>>> Proposal, I
>> > > >>>>>>> think Roman and Jihoon has already covered that pretty well.
>> am
>> > +1
>> > > on
>> > > >>>>>> that.
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>
>> > >
>> >
>> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
>> > > >>>>>>>
>> > > >>>>>>> I am strongly in favor of the separation of Proposal Review
>> and
>> > > >>>> (later)
>> > > >>>>>>> Code review PRs. My  main reasons:
>> > > >>>>>>> Most importantly code reviewing will introduce lot of noise
>> and
>> > > will
>> > > >>>>>>> ultimately make  the GitHub page unreadable.
>> > > >>>>>>> Avoid overlapping of work.
>> > > >>>>>>> Once code is written hard to think abstract.
>> > > >>>>>>> Separate page for Design review later can always be used it
>> as a
>> > > >>>> Design
>> > > >>>>>>> document that is readable and code free-ish.
>> > > >>>>>>> As i said the goal of this first round is to see if the
>> community
>> > > >>>> agree
>> > > >>>>>>> about such change, then make the process of design more
>> inclusive
>> > > >>>> thus
>> > > >>>>>>> other contributors can submit a counter proposals.
>> > > >>>>>>>
>> > > >>>>>>> [Step 2] IF everybody agree about that point Step 2 is to
>> define
>> > > >>>> which
>> > > >>>>>>> medium is used to Publish a primitive form of a CODE FREE
>> > Abstract
>> > > >>>>>> Proposal
>> > > >>>>>>> containing at least the following bullet points.
>> > > >>>>>>> - The problem description and motivation
>> > > >>>>>>> - Overview of the proposed change
>> > > >>>>>>> - Operational impact (compatibility/ plans to upgrades) public
>> > API
>> > > >>>>>> changes,
>> > > >>>>>>> configuration changes, algorithm, and so on
>> > > >>>>>>> - Expected benefits and drawbacks
>> > > >>>>>>> - Rationale and alternatives
>> > > >>>>>>> - Estimate Time to Deliver if possible.
>> > > >>>>>>>
>> > > >>>>>>> The way i think this can be is a Github issue where member of
>> the
>> > > >>>>>> community
>> > > >>>>>>> will interact via comments and the author will be updating the
>> > > >>>>>> description
>> > > >>>>>>> in the light of comments provided by the community.
>> > > >>>>>>>
>> > > >>>>>>> During and near the end of the design discussions the author/s
>> > can
>> > > >>>>> start
>> > > >>>>>>> writing POCs to help guide the review process this naturally
>> will
>> > > be
>> > > >>>> a
>> > > >>>>>> Pull
>> > > >>>>>>> request with actual code.
>> > > >>>>>>>
>> > > >>>>>>> *Now the most important thing is that we need to agree that
>> any
>> > > work
>> > > >>>>> that
>> > > >>>>>>> does not align with this formal process will be ignored and
>> the
>> > > >>>> author
>> > > >>>>>> will
>> > > >>>>>>> be asked to start with a DIP*
>> > > >>>>>>> *That is what i meant with  “If it didn’t happen on the
>> mailing
>> > > list,
>> > > >>>>> it
>> > > >>>>>>> didn’t happen.”*
>> > > >>>>>>>
>> > > >>>>>>> Thanks and happy coding!
>> > > >>>>>>>
>> > > >>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <gi...@apache.org>
>> > > wrote:
>> > > >>>>>>>
>> > > >>>>>>>> One of the advantages I see with a more formal process is
>> (like
>> > > >>>> Kafka
>> > > >>>>>>> KIPs)
>> > > >>>>>>>> is that it levels the playing field a bit and sets some
>> ground
>> > > >>>> rules
>> > > >>>>>> for
>> > > >>>>>>>> working together. In a way it can help encourage
>> contributions
>> > by
>> > > >>>>>> making
>> > > >>>>>>> it
>> > > >>>>>>>> clear what is expected of potential contributors.
>> > > >>>>>>>>
>> > > >>>>>>>> We have a design review process today that is not as formal
>> as
>> > > >>>> KIPs,
>> > > >>>>>> but
>> > > >>>>>>> is
>> > > >>>>>>>> somewhat heavier than the one you describe. Maybe we could
>> tweak
>> > > >>>> our
>> > > >>>>>>>> current one by starting to do design reviews separately from
>> > PRs.
>> > > >>>>> i.e.,
>> > > >>>>>>> for
>> > > >>>>>>>> anything that meets our 'design review' criteria, do that on
>> the
>> > > >>>> dev
>> > > >>>>>> list
>> > > >>>>>>>> or in a separate issue, and keep the PR focused on code-level
>> > > >>>> stuff.
>> > > >>>>>> That
>> > > >>>>>>>> way we don't end up trying to do both at once. And it makes
>> it
>> > > >>>> easier
>> > > >>>>>> to
>> > > >>>>>>>> start talking about design before the code is ready, which
>> would
>> > > be
>> > > >>>>>>> better.
>> > > >>>>>>>>
>> > > >>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <jhyde@apache.org
>> >
>> > > >>>> wrote:
>> > > >>>>>>>>
>> > > >>>>>>>>> It’s really hard to say no to a contribution when someone
>> has
>> > put
>> > > >>>>> in
>> > > >>>>>> a
>> > > >>>>>>>>> significant amount of work.
>> > > >>>>>>>>>
>> > > >>>>>>>>> The following approach is simple and works really well:
>> Before
>> > > >>>> you
>> > > >>>>>>> start
>> > > >>>>>>>>> work, log a case, describing the problem. When you have some
>> > > >>>> ideas
>> > > >>>>>>> about
>> > > >>>>>>>>> design, add those to the case. When you have a code branch,
>> add
>> > > >>>> its
>> > > >>>>>> URL
>> > > >>>>>>>> to
>> > > >>>>>>>>> the case. And so forth. At any point in the proceedings,
>> people
>> > > >>>> can
>> > > >>>>>>> chime
>> > > >>>>>>>>> in with their opinions.
>> > > >>>>>>>>>
>> > > >>>>>>>>> In my opinion, a formal “design review” process is not
>> > necessary.
>> > > >>>>>> Just
>> > > >>>>>>>>> build consensus iteratively, by starting the conversation
>> early
>> > > >>>> in
>> > > >>>>>> the
>> > > >>>>>>>>> process.
>> > > >>>>>>>>>
>> > > >>>>>>>>> Julian
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <gian@apache.org
>> >
>> > > >>>>> wrote:
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> In this particular case: please consider the PR as a
>> proposal.
>> > > >>>>>> Don't
>> > > >>>>>>>> feel
>> > > >>>>>>>>>> like just because there is code there that takes a certain
>> > > >>>>>> approach,
>> > > >>>>>>>> that
>> > > >>>>>>>>>> the approach is somehow sacred. I had to implement
>> something
>> > to
>> > > >>>>>>>>> crystallize
>> > > >>>>>>>>>> my own thinking about how the problem could be approached.
>> I
>> > > >>>>> won't
>> > > >>>>>> be
>> > > >>>>>>>>>> disappointed if, as a community, we decide a different
>> > > >>>> direction
>> > > >>>>> is
>> > > >>>>>>>>> better
>> > > >>>>>>>>>> and the code all gets thrown away. That's one of the
>> reasons
>> > > >>>>> that I
>> > > >>>>>>>>> removed
>> > > >>>>>>>>>> the 0.14.0 milestone that was added to the patch. (I don't
>> > want
>> > > >>>>> to
>> > > >>>>>>> rush
>> > > >>>>>>>>> it,
>> > > >>>>>>>>>> nor do I think that's a good idea.)
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> In general: Sounds like we could do with some more
>> > > >>>> formalization
>> > > >>>>>>> around
>> > > >>>>>>>>>> what a proposal looks like, which sorts of changes need
>> one,
>> > > >>>> and
>> > > >>>>>> when
>> > > >>>>>>>> in
>> > > >>>>>>>>>> the dev cycle it is appropriate. FWIW I think Kafka's
>> process
>> > > >>>> is
>> > > >>>>>> more
>> > > >>>>>>>> or
>> > > >>>>>>>>>> less fine, and would be okay with adopting it for Druid if
>> > > >>>> people
>> > > >>>>>>> like
>> > > >>>>>>>>> it.
>> > > >>>>>>>>>> Right now our standards for what requires a "design review"
>> > are
>> > > >>>>>> very
>> > > >>>>>>>>>> similar to the Kafka community standards for what requires
>> a
>> > > >>>> KIP,
>> > > >>>>>> so
>> > > >>>>>>> we
>> > > >>>>>>>>>> have some familiarity with those concepts. However we don't
>> > > >>>>>> separate
>> > > >>>>>>> PR
>> > > >>>>>>>>>> review and proposal discussion as strictly as they do,
>> which
>> > > >>>>> seems
>> > > >>>>>> to
>> > > >>>>>>>> be
>> > > >>>>>>>>>> the foundation for the feeling of exclusion that is being
>> felt
>> > > >>>>>> here.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Separately: I just redid the description on
>> > > >>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794 to be
>> > more
>> > > >>>>>>>>> proposal-y.
>> > > >>>>>>>>>> I followed the KIP style:
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>> > > >>>>>>>>> .
>> > > >>>>>>>>>> Please refresh the page and see if it looks more useful.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Gian
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <
>> jhyde@apache.org
>> > >
>> > > >>>>>>> wrote:
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> Slim,
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> I agree with your points that offline development is bad
>> for
>> > > >>>>>>>> community.
>> > > >>>>>>>>>>> But I don’t think you need much mentor help. You have
>> raised
>> > > >>>>> valid
>> > > >>>>>>>>> issues
>> > > >>>>>>>>>>> and the Druid community needs to decide what its
>> development
>> > > >>>>>>> practices
>> > > >>>>>>>>>>> should be.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Julian
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
>> > > >>>> bslim@apache.org>
>> > > >>>>>>>> wrote:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Hello everyone and hope you all have very good holidays.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> First, this email is not directed on the author or the PR
>> > > >>>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794  it
>> > > >>>> self,
>> > > >>>>>> but
>> > > >>>>>>> i
>> > > >>>>>>>>> see
>> > > >>>>>>>>>>>> this PR as a perfect example.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> One of the foundation of Apache Way or what i would
>> simply
>> > > >>>> call
>> > > >>>>>>> open
>> > > >>>>>>>>>>> source
>> > > >>>>>>>>>>>> community driven development is that "Technical decisions
>> > are
>> > > >>>>>>>>> discussed,
>> > > >>>>>>>>>>>> decided, and archived publicly.
>> > > >>>>>>>>>>>> developpement"
>> > > >>>>>>>>>>>> Which means that big technical  changes such as the one
>> > > >>>> brought
>> > > >>>>>> by
>> > > >>>>>>>>> #/6794
>> > > >>>>>>>>>>>> should have started as a proposal and round of
>> discussions
>> > > >>>>> about
>> > > >>>>>>> the
>> > > >>>>>>>>>>> major
>> > > >>>>>>>>>>>> changes designs not as 11K line of code.
>> > > >>>>>>>>>>>> I believe such openness will promote a lot of good
>> benefits
>> > > >>>>> such
>> > > >>>>>>> as:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> - ensures community health and growth.
>> > > >>>>>>>>>>>> - ensures everyone can participate not only the authors
>> and
>> > > >>>> his
>> > > >>>>>>>>>>> co-workers.
>> > > >>>>>>>>>>>> - ensures that the project is driven by the community and
>> > > >>>> not a
>> > > >>>>>>> given
>> > > >>>>>>>>>>>> company or an individual.
>> > > >>>>>>>>>>>> - ensures that there is consensus (not saying 100%
>> > > >>>> agreement;)
>> > > >>>>>>>> however
>> > > >>>>>>>>> it
>> > > >>>>>>>>>>>> means that all individuals will accept the current
>> progress
>> > > >>>> on
>> > > >>>>>> the
>> > > >>>>>>>>>>> project
>> > > >>>>>>>>>>>> until some better proposal is put forth.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Personally such BIG offline PR makes me feel excluded and
>> > > >>>>> doesn't
>> > > >>>>>>>> give
>> > > >>>>>>>>>>> me a
>> > > >>>>>>>>>>>> sense that i belong to  a community at all.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> To prevent such off list development i think as a Druid
>> > > >>>>> Community
>> > > >>>>>>> we
>> > > >>>>>>>>> need
>> > > >>>>>>>>>>>> to stick to the apache way “If it didn’t happen on the
>> > > >>>> mailing
>> > > >>>>>>> list,
>> > > >>>>>>>> it
>> > > >>>>>>>>>>>> didn’t happen.”
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> I would appreciate if some of the Apache mentor help with
>> > > >>>> this.
>> > > >>>>>>>>>>>> Thanks
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>
>> > > ---------------------------------------------------------------------
>> > > >>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>> > > >>>>>>>>>>> For additional commands, e-mail:
>> dev-help@druid.apache.org
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>
>> > ---------------------------------------------------------------------
>> > > >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>> > > >>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> --
>> > > >>>>>
>> > > >>>>> B-Slim
>> > > >>>>>
>> > > >>
>> > _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
>> > > >>>>>
>> > > >>>>
>> > > >>
>> > > >>
>> > > >>
>> ---------------------------------------------------------------------
>> > > >> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>> > > >> For additional commands, e-mail: dev-help@druid.apache.org
>> > > >>
>> > > >>
>> > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>> > > For additional commands, e-mail: dev-help@druid.apache.org
>> > >
>> > >
>> >
>>
>

Re: Off list major development

[Roman Leventov <le...@gmail.com>]

I see two important reasons why it makes sense to file an issue and
probably announce it in the mailing list, before writing a lot of code,
despite not having a clear picture of what it will be and any performance
data:
 1) somebody could already work on this problem privately in parallel, it
allows to avoid clash in people's works
 2) some people could quickly think about the problem field and share
high-level ideas that could wildly change the direction in which the author
(the person who is going to write code) will move in his work from early on.

Jihoon in
https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E
and
later Gian in this thread suggested that _every_ piece of work that should
be labelled as "Design Review" according to the current rules should be
accompanied by an issue. I don't agree with this, there are some PRs as
small as a few dozens of lines of code, that add some configuration
parameter and therefore should be labelled "Design Review". I don't thing a
separate proposal issue is needed for them, and even for a little larger
PRs too.

For the same reason, I also don't see the point of renaming "Design Review"
into "Proposal", as well as separating "Design Review" into "Proposal" and
something like "API Review". I think a single "Design Review" tag handles
it well.

Gian mentioned an idea that PRs that follow a "Design Review" proposal
issue shouldn't be "Design Review" themselves. I don't agree with this, I
think that actual code and performance data are important inputs that
should be re-evaluated at least by two people. I even think that it's very
desirable that at least two people read _each line of production code_ in
large PRs, although it's not what was done historically in Druid, because
large bodies of newly added code, with whole new classes and subsystems
added, are also coincidentally tested worse than already existing classes
and subsystems, including in production. It seems to me that those huge
code influxes is a major source of bugs, that could later take years to
squeeze out from the codebase.

On Wed, 9 Jan 2019 at 08:24, Clint Wylie <cl...@imply.io> wrote:

> Apologies for the delayed response.
>
> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <sl...@gmail.com>
> wrote:
>
> > I am wondering here what is the case where code first is better?
> >
>
> I don't think it's wrong to share ideas as early as possible, and after
> this discussion I think I am in favor of it too. I just meant that I don't
> think it's always necessarily the most productive discussion until code
> exists sometimes, with the types of thing I am thinking of are almost
> entirely limited to cases where things might sound good to anyone on paper
> but in reality need a large amount of experiments conducted and
> observations collected to determine that something is actually worth doing,
> which I imagine is mainly things like reworking internals for performance
> improvements.
>
> In the case of my combined proposal PR, I needed to prove that the thing I
> was working on was a good idea... and it wasn't directly. But I came up
> with another idea during the course of experiment turned into something
> compelling, so an initial proposal would have looked quite a lot different
> than what I ended up with. Once I had proven to myself that it was a good
> idea, then I was comfortable sharing with the wider community. I'm not
> certain how this would play out in an always proposal first model, maybe
> the first proposal exists, I personally reject it after updating with
> experiment results show it's a bad idea, continue experimenting and raise a
> new one after the experiments start looking promising?
>
>
> > Let's not be naive this is very rare that a contributor will accept that
> > his work is to be thrown, usually devs takes coding as personal creation
> > and they get attached to it.
> >
>
> I agree, just because a handful of the committers have this attitude, it
> isn't fair to expect the wider community to also, that's why I am in favor
> of formalizing the process.
>
> Can you please explain what is overbearing ? what can be changed to make it
> > easy ?
> > Most of the points are kind of the actual questions that you want to
> > address before hand anyway isn't ?
> >
>
> Sorry for the confusion, I said it's "not overbearing", I think it's fine.
>
> What are the complaints ?
>
>
> Is this and other previous threads not a complaint about opening a large PR
> without a proposal? :) I just mean that formalizing the process, even if a
> proposal has a reference PR opened with it near concurrently, could prevent
> these discussions from happening in the future because ground rules have
> been set and we are all on the same page I guess.
>
> I think we should also probably consider renaming the "design review" label
> to "proposal" or something to make it more clear that a PR is associated
> with a proposal. It might also be worth considering using github projects
> for particularly large things that involve multiple follow up PRs, but I
> haven't used them in much depth to know if they add anything.
>
> It seems like we are all converging on agreement to do a github issue
> proposal for larger changes (I would vote for announcing them on the dev
> list too for more visibility), so that design review is separated from code
> review. I guess my main concern was not wanting to discourage
> experimentation by walling it off behind mandatory discussions, but the
> more I think about it,  it doesn't technically change much about this
> process, it just requires a more formal proposal to accompany any
> experiments that are shared as a PR.
>
>
> On Tue, Jan 8, 2019 at 12:28 PM Gian Merlino <gi...@apache.org> wrote:
>
> > I think for us, choosing to use GitHub issues as discussion threads for
> > potential 'major' contributions would be a good idea, especially if we
> > encourage people to start them before PRs show up. Definitely agree that
> > all contributors should go through the same process -- I couldn't see it
> > working well any other way.
> >
> > On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <jh...@apache.org> wrote:
> >
> > > Statically, yes, GitHub PRs are the same as GitHub cases. But
> > dynamically,
> > > they are different, because you can only log a PR when you have
> finished
> > > work.
> > >
> > > A lot of other Apache projects use JIRA, so there is a clear
> distinction
> > > between cases and contributions. JIRA cases, especially when logged
> early
> > > in the lifecycle of a contribution, become long-running conversation
> > > threads with a lot of community participation. If the Druid chose to do
> > so,
> > > GitHub cases could be the same.
> > >
> > > Be careful that you do not treat “potential contributors” (by which I
> > > presume you mean non-committers) differently from committers and PMC
> > > members. Anyone starting a major piece of work should follow the same
> > > process. (Experienced committers probably have a somewhat better idea
> > what
> > > work will turn out to be “major”, so they get a little more leeway.)
> > >
> > > Julian
> > >
> > >
> > > > On Jan 7, 2019, at 12:10 PM, Gian Merlino <gi...@apache.org> wrote:
> > > >
> > > > I don't think there's a need to raise issues for every change: a
> small
> > > bug
> > > > fix or doc fix should just go straight to PR. (GitHub PRs show up as
> > > issues
> > > > in the issue-search UI/API, so it's not like this means the patch has
> > no
> > > > corresponding issue -- in a sense the PR _is_ the issue.)
> > > >
> > > > I do think it makes sense to encourage potential contributors to
> write
> > to
> > > > the dev list or raise an issue if they aren't sure if something would
> > > need
> > > > to go through a more heavy weight process.
> > > >
> > > > Fwiw we do have a set of 'design review' criteria already (we had a
> > > > discussion about this a couple years ago) at:
> > > > http://druid.io/community/#getting-your-changes-accepted. So we
> > > wouldn't be
> > > > starting from zero on defining that. We set it up back when we were
> > > trying
> > > > to _streamline_ our process -- we used to require two non-author +1s
> > for
> > > > _every_ change, even minor ones. The introduction of design review
> > > criteria
> > > > was meant to classify which PRs need that level of review and which
> > ones
> > > > are minor and can be merged with less review. I do think it helped
> with
> > > > getting minor PRs merged more quickly. The list of criteria is,
> > > >
> > > > - Major architectural changes or API changes
> > > > - HTTP requests and responses (e. g. a new HTTP endpoint)
> > > > - Interfaces for extensions
> > > > - Server configuration (e. g. altering the behavior of a config
> > property)
> > > > - Emitted metrics
> > > > - Other major changes, judged by the discretion of Druid committers
> > > >
> > > > Some of it is subjective, but it has been in place for a while, so
> it's
> > > at
> > > > least something we are relatively familiar with.
> > > >
> > > > On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <jh...@apache.org>
> wrote:
> > > >
> > > >> Small contributions don’t need any design review, whereas large
> > > >> contributions need significant review. I don’t think we should
> require
> > > an
> > > >> additional step for those (many) small contributions. But who
> decides
> > > >> whether a contribution fits into the small or large category?
> > > >>
> > > >> I think the solution is for authors to log a case (or send an email
> to
> > > >> dev) before they start work on any contribution. Then committers can
> > > >> request a more heavy-weight process if they think it is needed.
> > > >>
> > > >> Julian
> > > >>
> > > >>
> > > >>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <gi...@apache.org> wrote:
> > > >>>
> > > >>> It sounds like splitting design from code review is a common theme
> > in a
> > > >> few
> > > >>> of the posts here. How does everyone feel about making a point of
> > > >>> encouraging design reviews to be done as issues, separate from the
> > pull
> > > >>> request, with the expectations that (1) the design review issue
> > > >>> ("proposal") should generally appear somewhat _before_ the pull
> > > request;
> > > >>> (2) pull requests should _not_ have design review happen on them,
> > > meaning
> > > >>> there should no longer be PRs with design review tags, and we
> should
> > > move
> > > >>> the design review approval process to the issue rather than the PR.
> > > >>>
> > > >>> For (1), even if we encourage design review discussions to start
> > > before a
> > > >>> pull request appears, I don't see an issue with them running
> > > concurrently
> > > >>> for a while at some point.
> > > >>>
> > > >>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <jo...@apache.org>
> > wrote:
> > > >>>
> > > >>>> Thanks for raising these concerns!
> > > >>>>
> > > >>>> My initial thoughts:
> > > >>>> - I agree that separation of design review and code-level review
> for
> > > >> major
> > > >>>> changes would be more efficient
> > > >>>>
> > > >>>> - I agree that a clear, more formalized process for handling major
> > > >> changes
> > > >>>> would be helpful for contributors:
> > > >>>> - Define what is considered a major change
> > > >>>> - Define a standard proposal structure, KIP-style proposal format
> > > >> sounds
> > > >>>> good to me
> > > >>>>
> > > >>>> - I think it's too rigid to have a policy of "no code at all with
> > the
> > > >>>> initial proposal"
> > > >>>> - Code samples can be useful references for understanding aspects
> > of a
> > > >>>> design
> > > >>>> - In some cases it's necessary to run experiments to fully
> > understand
> > > a
> > > >>>> problem and determine an appropriate design, or to determine
> whether
> > > >>>> something is even worth doing before committing to the work of
> > > fleshing
> > > >> out
> > > >>>> a proposal, prototype code is a natural outcome of that and I'm
> not
> > > >> against
> > > >>>> someone providing such code for reference
> > > >>>> - I tend to view design/code as things that are often developed
> > > >>>> simultaneously in an intertwined way
> > > >>>>
> > > >>>>> Let's not be naive this is very rare that a contributor will
> accept
> > > >> that
> > > >>>> his work is to be thrown, usually devs takes coding as personal
> > > creation
> > > >>>> and they get attached to it.
> > > >>>>
> > > >>>> If we have a clear review process that emphasizes the need for
> early
> > > >>>> consensus building, with separate design and code review, then I
> > feel
> > > >> we've
> > > >>>> done enough and don't need a hard rule against having some code
> > linked
> > > >> with
> > > >>>> the initial proposal. If a potential contributor then still wants
> to
> > > go
> > > >>>> ahead and write a lot of code that may be rejected or change
> > > >> significantly,
> > > >>>> the risks were made clear.
> > > >>>>
> > > >>>>> Once code is written hard to think abstract.
> > > >>>>
> > > >>>> I can see the validity of the concern, but I personally don't see
> it
> > > as
> > > >> a
> > > >>>> huge risk. My impression from the Druid PR reviews I've seen is
> that
> > > our
> > > >>>> reviewers are able to keep abstract design vs. implementation
> > details
> > > >>>> separate and consider alternate designs when reviewing.
> > > >>>>
> > > >>>> To summarize I think it's probably enough to have a policy along
> the
> > > >> lines
> > > >>>> of:
> > > >>>> - Create more formalized guidelines for proposals and what changes
> > > >> require
> > > >>>> proposals
> > > >>>> - Separate design and code review for major changes, with design
> > > review
> > > >>>> first, code-level review after reaching consensus on the design.
> > > >>>> - Code before the design review is completed is just for
> reference,
> > > not
> > > >>>> regarded as a candidate for review/merging.
> > > >>>>
> > > >>>> - Jon
> > > >>>>
> > > >>>>
> > > >>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > > >> slim.bouguerra@gmail.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <
> clint.wylie@imply.io>
> > > >>>> wrote:
> > > >>>>>
> > > >>>>>> I am definitely biased in this matter as an owner of another
> large
> > > PR
> > > >>>>> that
> > > >>>>>> wasn't preceded by a direct proposal or dev list discussion, and
> > in
> > > >>>>> general
> > > >>>>>> I agree that proposal first is usually better, but I think in
> some
> > > >>>> rarer
> > > >>>>>> cases approaching a problem code first *is* the most appropriate
> > way
> > > >> to
> > > >>>>>> have a discussion.
> > > >>>>>
> > > >>>>>
> > > >>>>> I am wondering here what is the case where code first is better?
> > > >>>>> In general when you are writing code you have an idea about what
> > you
> > > >> want
> > > >>>>> to change, why you want to change and why you want to change it.
> > > >>>>> I do not see what is wrong with sharing this primitive ideas and
> > > >> thoughts
> > > >>>>> as an abstract proposal (at least to avoid overlapping)
> > > >>>>>
> > > >>>>> I see nothing wrong with it so long as the author
> > > >>>>>> accepts that the PR is treated as a combined proposal and proof
> of
> > > >>>>> concept,
> > > >>>>>> and fair game to be radically changed via discussion or even
> > > rejected,
> > > >>>>>> which sounds like Gian's attitude on the matter and is mine as
> > well
> > > >>>> with
> > > >>>>> my
> > > >>>>>> compression stuff.
> > > >>>>>
> > > >>>>>
> > > >>>>> Let's not be naive this is very rare that a contributor will
> accept
> > > >> that
> > > >>>>> his work is to be thrown, usually devs takes coding as personal
> > > >> creation
> > > >>>>> and they get attached to it.
> > > >>>>> To my point you can take a look on some old issue in the Druid
> > forum
> > > >>>>>
> > > >>>>
> > > >>
> > >
> >
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> > > >>>>> and am sure other communities have similar problems.
> > > >>>>> So leaving the door open to some side cases is not a good idea in
> > my
> > > >>>>> opinion and will lead to similar issue in the future.
> > > >>>>>
> > > >>>>> This seems to me especially likely to happen in cases
> > > >>>>>> where an approach still needs proven to be a viable idea *to the
> > > >>>> author*,
> > > >>>>>> so that a much more productive discussion can be had in the
> first
> > > >>>> place.
> > > >>>>>>
> > > >>>>>> I think there is a trade off, I don't think we want to
> discourage
> > > >>>>>> experimentation by walling it off behind mandatory discussions
> > > before
> > > >>>> it
> > > >>>>>> can even start, but I do think formalizing the process for large
> > > >>>> changes
> > > >>>>> is
> > > >>>>>> a good thing, especially since we probably can't expect the
> wider
> > > >>>>> community
> > > >>>>>> to have the same attitude towards a large PR getting discarded
> as
> > a
> > > >>>>>> committer might. I think the Kafka approach is reasonable, a bit
> > > more
> > > >>>>>> formal than our design review process but not overbearing.
> > > >>>>>
> > > >>>>>
> > > >>>>> Can you please explain what is overbearing ? what can be changed
> to
> > > >> make
> > > >>>> it
> > > >>>>> easy ?
> > > >>>>> Most of the points are kind of the actual questions that you want
> > to
> > > >>>>> address before hand anyway isn't ?
> > > >>>>>
> > > >>>>>
> > > >>>>>> Going code first
> > > >>>>>> should be in general discouraged, but when it does happen, it
> > seems
> > > >>>> like
> > > >>>>>> opening DIP/an issue/starting a mailing list thread or whatever
> we
> > > go
> > > >>>>> with
> > > >>>>>> to have a more high level design discussion alongside the
> > reference
> > > PR
> > > >>>>>> could alleviate some of these complaints?
> > > >>>>>
> > > >>>>>
> > > >>>>> What are the complaints ?
> > > >>>>>
> > > >>>>>
> > > >>>>>> +1 for "DIP" heh, I think making
> > > >>>>>> them in the form of github issues is probably appropriate, with
> a
> > > dev
> > > >>>>> list
> > > >>>>>> thread to announce them perhaps?
> > > >>>>>>
> > > >>>>>
> > > >>>>> I think  github issue with [Proposal] header like
> > > >>>>> https://github.com/apache/incubator-druid/issues/4349 is good to
> > me,
> > > >>>>>
> > > >>>>> Thanks!
> > > >>>>>
> > > >>>>>
> > > >>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <
> bslim@apache.org>
> > > >>>> wrote:
> > > >>>>>>
> > > >>>>>>> Thanks everyone for interacting with this thread.
> > > >>>>>>>
> > > >>>>>>> The fact that i, Roman, Jihoon  and others in the past (FJ
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > >
> >
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ)
> > > >>>>>>> raised this point indicates that PRs without a proposal are
> > indeed
> > > an
> > > >>>>>> issue
> > > >>>>>>> and we need to solve it.
> > > >>>>>>>
> > > >>>>>>> Something Similar to KIP maybe called DIPs is fine with me.
> > > >>>>>>> What i strive to see is the following:
> > > >>>>>>>
> > > >>>>>>> [Step 1] formalize what is the kind of work that needs a formal
> > > >>>>>> Proposal, I
> > > >>>>>>> think Roman and Jihoon has already covered that pretty well. am
> > +1
> > > on
> > > >>>>>> that.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> > > >>>>>>>
> > > >>>>>>> I am strongly in favor of the separation of Proposal Review and
> > > >>>> (later)
> > > >>>>>>> Code review PRs. My  main reasons:
> > > >>>>>>> Most importantly code reviewing will introduce lot of noise and
> > > will
> > > >>>>>>> ultimately make  the GitHub page unreadable.
> > > >>>>>>> Avoid overlapping of work.
> > > >>>>>>> Once code is written hard to think abstract.
> > > >>>>>>> Separate page for Design review later can always be used it as
> a
> > > >>>> Design
> > > >>>>>>> document that is readable and code free-ish.
> > > >>>>>>> As i said the goal of this first round is to see if the
> community
> > > >>>> agree
> > > >>>>>>> about such change, then make the process of design more
> inclusive
> > > >>>> thus
> > > >>>>>>> other contributors can submit a counter proposals.
> > > >>>>>>>
> > > >>>>>>> [Step 2] IF everybody agree about that point Step 2 is to
> define
> > > >>>> which
> > > >>>>>>> medium is used to Publish a primitive form of a CODE FREE
> > Abstract
> > > >>>>>> Proposal
> > > >>>>>>> containing at least the following bullet points.
> > > >>>>>>> - The problem description and motivation
> > > >>>>>>> - Overview of the proposed change
> > > >>>>>>> - Operational impact (compatibility/ plans to upgrades) public
> > API
> > > >>>>>> changes,
> > > >>>>>>> configuration changes, algorithm, and so on
> > > >>>>>>> - Expected benefits and drawbacks
> > > >>>>>>> - Rationale and alternatives
> > > >>>>>>> - Estimate Time to Deliver if possible.
> > > >>>>>>>
> > > >>>>>>> The way i think this can be is a Github issue where member of
> the
> > > >>>>>> community
> > > >>>>>>> will interact via comments and the author will be updating the
> > > >>>>>> description
> > > >>>>>>> in the light of comments provided by the community.
> > > >>>>>>>
> > > >>>>>>> During and near the end of the design discussions the author/s
> > can
> > > >>>>> start
> > > >>>>>>> writing POCs to help guide the review process this naturally
> will
> > > be
> > > >>>> a
> > > >>>>>> Pull
> > > >>>>>>> request with actual code.
> > > >>>>>>>
> > > >>>>>>> *Now the most important thing is that we need to agree that any
> > > work
> > > >>>>> that
> > > >>>>>>> does not align with this formal process will be ignored and the
> > > >>>> author
> > > >>>>>> will
> > > >>>>>>> be asked to start with a DIP*
> > > >>>>>>> *That is what i meant with  “If it didn’t happen on the mailing
> > > list,
> > > >>>>> it
> > > >>>>>>> didn’t happen.”*
> > > >>>>>>>
> > > >>>>>>> Thanks and happy coding!
> > > >>>>>>>
> > > >>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <gi...@apache.org>
> > > wrote:
> > > >>>>>>>
> > > >>>>>>>> One of the advantages I see with a more formal process is
> (like
> > > >>>> Kafka
> > > >>>>>>> KIPs)
> > > >>>>>>>> is that it levels the playing field a bit and sets some ground
> > > >>>> rules
> > > >>>>>> for
> > > >>>>>>>> working together. In a way it can help encourage contributions
> > by
> > > >>>>>> making
> > > >>>>>>> it
> > > >>>>>>>> clear what is expected of potential contributors.
> > > >>>>>>>>
> > > >>>>>>>> We have a design review process today that is not as formal as
> > > >>>> KIPs,
> > > >>>>>> but
> > > >>>>>>> is
> > > >>>>>>>> somewhat heavier than the one you describe. Maybe we could
> tweak
> > > >>>> our
> > > >>>>>>>> current one by starting to do design reviews separately from
> > PRs.
> > > >>>>> i.e.,
> > > >>>>>>> for
> > > >>>>>>>> anything that meets our 'design review' criteria, do that on
> the
> > > >>>> dev
> > > >>>>>> list
> > > >>>>>>>> or in a separate issue, and keep the PR focused on code-level
> > > >>>> stuff.
> > > >>>>>> That
> > > >>>>>>>> way we don't end up trying to do both at once. And it makes it
> > > >>>> easier
> > > >>>>>> to
> > > >>>>>>>> start talking about design before the code is ready, which
> would
> > > be
> > > >>>>>>> better.
> > > >>>>>>>>
> > > >>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <jh...@apache.org>
> > > >>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> It’s really hard to say no to a contribution when someone has
> > put
> > > >>>>> in
> > > >>>>>> a
> > > >>>>>>>>> significant amount of work.
> > > >>>>>>>>>
> > > >>>>>>>>> The following approach is simple and works really well:
> Before
> > > >>>> you
> > > >>>>>>> start
> > > >>>>>>>>> work, log a case, describing the problem. When you have some
> > > >>>> ideas
> > > >>>>>>> about
> > > >>>>>>>>> design, add those to the case. When you have a code branch,
> add
> > > >>>> its
> > > >>>>>> URL
> > > >>>>>>>> to
> > > >>>>>>>>> the case. And so forth. At any point in the proceedings,
> people
> > > >>>> can
> > > >>>>>>> chime
> > > >>>>>>>>> in with their opinions.
> > > >>>>>>>>>
> > > >>>>>>>>> In my opinion, a formal “design review” process is not
> > necessary.
> > > >>>>>> Just
> > > >>>>>>>>> build consensus iteratively, by starting the conversation
> early
> > > >>>> in
> > > >>>>>> the
> > > >>>>>>>>> process.
> > > >>>>>>>>>
> > > >>>>>>>>> Julian
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <gi...@apache.org>
> > > >>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>> In this particular case: please consider the PR as a
> proposal.
> > > >>>>>> Don't
> > > >>>>>>>> feel
> > > >>>>>>>>>> like just because there is code there that takes a certain
> > > >>>>>> approach,
> > > >>>>>>>> that
> > > >>>>>>>>>> the approach is somehow sacred. I had to implement something
> > to
> > > >>>>>>>>> crystallize
> > > >>>>>>>>>> my own thinking about how the problem could be approached. I
> > > >>>>> won't
> > > >>>>>> be
> > > >>>>>>>>>> disappointed if, as a community, we decide a different
> > > >>>> direction
> > > >>>>> is
> > > >>>>>>>>> better
> > > >>>>>>>>>> and the code all gets thrown away. That's one of the reasons
> > > >>>>> that I
> > > >>>>>>>>> removed
> > > >>>>>>>>>> the 0.14.0 milestone that was added to the patch. (I don't
> > want
> > > >>>>> to
> > > >>>>>>> rush
> > > >>>>>>>>> it,
> > > >>>>>>>>>> nor do I think that's a good idea.)
> > > >>>>>>>>>>
> > > >>>>>>>>>> In general: Sounds like we could do with some more
> > > >>>> formalization
> > > >>>>>>> around
> > > >>>>>>>>>> what a proposal looks like, which sorts of changes need one,
> > > >>>> and
> > > >>>>>> when
> > > >>>>>>>> in
> > > >>>>>>>>>> the dev cycle it is appropriate. FWIW I think Kafka's
> process
> > > >>>> is
> > > >>>>>> more
> > > >>>>>>>> or
> > > >>>>>>>>>> less fine, and would be okay with adopting it for Druid if
> > > >>>> people
> > > >>>>>>> like
> > > >>>>>>>>> it.
> > > >>>>>>>>>> Right now our standards for what requires a "design review"
> > are
> > > >>>>>> very
> > > >>>>>>>>>> similar to the Kafka community standards for what requires a
> > > >>>> KIP,
> > > >>>>>> so
> > > >>>>>>> we
> > > >>>>>>>>>> have some familiarity with those concepts. However we don't
> > > >>>>>> separate
> > > >>>>>>> PR
> > > >>>>>>>>>> review and proposal discussion as strictly as they do, which
> > > >>>>> seems
> > > >>>>>> to
> > > >>>>>>>> be
> > > >>>>>>>>>> the foundation for the feeling of exclusion that is being
> felt
> > > >>>>>> here.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Separately: I just redid the description on
> > > >>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794 to be
> > more
> > > >>>>>>>>> proposal-y.
> > > >>>>>>>>>> I followed the KIP style:
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > >>>>>>>>> .
> > > >>>>>>>>>> Please refresh the page and see if it looks more useful.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Gian
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <
> jhyde@apache.org
> > >
> > > >>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Slim,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I agree with your points that offline development is bad
> for
> > > >>>>>>>> community.
> > > >>>>>>>>>>> But I don’t think you need much mentor help. You have
> raised
> > > >>>>> valid
> > > >>>>>>>>> issues
> > > >>>>>>>>>>> and the Druid community needs to decide what its
> development
> > > >>>>>>> practices
> > > >>>>>>>>>>> should be.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Julian
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
> > > >>>> bslim@apache.org>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Hello everyone and hope you all have very good holidays.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> First, this email is not directed on the author or the PR
> > > >>>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794  it
> > > >>>> self,
> > > >>>>>> but
> > > >>>>>>> i
> > > >>>>>>>>> see
> > > >>>>>>>>>>>> this PR as a perfect example.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> One of the foundation of Apache Way or what i would simply
> > > >>>> call
> > > >>>>>>> open
> > > >>>>>>>>>>> source
> > > >>>>>>>>>>>> community driven development is that "Technical decisions
> > are
> > > >>>>>>>>> discussed,
> > > >>>>>>>>>>>> decided, and archived publicly.
> > > >>>>>>>>>>>> developpement"
> > > >>>>>>>>>>>> Which means that big technical  changes such as the one
> > > >>>> brought
> > > >>>>>> by
> > > >>>>>>>>> #/6794
> > > >>>>>>>>>>>> should have started as a proposal and round of discussions
> > > >>>>> about
> > > >>>>>>> the
> > > >>>>>>>>>>> major
> > > >>>>>>>>>>>> changes designs not as 11K line of code.
> > > >>>>>>>>>>>> I believe such openness will promote a lot of good
> benefits
> > > >>>>> such
> > > >>>>>>> as:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> - ensures community health and growth.
> > > >>>>>>>>>>>> - ensures everyone can participate not only the authors
> and
> > > >>>> his
> > > >>>>>>>>>>> co-workers.
> > > >>>>>>>>>>>> - ensures that the project is driven by the community and
> > > >>>> not a
> > > >>>>>>> given
> > > >>>>>>>>>>>> company or an individual.
> > > >>>>>>>>>>>> - ensures that there is consensus (not saying 100%
> > > >>>> agreement;)
> > > >>>>>>>> however
> > > >>>>>>>>> it
> > > >>>>>>>>>>>> means that all individuals will accept the current
> progress
> > > >>>> on
> > > >>>>>> the
> > > >>>>>>>>>>> project
> > > >>>>>>>>>>>> until some better proposal is put forth.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Personally such BIG offline PR makes me feel excluded and
> > > >>>>> doesn't
> > > >>>>>>>> give
> > > >>>>>>>>>>> me a
> > > >>>>>>>>>>>> sense that i belong to  a community at all.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> To prevent such off list development i think as a Druid
> > > >>>>> Community
> > > >>>>>>> we
> > > >>>>>>>>> need
> > > >>>>>>>>>>>> to stick to the apache way “If it didn’t happen on the
> > > >>>> mailing
> > > >>>>>>> list,
> > > >>>>>>>> it
> > > >>>>>>>>>>>> didn’t happen.”
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I would appreciate if some of the Apache mentor help with
> > > >>>> this.
> > > >>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>
> > > ---------------------------------------------------------------------
> > > >>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > >>>>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>
> > ---------------------------------------------------------------------
> > > >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > >>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> --
> > > >>>>>
> > > >>>>> B-Slim
> > > >>>>>
> > > >>
> > _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> > > >>>>>
> > > >>>>
> > > >>
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > >> For additional commands, e-mail: dev-help@druid.apache.org
> > > >>
> > > >>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > For additional commands, e-mail: dev-help@druid.apache.org
> > >
> > >
> >
>

Re: Off list major development

[Clint Wylie <cl...@imply.io>]

Apologies for the delayed response.

On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <sl...@gmail.com>
wrote:

> I am wondering here what is the case where code first is better?
>

I don't think it's wrong to share ideas as early as possible, and after
this discussion I think I am in favor of it too. I just meant that I don't
think it's always necessarily the most productive discussion until code
exists sometimes, with the types of thing I am thinking of are almost
entirely limited to cases where things might sound good to anyone on paper
but in reality need a large amount of experiments conducted and
observations collected to determine that something is actually worth doing,
which I imagine is mainly things like reworking internals for performance
improvements.

In the case of my combined proposal PR, I needed to prove that the thing I
was working on was a good idea... and it wasn't directly. But I came up
with another idea during the course of experiment turned into something
compelling, so an initial proposal would have looked quite a lot different
than what I ended up with. Once I had proven to myself that it was a good
idea, then I was comfortable sharing with the wider community. I'm not
certain how this would play out in an always proposal first model, maybe
the first proposal exists, I personally reject it after updating with
experiment results show it's a bad idea, continue experimenting and raise a
new one after the experiments start looking promising?


> Let's not be naive this is very rare that a contributor will accept that
> his work is to be thrown, usually devs takes coding as personal creation
> and they get attached to it.
>

I agree, just because a handful of the committers have this attitude, it
isn't fair to expect the wider community to also, that's why I am in favor
of formalizing the process.

Can you please explain what is overbearing ? what can be changed to make it
> easy ?
> Most of the points are kind of the actual questions that you want to
> address before hand anyway isn't ?
>

Sorry for the confusion, I said it's "not overbearing", I think it's fine.

What are the complaints ?


Is this and other previous threads not a complaint about opening a large PR
without a proposal? :) I just mean that formalizing the process, even if a
proposal has a reference PR opened with it near concurrently, could prevent
these discussions from happening in the future because ground rules have
been set and we are all on the same page I guess.

I think we should also probably consider renaming the "design review" label
to "proposal" or something to make it more clear that a PR is associated
with a proposal. It might also be worth considering using github projects
for particularly large things that involve multiple follow up PRs, but I
haven't used them in much depth to know if they add anything.

It seems like we are all converging on agreement to do a github issue
proposal for larger changes (I would vote for announcing them on the dev
list too for more visibility), so that design review is separated from code
review. I guess my main concern was not wanting to discourage
experimentation by walling it off behind mandatory discussions, but the
more I think about it,  it doesn't technically change much about this
process, it just requires a more formal proposal to accompany any
experiments that are shared as a PR.


On Tue, Jan 8, 2019 at 12:28 PM Gian Merlino <gi...@apache.org> wrote:

> I think for us, choosing to use GitHub issues as discussion threads for
> potential 'major' contributions would be a good idea, especially if we
> encourage people to start them before PRs show up. Definitely agree that
> all contributors should go through the same process -- I couldn't see it
> working well any other way.
>
> On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <jh...@apache.org> wrote:
>
> > Statically, yes, GitHub PRs are the same as GitHub cases. But
> dynamically,
> > they are different, because you can only log a PR when you have finished
> > work.
> >
> > A lot of other Apache projects use JIRA, so there is a clear distinction
> > between cases and contributions. JIRA cases, especially when logged early
> > in the lifecycle of a contribution, become long-running conversation
> > threads with a lot of community participation. If the Druid chose to do
> so,
> > GitHub cases could be the same.
> >
> > Be careful that you do not treat “potential contributors” (by which I
> > presume you mean non-committers) differently from committers and PMC
> > members. Anyone starting a major piece of work should follow the same
> > process. (Experienced committers probably have a somewhat better idea
> what
> > work will turn out to be “major”, so they get a little more leeway.)
> >
> > Julian
> >
> >
> > > On Jan 7, 2019, at 12:10 PM, Gian Merlino <gi...@apache.org> wrote:
> > >
> > > I don't think there's a need to raise issues for every change: a small
> > bug
> > > fix or doc fix should just go straight to PR. (GitHub PRs show up as
> > issues
> > > in the issue-search UI/API, so it's not like this means the patch has
> no
> > > corresponding issue -- in a sense the PR _is_ the issue.)
> > >
> > > I do think it makes sense to encourage potential contributors to write
> to
> > > the dev list or raise an issue if they aren't sure if something would
> > need
> > > to go through a more heavy weight process.
> > >
> > > Fwiw we do have a set of 'design review' criteria already (we had a
> > > discussion about this a couple years ago) at:
> > > http://druid.io/community/#getting-your-changes-accepted. So we
> > wouldn't be
> > > starting from zero on defining that. We set it up back when we were
> > trying
> > > to _streamline_ our process -- we used to require two non-author +1s
> for
> > > _every_ change, even minor ones. The introduction of design review
> > criteria
> > > was meant to classify which PRs need that level of review and which
> ones
> > > are minor and can be merged with less review. I do think it helped with
> > > getting minor PRs merged more quickly. The list of criteria is,
> > >
> > > - Major architectural changes or API changes
> > > - HTTP requests and responses (e. g. a new HTTP endpoint)
> > > - Interfaces for extensions
> > > - Server configuration (e. g. altering the behavior of a config
> property)
> > > - Emitted metrics
> > > - Other major changes, judged by the discretion of Druid committers
> > >
> > > Some of it is subjective, but it has been in place for a while, so it's
> > at
> > > least something we are relatively familiar with.
> > >
> > > On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <jh...@apache.org> wrote:
> > >
> > >> Small contributions don’t need any design review, whereas large
> > >> contributions need significant review. I don’t think we should require
> > an
> > >> additional step for those (many) small contributions. But who decides
> > >> whether a contribution fits into the small or large category?
> > >>
> > >> I think the solution is for authors to log a case (or send an email to
> > >> dev) before they start work on any contribution. Then committers can
> > >> request a more heavy-weight process if they think it is needed.
> > >>
> > >> Julian
> > >>
> > >>
> > >>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <gi...@apache.org> wrote:
> > >>>
> > >>> It sounds like splitting design from code review is a common theme
> in a
> > >> few
> > >>> of the posts here. How does everyone feel about making a point of
> > >>> encouraging design reviews to be done as issues, separate from the
> pull
> > >>> request, with the expectations that (1) the design review issue
> > >>> ("proposal") should generally appear somewhat _before_ the pull
> > request;
> > >>> (2) pull requests should _not_ have design review happen on them,
> > meaning
> > >>> there should no longer be PRs with design review tags, and we should
> > move
> > >>> the design review approval process to the issue rather than the PR.
> > >>>
> > >>> For (1), even if we encourage design review discussions to start
> > before a
> > >>> pull request appears, I don't see an issue with them running
> > concurrently
> > >>> for a while at some point.
> > >>>
> > >>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <jo...@apache.org>
> wrote:
> > >>>
> > >>>> Thanks for raising these concerns!
> > >>>>
> > >>>> My initial thoughts:
> > >>>> - I agree that separation of design review and code-level review for
> > >> major
> > >>>> changes would be more efficient
> > >>>>
> > >>>> - I agree that a clear, more formalized process for handling major
> > >> changes
> > >>>> would be helpful for contributors:
> > >>>> - Define what is considered a major change
> > >>>> - Define a standard proposal structure, KIP-style proposal format
> > >> sounds
> > >>>> good to me
> > >>>>
> > >>>> - I think it's too rigid to have a policy of "no code at all with
> the
> > >>>> initial proposal"
> > >>>> - Code samples can be useful references for understanding aspects
> of a
> > >>>> design
> > >>>> - In some cases it's necessary to run experiments to fully
> understand
> > a
> > >>>> problem and determine an appropriate design, or to determine whether
> > >>>> something is even worth doing before committing to the work of
> > fleshing
> > >> out
> > >>>> a proposal, prototype code is a natural outcome of that and I'm not
> > >> against
> > >>>> someone providing such code for reference
> > >>>> - I tend to view design/code as things that are often developed
> > >>>> simultaneously in an intertwined way
> > >>>>
> > >>>>> Let's not be naive this is very rare that a contributor will accept
> > >> that
> > >>>> his work is to be thrown, usually devs takes coding as personal
> > creation
> > >>>> and they get attached to it.
> > >>>>
> > >>>> If we have a clear review process that emphasizes the need for early
> > >>>> consensus building, with separate design and code review, then I
> feel
> > >> we've
> > >>>> done enough and don't need a hard rule against having some code
> linked
> > >> with
> > >>>> the initial proposal. If a potential contributor then still wants to
> > go
> > >>>> ahead and write a lot of code that may be rejected or change
> > >> significantly,
> > >>>> the risks were made clear.
> > >>>>
> > >>>>> Once code is written hard to think abstract.
> > >>>>
> > >>>> I can see the validity of the concern, but I personally don't see it
> > as
> > >> a
> > >>>> huge risk. My impression from the Druid PR reviews I've seen is that
> > our
> > >>>> reviewers are able to keep abstract design vs. implementation
> details
> > >>>> separate and consider alternate designs when reviewing.
> > >>>>
> > >>>> To summarize I think it's probably enough to have a policy along the
> > >> lines
> > >>>> of:
> > >>>> - Create more formalized guidelines for proposals and what changes
> > >> require
> > >>>> proposals
> > >>>> - Separate design and code review for major changes, with design
> > review
> > >>>> first, code-level review after reaching consensus on the design.
> > >>>> - Code before the design review is completed is just for reference,
> > not
> > >>>> regarded as a candidate for review/merging.
> > >>>>
> > >>>> - Jon
> > >>>>
> > >>>>
> > >>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> > >> slim.bouguerra@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <cl...@imply.io>
> > >>>> wrote:
> > >>>>>
> > >>>>>> I am definitely biased in this matter as an owner of another large
> > PR
> > >>>>> that
> > >>>>>> wasn't preceded by a direct proposal or dev list discussion, and
> in
> > >>>>> general
> > >>>>>> I agree that proposal first is usually better, but I think in some
> > >>>> rarer
> > >>>>>> cases approaching a problem code first *is* the most appropriate
> way
> > >> to
> > >>>>>> have a discussion.
> > >>>>>
> > >>>>>
> > >>>>> I am wondering here what is the case where code first is better?
> > >>>>> In general when you are writing code you have an idea about what
> you
> > >> want
> > >>>>> to change, why you want to change and why you want to change it.
> > >>>>> I do not see what is wrong with sharing this primitive ideas and
> > >> thoughts
> > >>>>> as an abstract proposal (at least to avoid overlapping)
> > >>>>>
> > >>>>> I see nothing wrong with it so long as the author
> > >>>>>> accepts that the PR is treated as a combined proposal and proof of
> > >>>>> concept,
> > >>>>>> and fair game to be radically changed via discussion or even
> > rejected,
> > >>>>>> which sounds like Gian's attitude on the matter and is mine as
> well
> > >>>> with
> > >>>>> my
> > >>>>>> compression stuff.
> > >>>>>
> > >>>>>
> > >>>>> Let's not be naive this is very rare that a contributor will accept
> > >> that
> > >>>>> his work is to be thrown, usually devs takes coding as personal
> > >> creation
> > >>>>> and they get attached to it.
> > >>>>> To my point you can take a look on some old issue in the Druid
> forum
> > >>>>>
> > >>>>
> > >>
> >
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> > >>>>> and am sure other communities have similar problems.
> > >>>>> So leaving the door open to some side cases is not a good idea in
> my
> > >>>>> opinion and will lead to similar issue in the future.
> > >>>>>
> > >>>>> This seems to me especially likely to happen in cases
> > >>>>>> where an approach still needs proven to be a viable idea *to the
> > >>>> author*,
> > >>>>>> so that a much more productive discussion can be had in the first
> > >>>> place.
> > >>>>>>
> > >>>>>> I think there is a trade off, I don't think we want to discourage
> > >>>>>> experimentation by walling it off behind mandatory discussions
> > before
> > >>>> it
> > >>>>>> can even start, but I do think formalizing the process for large
> > >>>> changes
> > >>>>> is
> > >>>>>> a good thing, especially since we probably can't expect the wider
> > >>>>> community
> > >>>>>> to have the same attitude towards a large PR getting discarded as
> a
> > >>>>>> committer might. I think the Kafka approach is reasonable, a bit
> > more
> > >>>>>> formal than our design review process but not overbearing.
> > >>>>>
> > >>>>>
> > >>>>> Can you please explain what is overbearing ? what can be changed to
> > >> make
> > >>>> it
> > >>>>> easy ?
> > >>>>> Most of the points are kind of the actual questions that you want
> to
> > >>>>> address before hand anyway isn't ?
> > >>>>>
> > >>>>>
> > >>>>>> Going code first
> > >>>>>> should be in general discouraged, but when it does happen, it
> seems
> > >>>> like
> > >>>>>> opening DIP/an issue/starting a mailing list thread or whatever we
> > go
> > >>>>> with
> > >>>>>> to have a more high level design discussion alongside the
> reference
> > PR
> > >>>>>> could alleviate some of these complaints?
> > >>>>>
> > >>>>>
> > >>>>> What are the complaints ?
> > >>>>>
> > >>>>>
> > >>>>>> +1 for "DIP" heh, I think making
> > >>>>>> them in the form of github issues is probably appropriate, with a
> > dev
> > >>>>> list
> > >>>>>> thread to announce them perhaps?
> > >>>>>>
> > >>>>>
> > >>>>> I think  github issue with [Proposal] header like
> > >>>>> https://github.com/apache/incubator-druid/issues/4349 is good to
> me,
> > >>>>>
> > >>>>> Thanks!
> > >>>>>
> > >>>>>
> > >>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <bs...@apache.org>
> > >>>> wrote:
> > >>>>>>
> > >>>>>>> Thanks everyone for interacting with this thread.
> > >>>>>>>
> > >>>>>>> The fact that i, Roman, Jihoon  and others in the past (FJ
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ)
> > >>>>>>> raised this point indicates that PRs without a proposal are
> indeed
> > an
> > >>>>>> issue
> > >>>>>>> and we need to solve it.
> > >>>>>>>
> > >>>>>>> Something Similar to KIP maybe called DIPs is fine with me.
> > >>>>>>> What i strive to see is the following:
> > >>>>>>>
> > >>>>>>> [Step 1] formalize what is the kind of work that needs a formal
> > >>>>>> Proposal, I
> > >>>>>>> think Roman and Jihoon has already covered that pretty well. am
> +1
> > on
> > >>>>>> that.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> > >>>>>>>
> > >>>>>>> I am strongly in favor of the separation of Proposal Review and
> > >>>> (later)
> > >>>>>>> Code review PRs. My  main reasons:
> > >>>>>>> Most importantly code reviewing will introduce lot of noise and
> > will
> > >>>>>>> ultimately make  the GitHub page unreadable.
> > >>>>>>> Avoid overlapping of work.
> > >>>>>>> Once code is written hard to think abstract.
> > >>>>>>> Separate page for Design review later can always be used it as a
> > >>>> Design
> > >>>>>>> document that is readable and code free-ish.
> > >>>>>>> As i said the goal of this first round is to see if the community
> > >>>> agree
> > >>>>>>> about such change, then make the process of design more inclusive
> > >>>> thus
> > >>>>>>> other contributors can submit a counter proposals.
> > >>>>>>>
> > >>>>>>> [Step 2] IF everybody agree about that point Step 2 is to define
> > >>>> which
> > >>>>>>> medium is used to Publish a primitive form of a CODE FREE
> Abstract
> > >>>>>> Proposal
> > >>>>>>> containing at least the following bullet points.
> > >>>>>>> - The problem description and motivation
> > >>>>>>> - Overview of the proposed change
> > >>>>>>> - Operational impact (compatibility/ plans to upgrades) public
> API
> > >>>>>> changes,
> > >>>>>>> configuration changes, algorithm, and so on
> > >>>>>>> - Expected benefits and drawbacks
> > >>>>>>> - Rationale and alternatives
> > >>>>>>> - Estimate Time to Deliver if possible.
> > >>>>>>>
> > >>>>>>> The way i think this can be is a Github issue where member of the
> > >>>>>> community
> > >>>>>>> will interact via comments and the author will be updating the
> > >>>>>> description
> > >>>>>>> in the light of comments provided by the community.
> > >>>>>>>
> > >>>>>>> During and near the end of the design discussions the author/s
> can
> > >>>>> start
> > >>>>>>> writing POCs to help guide the review process this naturally will
> > be
> > >>>> a
> > >>>>>> Pull
> > >>>>>>> request with actual code.
> > >>>>>>>
> > >>>>>>> *Now the most important thing is that we need to agree that any
> > work
> > >>>>> that
> > >>>>>>> does not align with this formal process will be ignored and the
> > >>>> author
> > >>>>>> will
> > >>>>>>> be asked to start with a DIP*
> > >>>>>>> *That is what i meant with  “If it didn’t happen on the mailing
> > list,
> > >>>>> it
> > >>>>>>> didn’t happen.”*
> > >>>>>>>
> > >>>>>>> Thanks and happy coding!
> > >>>>>>>
> > >>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <gi...@apache.org>
> > wrote:
> > >>>>>>>
> > >>>>>>>> One of the advantages I see with a more formal process is (like
> > >>>> Kafka
> > >>>>>>> KIPs)
> > >>>>>>>> is that it levels the playing field a bit and sets some ground
> > >>>> rules
> > >>>>>> for
> > >>>>>>>> working together. In a way it can help encourage contributions
> by
> > >>>>>> making
> > >>>>>>> it
> > >>>>>>>> clear what is expected of potential contributors.
> > >>>>>>>>
> > >>>>>>>> We have a design review process today that is not as formal as
> > >>>> KIPs,
> > >>>>>> but
> > >>>>>>> is
> > >>>>>>>> somewhat heavier than the one you describe. Maybe we could tweak
> > >>>> our
> > >>>>>>>> current one by starting to do design reviews separately from
> PRs.
> > >>>>> i.e.,
> > >>>>>>> for
> > >>>>>>>> anything that meets our 'design review' criteria, do that on the
> > >>>> dev
> > >>>>>> list
> > >>>>>>>> or in a separate issue, and keep the PR focused on code-level
> > >>>> stuff.
> > >>>>>> That
> > >>>>>>>> way we don't end up trying to do both at once. And it makes it
> > >>>> easier
> > >>>>>> to
> > >>>>>>>> start talking about design before the code is ready, which would
> > be
> > >>>>>>> better.
> > >>>>>>>>
> > >>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <jh...@apache.org>
> > >>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> It’s really hard to say no to a contribution when someone has
> put
> > >>>>> in
> > >>>>>> a
> > >>>>>>>>> significant amount of work.
> > >>>>>>>>>
> > >>>>>>>>> The following approach is simple and works really well: Before
> > >>>> you
> > >>>>>>> start
> > >>>>>>>>> work, log a case, describing the problem. When you have some
> > >>>> ideas
> > >>>>>>> about
> > >>>>>>>>> design, add those to the case. When you have a code branch, add
> > >>>> its
> > >>>>>> URL
> > >>>>>>>> to
> > >>>>>>>>> the case. And so forth. At any point in the proceedings, people
> > >>>> can
> > >>>>>>> chime
> > >>>>>>>>> in with their opinions.
> > >>>>>>>>>
> > >>>>>>>>> In my opinion, a formal “design review” process is not
> necessary.
> > >>>>>> Just
> > >>>>>>>>> build consensus iteratively, by starting the conversation early
> > >>>> in
> > >>>>>> the
> > >>>>>>>>> process.
> > >>>>>>>>>
> > >>>>>>>>> Julian
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <gi...@apache.org>
> > >>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> In this particular case: please consider the PR as a proposal.
> > >>>>>> Don't
> > >>>>>>>> feel
> > >>>>>>>>>> like just because there is code there that takes a certain
> > >>>>>> approach,
> > >>>>>>>> that
> > >>>>>>>>>> the approach is somehow sacred. I had to implement something
> to
> > >>>>>>>>> crystallize
> > >>>>>>>>>> my own thinking about how the problem could be approached. I
> > >>>>> won't
> > >>>>>> be
> > >>>>>>>>>> disappointed if, as a community, we decide a different
> > >>>> direction
> > >>>>> is
> > >>>>>>>>> better
> > >>>>>>>>>> and the code all gets thrown away. That's one of the reasons
> > >>>>> that I
> > >>>>>>>>> removed
> > >>>>>>>>>> the 0.14.0 milestone that was added to the patch. (I don't
> want
> > >>>>> to
> > >>>>>>> rush
> > >>>>>>>>> it,
> > >>>>>>>>>> nor do I think that's a good idea.)
> > >>>>>>>>>>
> > >>>>>>>>>> In general: Sounds like we could do with some more
> > >>>> formalization
> > >>>>>>> around
> > >>>>>>>>>> what a proposal looks like, which sorts of changes need one,
> > >>>> and
> > >>>>>> when
> > >>>>>>>> in
> > >>>>>>>>>> the dev cycle it is appropriate. FWIW I think Kafka's process
> > >>>> is
> > >>>>>> more
> > >>>>>>>> or
> > >>>>>>>>>> less fine, and would be okay with adopting it for Druid if
> > >>>> people
> > >>>>>>> like
> > >>>>>>>>> it.
> > >>>>>>>>>> Right now our standards for what requires a "design review"
> are
> > >>>>>> very
> > >>>>>>>>>> similar to the Kafka community standards for what requires a
> > >>>> KIP,
> > >>>>>> so
> > >>>>>>> we
> > >>>>>>>>>> have some familiarity with those concepts. However we don't
> > >>>>>> separate
> > >>>>>>> PR
> > >>>>>>>>>> review and proposal discussion as strictly as they do, which
> > >>>>> seems
> > >>>>>> to
> > >>>>>>>> be
> > >>>>>>>>>> the foundation for the feeling of exclusion that is being felt
> > >>>>>> here.
> > >>>>>>>>>>
> > >>>>>>>>>> Separately: I just redid the description on
> > >>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794 to be
> more
> > >>>>>>>>> proposal-y.
> > >>>>>>>>>> I followed the KIP style:
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >>>>>>>>> .
> > >>>>>>>>>> Please refresh the page and see if it looks more useful.
> > >>>>>>>>>>
> > >>>>>>>>>> Gian
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jhyde@apache.org
> >
> > >>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Slim,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I agree with your points that offline development is bad for
> > >>>>>>>> community.
> > >>>>>>>>>>> But I don’t think you need much mentor help. You have raised
> > >>>>> valid
> > >>>>>>>>> issues
> > >>>>>>>>>>> and the Druid community needs to decide what its development
> > >>>>>>> practices
> > >>>>>>>>>>> should be.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Julian
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
> > >>>> bslim@apache.org>
> > >>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hello everyone and hope you all have very good holidays.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> First, this email is not directed on the author or the PR
> > >>>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794  it
> > >>>> self,
> > >>>>>> but
> > >>>>>>> i
> > >>>>>>>>> see
> > >>>>>>>>>>>> this PR as a perfect example.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> One of the foundation of Apache Way or what i would simply
> > >>>> call
> > >>>>>>> open
> > >>>>>>>>>>> source
> > >>>>>>>>>>>> community driven development is that "Technical decisions
> are
> > >>>>>>>>> discussed,
> > >>>>>>>>>>>> decided, and archived publicly.
> > >>>>>>>>>>>> developpement"
> > >>>>>>>>>>>> Which means that big technical  changes such as the one
> > >>>> brought
> > >>>>>> by
> > >>>>>>>>> #/6794
> > >>>>>>>>>>>> should have started as a proposal and round of discussions
> > >>>>> about
> > >>>>>>> the
> > >>>>>>>>>>> major
> > >>>>>>>>>>>> changes designs not as 11K line of code.
> > >>>>>>>>>>>> I believe such openness will promote a lot of good benefits
> > >>>>> such
> > >>>>>>> as:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> - ensures community health and growth.
> > >>>>>>>>>>>> - ensures everyone can participate not only the authors and
> > >>>> his
> > >>>>>>>>>>> co-workers.
> > >>>>>>>>>>>> - ensures that the project is driven by the community and
> > >>>> not a
> > >>>>>>> given
> > >>>>>>>>>>>> company or an individual.
> > >>>>>>>>>>>> - ensures that there is consensus (not saying 100%
> > >>>> agreement;)
> > >>>>>>>> however
> > >>>>>>>>> it
> > >>>>>>>>>>>> means that all individuals will accept the current progress
> > >>>> on
> > >>>>>> the
> > >>>>>>>>>>> project
> > >>>>>>>>>>>> until some better proposal is put forth.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Personally such BIG offline PR makes me feel excluded and
> > >>>>> doesn't
> > >>>>>>>> give
> > >>>>>>>>>>> me a
> > >>>>>>>>>>>> sense that i belong to  a community at all.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> To prevent such off list development i think as a Druid
> > >>>>> Community
> > >>>>>>> we
> > >>>>>>>>> need
> > >>>>>>>>>>>> to stick to the apache way “If it didn’t happen on the
> > >>>> mailing
> > >>>>>>> list,
> > >>>>>>>> it
> > >>>>>>>>>>>> didn’t happen.”
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I would appreciate if some of the Apache mentor help with
> > >>>> this.
> > >>>>>>>>>>>> Thanks
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>
> > ---------------------------------------------------------------------
> > >>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > >>>>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>
> ---------------------------------------------------------------------
> > >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > >>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>>
> > >>>>> B-Slim
> > >>>>>
> > >>
> _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> > >>>>>
> > >>>>
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > >> For additional commands, e-mail: dev-help@druid.apache.org
> > >>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > For additional commands, e-mail: dev-help@druid.apache.org
> >
> >
>

Re: Off list major development

[Gian Merlino <gi...@apache.org>]

I think for us, choosing to use GitHub issues as discussion threads for
potential 'major' contributions would be a good idea, especially if we
encourage people to start them before PRs show up. Definitely agree that
all contributors should go through the same process -- I couldn't see it
working well any other way.

On Mon, Jan 7, 2019 at 12:23 PM Julian Hyde <jh...@apache.org> wrote:

> Statically, yes, GitHub PRs are the same as GitHub cases. But dynamically,
> they are different, because you can only log a PR when you have finished
> work.
>
> A lot of other Apache projects use JIRA, so there is a clear distinction
> between cases and contributions. JIRA cases, especially when logged early
> in the lifecycle of a contribution, become long-running conversation
> threads with a lot of community participation. If the Druid chose to do so,
> GitHub cases could be the same.
>
> Be careful that you do not treat “potential contributors” (by which I
> presume you mean non-committers) differently from committers and PMC
> members. Anyone starting a major piece of work should follow the same
> process. (Experienced committers probably have a somewhat better idea what
> work will turn out to be “major”, so they get a little more leeway.)
>
> Julian
>
>
> > On Jan 7, 2019, at 12:10 PM, Gian Merlino <gi...@apache.org> wrote:
> >
> > I don't think there's a need to raise issues for every change: a small
> bug
> > fix or doc fix should just go straight to PR. (GitHub PRs show up as
> issues
> > in the issue-search UI/API, so it's not like this means the patch has no
> > corresponding issue -- in a sense the PR _is_ the issue.)
> >
> > I do think it makes sense to encourage potential contributors to write to
> > the dev list or raise an issue if they aren't sure if something would
> need
> > to go through a more heavy weight process.
> >
> > Fwiw we do have a set of 'design review' criteria already (we had a
> > discussion about this a couple years ago) at:
> > http://druid.io/community/#getting-your-changes-accepted. So we
> wouldn't be
> > starting from zero on defining that. We set it up back when we were
> trying
> > to _streamline_ our process -- we used to require two non-author +1s for
> > _every_ change, even minor ones. The introduction of design review
> criteria
> > was meant to classify which PRs need that level of review and which ones
> > are minor and can be merged with less review. I do think it helped with
> > getting minor PRs merged more quickly. The list of criteria is,
> >
> > - Major architectural changes or API changes
> > - HTTP requests and responses (e. g. a new HTTP endpoint)
> > - Interfaces for extensions
> > - Server configuration (e. g. altering the behavior of a config property)
> > - Emitted metrics
> > - Other major changes, judged by the discretion of Druid committers
> >
> > Some of it is subjective, but it has been in place for a while, so it's
> at
> > least something we are relatively familiar with.
> >
> > On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <jh...@apache.org> wrote:
> >
> >> Small contributions don’t need any design review, whereas large
> >> contributions need significant review. I don’t think we should require
> an
> >> additional step for those (many) small contributions. But who decides
> >> whether a contribution fits into the small or large category?
> >>
> >> I think the solution is for authors to log a case (or send an email to
> >> dev) before they start work on any contribution. Then committers can
> >> request a more heavy-weight process if they think it is needed.
> >>
> >> Julian
> >>
> >>
> >>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <gi...@apache.org> wrote:
> >>>
> >>> It sounds like splitting design from code review is a common theme in a
> >> few
> >>> of the posts here. How does everyone feel about making a point of
> >>> encouraging design reviews to be done as issues, separate from the pull
> >>> request, with the expectations that (1) the design review issue
> >>> ("proposal") should generally appear somewhat _before_ the pull
> request;
> >>> (2) pull requests should _not_ have design review happen on them,
> meaning
> >>> there should no longer be PRs with design review tags, and we should
> move
> >>> the design review approval process to the issue rather than the PR.
> >>>
> >>> For (1), even if we encourage design review discussions to start
> before a
> >>> pull request appears, I don't see an issue with them running
> concurrently
> >>> for a while at some point.
> >>>
> >>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <jo...@apache.org> wrote:
> >>>
> >>>> Thanks for raising these concerns!
> >>>>
> >>>> My initial thoughts:
> >>>> - I agree that separation of design review and code-level review for
> >> major
> >>>> changes would be more efficient
> >>>>
> >>>> - I agree that a clear, more formalized process for handling major
> >> changes
> >>>> would be helpful for contributors:
> >>>> - Define what is considered a major change
> >>>> - Define a standard proposal structure, KIP-style proposal format
> >> sounds
> >>>> good to me
> >>>>
> >>>> - I think it's too rigid to have a policy of "no code at all with the
> >>>> initial proposal"
> >>>> - Code samples can be useful references for understanding aspects of a
> >>>> design
> >>>> - In some cases it's necessary to run experiments to fully understand
> a
> >>>> problem and determine an appropriate design, or to determine whether
> >>>> something is even worth doing before committing to the work of
> fleshing
> >> out
> >>>> a proposal, prototype code is a natural outcome of that and I'm not
> >> against
> >>>> someone providing such code for reference
> >>>> - I tend to view design/code as things that are often developed
> >>>> simultaneously in an intertwined way
> >>>>
> >>>>> Let's not be naive this is very rare that a contributor will accept
> >> that
> >>>> his work is to be thrown, usually devs takes coding as personal
> creation
> >>>> and they get attached to it.
> >>>>
> >>>> If we have a clear review process that emphasizes the need for early
> >>>> consensus building, with separate design and code review, then I feel
> >> we've
> >>>> done enough and don't need a hard rule against having some code linked
> >> with
> >>>> the initial proposal. If a potential contributor then still wants to
> go
> >>>> ahead and write a lot of code that may be rejected or change
> >> significantly,
> >>>> the risks were made clear.
> >>>>
> >>>>> Once code is written hard to think abstract.
> >>>>
> >>>> I can see the validity of the concern, but I personally don't see it
> as
> >> a
> >>>> huge risk. My impression from the Druid PR reviews I've seen is that
> our
> >>>> reviewers are able to keep abstract design vs. implementation details
> >>>> separate and consider alternate designs when reviewing.
> >>>>
> >>>> To summarize I think it's probably enough to have a policy along the
> >> lines
> >>>> of:
> >>>> - Create more formalized guidelines for proposals and what changes
> >> require
> >>>> proposals
> >>>> - Separate design and code review for major changes, with design
> review
> >>>> first, code-level review after reaching consensus on the design.
> >>>> - Code before the design review is completed is just for reference,
> not
> >>>> regarded as a candidate for review/merging.
> >>>>
> >>>> - Jon
> >>>>
> >>>>
> >>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> >> slim.bouguerra@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <cl...@imply.io>
> >>>> wrote:
> >>>>>
> >>>>>> I am definitely biased in this matter as an owner of another large
> PR
> >>>>> that
> >>>>>> wasn't preceded by a direct proposal or dev list discussion, and in
> >>>>> general
> >>>>>> I agree that proposal first is usually better, but I think in some
> >>>> rarer
> >>>>>> cases approaching a problem code first *is* the most appropriate way
> >> to
> >>>>>> have a discussion.
> >>>>>
> >>>>>
> >>>>> I am wondering here what is the case where code first is better?
> >>>>> In general when you are writing code you have an idea about what you
> >> want
> >>>>> to change, why you want to change and why you want to change it.
> >>>>> I do not see what is wrong with sharing this primitive ideas and
> >> thoughts
> >>>>> as an abstract proposal (at least to avoid overlapping)
> >>>>>
> >>>>> I see nothing wrong with it so long as the author
> >>>>>> accepts that the PR is treated as a combined proposal and proof of
> >>>>> concept,
> >>>>>> and fair game to be radically changed via discussion or even
> rejected,
> >>>>>> which sounds like Gian's attitude on the matter and is mine as well
> >>>> with
> >>>>> my
> >>>>>> compression stuff.
> >>>>>
> >>>>>
> >>>>> Let's not be naive this is very rare that a contributor will accept
> >> that
> >>>>> his work is to be thrown, usually devs takes coding as personal
> >> creation
> >>>>> and they get attached to it.
> >>>>> To my point you can take a look on some old issue in the Druid forum
> >>>>>
> >>>>
> >>
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> >>>>> and am sure other communities have similar problems.
> >>>>> So leaving the door open to some side cases is not a good idea in my
> >>>>> opinion and will lead to similar issue in the future.
> >>>>>
> >>>>> This seems to me especially likely to happen in cases
> >>>>>> where an approach still needs proven to be a viable idea *to the
> >>>> author*,
> >>>>>> so that a much more productive discussion can be had in the first
> >>>> place.
> >>>>>>
> >>>>>> I think there is a trade off, I don't think we want to discourage
> >>>>>> experimentation by walling it off behind mandatory discussions
> before
> >>>> it
> >>>>>> can even start, but I do think formalizing the process for large
> >>>> changes
> >>>>> is
> >>>>>> a good thing, especially since we probably can't expect the wider
> >>>>> community
> >>>>>> to have the same attitude towards a large PR getting discarded as a
> >>>>>> committer might. I think the Kafka approach is reasonable, a bit
> more
> >>>>>> formal than our design review process but not overbearing.
> >>>>>
> >>>>>
> >>>>> Can you please explain what is overbearing ? what can be changed to
> >> make
> >>>> it
> >>>>> easy ?
> >>>>> Most of the points are kind of the actual questions that you want to
> >>>>> address before hand anyway isn't ?
> >>>>>
> >>>>>
> >>>>>> Going code first
> >>>>>> should be in general discouraged, but when it does happen, it seems
> >>>> like
> >>>>>> opening DIP/an issue/starting a mailing list thread or whatever we
> go
> >>>>> with
> >>>>>> to have a more high level design discussion alongside the reference
> PR
> >>>>>> could alleviate some of these complaints?
> >>>>>
> >>>>>
> >>>>> What are the complaints ?
> >>>>>
> >>>>>
> >>>>>> +1 for "DIP" heh, I think making
> >>>>>> them in the form of github issues is probably appropriate, with a
> dev
> >>>>> list
> >>>>>> thread to announce them perhaps?
> >>>>>>
> >>>>>
> >>>>> I think  github issue with [Proposal] header like
> >>>>> https://github.com/apache/incubator-druid/issues/4349 is good to me,
> >>>>>
> >>>>> Thanks!
> >>>>>
> >>>>>
> >>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <bs...@apache.org>
> >>>> wrote:
> >>>>>>
> >>>>>>> Thanks everyone for interacting with this thread.
> >>>>>>>
> >>>>>>> The fact that i, Roman, Jihoon  and others in the past (FJ
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ)
> >>>>>>> raised this point indicates that PRs without a proposal are indeed
> an
> >>>>>> issue
> >>>>>>> and we need to solve it.
> >>>>>>>
> >>>>>>> Something Similar to KIP maybe called DIPs is fine with me.
> >>>>>>> What i strive to see is the following:
> >>>>>>>
> >>>>>>> [Step 1] formalize what is the kind of work that needs a formal
> >>>>>> Proposal, I
> >>>>>>> think Roman and Jihoon has already covered that pretty well. am +1
> on
> >>>>>> that.
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> >>>>>>>
> >>>>>>> I am strongly in favor of the separation of Proposal Review and
> >>>> (later)
> >>>>>>> Code review PRs. My  main reasons:
> >>>>>>> Most importantly code reviewing will introduce lot of noise and
> will
> >>>>>>> ultimately make  the GitHub page unreadable.
> >>>>>>> Avoid overlapping of work.
> >>>>>>> Once code is written hard to think abstract.
> >>>>>>> Separate page for Design review later can always be used it as a
> >>>> Design
> >>>>>>> document that is readable and code free-ish.
> >>>>>>> As i said the goal of this first round is to see if the community
> >>>> agree
> >>>>>>> about such change, then make the process of design more inclusive
> >>>> thus
> >>>>>>> other contributors can submit a counter proposals.
> >>>>>>>
> >>>>>>> [Step 2] IF everybody agree about that point Step 2 is to define
> >>>> which
> >>>>>>> medium is used to Publish a primitive form of a CODE FREE Abstract
> >>>>>> Proposal
> >>>>>>> containing at least the following bullet points.
> >>>>>>> - The problem description and motivation
> >>>>>>> - Overview of the proposed change
> >>>>>>> - Operational impact (compatibility/ plans to upgrades) public API
> >>>>>> changes,
> >>>>>>> configuration changes, algorithm, and so on
> >>>>>>> - Expected benefits and drawbacks
> >>>>>>> - Rationale and alternatives
> >>>>>>> - Estimate Time to Deliver if possible.
> >>>>>>>
> >>>>>>> The way i think this can be is a Github issue where member of the
> >>>>>> community
> >>>>>>> will interact via comments and the author will be updating the
> >>>>>> description
> >>>>>>> in the light of comments provided by the community.
> >>>>>>>
> >>>>>>> During and near the end of the design discussions the author/s can
> >>>>> start
> >>>>>>> writing POCs to help guide the review process this naturally will
> be
> >>>> a
> >>>>>> Pull
> >>>>>>> request with actual code.
> >>>>>>>
> >>>>>>> *Now the most important thing is that we need to agree that any
> work
> >>>>> that
> >>>>>>> does not align with this formal process will be ignored and the
> >>>> author
> >>>>>> will
> >>>>>>> be asked to start with a DIP*
> >>>>>>> *That is what i meant with  “If it didn’t happen on the mailing
> list,
> >>>>> it
> >>>>>>> didn’t happen.”*
> >>>>>>>
> >>>>>>> Thanks and happy coding!
> >>>>>>>
> >>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <gi...@apache.org>
> wrote:
> >>>>>>>
> >>>>>>>> One of the advantages I see with a more formal process is (like
> >>>> Kafka
> >>>>>>> KIPs)
> >>>>>>>> is that it levels the playing field a bit and sets some ground
> >>>> rules
> >>>>>> for
> >>>>>>>> working together. In a way it can help encourage contributions by
> >>>>>> making
> >>>>>>> it
> >>>>>>>> clear what is expected of potential contributors.
> >>>>>>>>
> >>>>>>>> We have a design review process today that is not as formal as
> >>>> KIPs,
> >>>>>> but
> >>>>>>> is
> >>>>>>>> somewhat heavier than the one you describe. Maybe we could tweak
> >>>> our
> >>>>>>>> current one by starting to do design reviews separately from PRs.
> >>>>> i.e.,
> >>>>>>> for
> >>>>>>>> anything that meets our 'design review' criteria, do that on the
> >>>> dev
> >>>>>> list
> >>>>>>>> or in a separate issue, and keep the PR focused on code-level
> >>>> stuff.
> >>>>>> That
> >>>>>>>> way we don't end up trying to do both at once. And it makes it
> >>>> easier
> >>>>>> to
> >>>>>>>> start talking about design before the code is ready, which would
> be
> >>>>>>> better.
> >>>>>>>>
> >>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <jh...@apache.org>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>> It’s really hard to say no to a contribution when someone has put
> >>>>> in
> >>>>>> a
> >>>>>>>>> significant amount of work.
> >>>>>>>>>
> >>>>>>>>> The following approach is simple and works really well: Before
> >>>> you
> >>>>>>> start
> >>>>>>>>> work, log a case, describing the problem. When you have some
> >>>> ideas
> >>>>>>> about
> >>>>>>>>> design, add those to the case. When you have a code branch, add
> >>>> its
> >>>>>> URL
> >>>>>>>> to
> >>>>>>>>> the case. And so forth. At any point in the proceedings, people
> >>>> can
> >>>>>>> chime
> >>>>>>>>> in with their opinions.
> >>>>>>>>>
> >>>>>>>>> In my opinion, a formal “design review” process is not necessary.
> >>>>>> Just
> >>>>>>>>> build consensus iteratively, by starting the conversation early
> >>>> in
> >>>>>> the
> >>>>>>>>> process.
> >>>>>>>>>
> >>>>>>>>> Julian
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <gi...@apache.org>
> >>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> In this particular case: please consider the PR as a proposal.
> >>>>>> Don't
> >>>>>>>> feel
> >>>>>>>>>> like just because there is code there that takes a certain
> >>>>>> approach,
> >>>>>>>> that
> >>>>>>>>>> the approach is somehow sacred. I had to implement something to
> >>>>>>>>> crystallize
> >>>>>>>>>> my own thinking about how the problem could be approached. I
> >>>>> won't
> >>>>>> be
> >>>>>>>>>> disappointed if, as a community, we decide a different
> >>>> direction
> >>>>> is
> >>>>>>>>> better
> >>>>>>>>>> and the code all gets thrown away. That's one of the reasons
> >>>>> that I
> >>>>>>>>> removed
> >>>>>>>>>> the 0.14.0 milestone that was added to the patch. (I don't want
> >>>>> to
> >>>>>>> rush
> >>>>>>>>> it,
> >>>>>>>>>> nor do I think that's a good idea.)
> >>>>>>>>>>
> >>>>>>>>>> In general: Sounds like we could do with some more
> >>>> formalization
> >>>>>>> around
> >>>>>>>>>> what a proposal looks like, which sorts of changes need one,
> >>>> and
> >>>>>> when
> >>>>>>>> in
> >>>>>>>>>> the dev cycle it is appropriate. FWIW I think Kafka's process
> >>>> is
> >>>>>> more
> >>>>>>>> or
> >>>>>>>>>> less fine, and would be okay with adopting it for Druid if
> >>>> people
> >>>>>>> like
> >>>>>>>>> it.
> >>>>>>>>>> Right now our standards for what requires a "design review" are
> >>>>>> very
> >>>>>>>>>> similar to the Kafka community standards for what requires a
> >>>> KIP,
> >>>>>> so
> >>>>>>> we
> >>>>>>>>>> have some familiarity with those concepts. However we don't
> >>>>>> separate
> >>>>>>> PR
> >>>>>>>>>> review and proposal discussion as strictly as they do, which
> >>>>> seems
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>> the foundation for the feeling of exclusion that is being felt
> >>>>>> here.
> >>>>>>>>>>
> >>>>>>>>>> Separately: I just redid the description on
> >>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794 to be more
> >>>>>>>>> proposal-y.
> >>>>>>>>>> I followed the KIP style:
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >>>>>>>>> .
> >>>>>>>>>> Please refresh the page and see if it looks more useful.
> >>>>>>>>>>
> >>>>>>>>>> Gian
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Slim,
> >>>>>>>>>>>
> >>>>>>>>>>> I agree with your points that offline development is bad for
> >>>>>>>> community.
> >>>>>>>>>>> But I don’t think you need much mentor help. You have raised
> >>>>> valid
> >>>>>>>>> issues
> >>>>>>>>>>> and the Druid community needs to decide what its development
> >>>>>>> practices
> >>>>>>>>>>> should be.
> >>>>>>>>>>>
> >>>>>>>>>>> Julian
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
> >>>> bslim@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hello everyone and hope you all have very good holidays.
> >>>>>>>>>>>>
> >>>>>>>>>>>> First, this email is not directed on the author or the PR
> >>>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794  it
> >>>> self,
> >>>>>> but
> >>>>>>> i
> >>>>>>>>> see
> >>>>>>>>>>>> this PR as a perfect example.
> >>>>>>>>>>>>
> >>>>>>>>>>>> One of the foundation of Apache Way or what i would simply
> >>>> call
> >>>>>>> open
> >>>>>>>>>>> source
> >>>>>>>>>>>> community driven development is that "Technical decisions are
> >>>>>>>>> discussed,
> >>>>>>>>>>>> decided, and archived publicly.
> >>>>>>>>>>>> developpement"
> >>>>>>>>>>>> Which means that big technical  changes such as the one
> >>>> brought
> >>>>>> by
> >>>>>>>>> #/6794
> >>>>>>>>>>>> should have started as a proposal and round of discussions
> >>>>> about
> >>>>>>> the
> >>>>>>>>>>> major
> >>>>>>>>>>>> changes designs not as 11K line of code.
> >>>>>>>>>>>> I believe such openness will promote a lot of good benefits
> >>>>> such
> >>>>>>> as:
> >>>>>>>>>>>>
> >>>>>>>>>>>> - ensures community health and growth.
> >>>>>>>>>>>> - ensures everyone can participate not only the authors and
> >>>> his
> >>>>>>>>>>> co-workers.
> >>>>>>>>>>>> - ensures that the project is driven by the community and
> >>>> not a
> >>>>>>> given
> >>>>>>>>>>>> company or an individual.
> >>>>>>>>>>>> - ensures that there is consensus (not saying 100%
> >>>> agreement;)
> >>>>>>>> however
> >>>>>>>>> it
> >>>>>>>>>>>> means that all individuals will accept the current progress
> >>>> on
> >>>>>> the
> >>>>>>>>>>> project
> >>>>>>>>>>>> until some better proposal is put forth.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Personally such BIG offline PR makes me feel excluded and
> >>>>> doesn't
> >>>>>>>> give
> >>>>>>>>>>> me a
> >>>>>>>>>>>> sense that i belong to  a community at all.
> >>>>>>>>>>>>
> >>>>>>>>>>>> To prevent such off list development i think as a Druid
> >>>>> Community
> >>>>>>> we
> >>>>>>>>> need
> >>>>>>>>>>>> to stick to the apache way “If it didn’t happen on the
> >>>> mailing
> >>>>>>> list,
> >>>>>>>> it
> >>>>>>>>>>>> didn’t happen.”
> >>>>>>>>>>>>
> >>>>>>>>>>>> I would appreciate if some of the Apache mentor help with
> >>>> this.
> >>>>>>>>>>>> Thanks
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>
> ---------------------------------------------------------------------
> >>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> >>>>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> >>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> B-Slim
> >>>>>
> >> _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> >>>>>
> >>>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> >> For additional commands, e-mail: dev-help@druid.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> For additional commands, e-mail: dev-help@druid.apache.org
>
>

Re: Off list major development

[Julian Hyde <jh...@apache.org>]

Statically, yes, GitHub PRs are the same as GitHub cases. But dynamically, they are different, because you can only log a PR when you have finished work.

A lot of other Apache projects use JIRA, so there is a clear distinction between cases and contributions. JIRA cases, especially when logged early in the lifecycle of a contribution, become long-running conversation threads with a lot of community participation. If the Druid chose to do so, GitHub cases could be the same.

Be careful that you do not treat “potential contributors” (by which I presume you mean non-committers) differently from committers and PMC members. Anyone starting a major piece of work should follow the same process. (Experienced committers probably have a somewhat better idea what work will turn out to be “major”, so they get a little more leeway.)

Julian


> On Jan 7, 2019, at 12:10 PM, Gian Merlino <gi...@apache.org> wrote:
> 
> I don't think there's a need to raise issues for every change: a small bug
> fix or doc fix should just go straight to PR. (GitHub PRs show up as issues
> in the issue-search UI/API, so it's not like this means the patch has no
> corresponding issue -- in a sense the PR _is_ the issue.)
> 
> I do think it makes sense to encourage potential contributors to write to
> the dev list or raise an issue if they aren't sure if something would need
> to go through a more heavy weight process.
> 
> Fwiw we do have a set of 'design review' criteria already (we had a
> discussion about this a couple years ago) at:
> http://druid.io/community/#getting-your-changes-accepted. So we wouldn't be
> starting from zero on defining that. We set it up back when we were trying
> to _streamline_ our process -- we used to require two non-author +1s for
> _every_ change, even minor ones. The introduction of design review criteria
> was meant to classify which PRs need that level of review and which ones
> are minor and can be merged with less review. I do think it helped with
> getting minor PRs merged more quickly. The list of criteria is,
> 
> - Major architectural changes or API changes
> - HTTP requests and responses (e. g. a new HTTP endpoint)
> - Interfaces for extensions
> - Server configuration (e. g. altering the behavior of a config property)
> - Emitted metrics
> - Other major changes, judged by the discretion of Druid committers
> 
> Some of it is subjective, but it has been in place for a while, so it's at
> least something we are relatively familiar with.
> 
> On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <jh...@apache.org> wrote:
> 
>> Small contributions don’t need any design review, whereas large
>> contributions need significant review. I don’t think we should require an
>> additional step for those (many) small contributions. But who decides
>> whether a contribution fits into the small or large category?
>> 
>> I think the solution is for authors to log a case (or send an email to
>> dev) before they start work on any contribution. Then committers can
>> request a more heavy-weight process if they think it is needed.
>> 
>> Julian
>> 
>> 
>>> On Jan 7, 2019, at 11:24 AM, Gian Merlino <gi...@apache.org> wrote:
>>> 
>>> It sounds like splitting design from code review is a common theme in a
>> few
>>> of the posts here. How does everyone feel about making a point of
>>> encouraging design reviews to be done as issues, separate from the pull
>>> request, with the expectations that (1) the design review issue
>>> ("proposal") should generally appear somewhat _before_ the pull request;
>>> (2) pull requests should _not_ have design review happen on them, meaning
>>> there should no longer be PRs with design review tags, and we should move
>>> the design review approval process to the issue rather than the PR.
>>> 
>>> For (1), even if we encourage design review discussions to start before a
>>> pull request appears, I don't see an issue with them running concurrently
>>> for a while at some point.
>>> 
>>> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <jo...@apache.org> wrote:
>>> 
>>>> Thanks for raising these concerns!
>>>> 
>>>> My initial thoughts:
>>>> - I agree that separation of design review and code-level review for
>> major
>>>> changes would be more efficient
>>>> 
>>>> - I agree that a clear, more formalized process for handling major
>> changes
>>>> would be helpful for contributors:
>>>> - Define what is considered a major change
>>>> - Define a standard proposal structure, KIP-style proposal format
>> sounds
>>>> good to me
>>>> 
>>>> - I think it's too rigid to have a policy of "no code at all with the
>>>> initial proposal"
>>>> - Code samples can be useful references for understanding aspects of a
>>>> design
>>>> - In some cases it's necessary to run experiments to fully understand a
>>>> problem and determine an appropriate design, or to determine whether
>>>> something is even worth doing before committing to the work of fleshing
>> out
>>>> a proposal, prototype code is a natural outcome of that and I'm not
>> against
>>>> someone providing such code for reference
>>>> - I tend to view design/code as things that are often developed
>>>> simultaneously in an intertwined way
>>>> 
>>>>> Let's not be naive this is very rare that a contributor will accept
>> that
>>>> his work is to be thrown, usually devs takes coding as personal creation
>>>> and they get attached to it.
>>>> 
>>>> If we have a clear review process that emphasizes the need for early
>>>> consensus building, with separate design and code review, then I feel
>> we've
>>>> done enough and don't need a hard rule against having some code linked
>> with
>>>> the initial proposal. If a potential contributor then still wants to go
>>>> ahead and write a lot of code that may be rejected or change
>> significantly,
>>>> the risks were made clear.
>>>> 
>>>>> Once code is written hard to think abstract.
>>>> 
>>>> I can see the validity of the concern, but I personally don't see it as
>> a
>>>> huge risk. My impression from the Druid PR reviews I've seen is that our
>>>> reviewers are able to keep abstract design vs. implementation details
>>>> separate and consider alternate designs when reviewing.
>>>> 
>>>> To summarize I think it's probably enough to have a policy along the
>> lines
>>>> of:
>>>> - Create more formalized guidelines for proposals and what changes
>> require
>>>> proposals
>>>> - Separate design and code review for major changes, with design review
>>>> first, code-level review after reaching consensus on the design.
>>>> - Code before the design review is completed is just for reference, not
>>>> regarded as a candidate for review/merging.
>>>> 
>>>> - Jon
>>>> 
>>>> 
>>>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
>> slim.bouguerra@gmail.com>
>>>> wrote:
>>>> 
>>>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <cl...@imply.io>
>>>> wrote:
>>>>> 
>>>>>> I am definitely biased in this matter as an owner of another large PR
>>>>> that
>>>>>> wasn't preceded by a direct proposal or dev list discussion, and in
>>>>> general
>>>>>> I agree that proposal first is usually better, but I think in some
>>>> rarer
>>>>>> cases approaching a problem code first *is* the most appropriate way
>> to
>>>>>> have a discussion.
>>>>> 
>>>>> 
>>>>> I am wondering here what is the case where code first is better?
>>>>> In general when you are writing code you have an idea about what you
>> want
>>>>> to change, why you want to change and why you want to change it.
>>>>> I do not see what is wrong with sharing this primitive ideas and
>> thoughts
>>>>> as an abstract proposal (at least to avoid overlapping)
>>>>> 
>>>>> I see nothing wrong with it so long as the author
>>>>>> accepts that the PR is treated as a combined proposal and proof of
>>>>> concept,
>>>>>> and fair game to be radically changed via discussion or even rejected,
>>>>>> which sounds like Gian's attitude on the matter and is mine as well
>>>> with
>>>>> my
>>>>>> compression stuff.
>>>>> 
>>>>> 
>>>>> Let's not be naive this is very rare that a contributor will accept
>> that
>>>>> his work is to be thrown, usually devs takes coding as personal
>> creation
>>>>> and they get attached to it.
>>>>> To my point you can take a look on some old issue in the Druid forum
>>>>> 
>>>> 
>> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
>>>>> and am sure other communities have similar problems.
>>>>> So leaving the door open to some side cases is not a good idea in my
>>>>> opinion and will lead to similar issue in the future.
>>>>> 
>>>>> This seems to me especially likely to happen in cases
>>>>>> where an approach still needs proven to be a viable idea *to the
>>>> author*,
>>>>>> so that a much more productive discussion can be had in the first
>>>> place.
>>>>>> 
>>>>>> I think there is a trade off, I don't think we want to discourage
>>>>>> experimentation by walling it off behind mandatory discussions before
>>>> it
>>>>>> can even start, but I do think formalizing the process for large
>>>> changes
>>>>> is
>>>>>> a good thing, especially since we probably can't expect the wider
>>>>> community
>>>>>> to have the same attitude towards a large PR getting discarded as a
>>>>>> committer might. I think the Kafka approach is reasonable, a bit more
>>>>>> formal than our design review process but not overbearing.
>>>>> 
>>>>> 
>>>>> Can you please explain what is overbearing ? what can be changed to
>> make
>>>> it
>>>>> easy ?
>>>>> Most of the points are kind of the actual questions that you want to
>>>>> address before hand anyway isn't ?
>>>>> 
>>>>> 
>>>>>> Going code first
>>>>>> should be in general discouraged, but when it does happen, it seems
>>>> like
>>>>>> opening DIP/an issue/starting a mailing list thread or whatever we go
>>>>> with
>>>>>> to have a more high level design discussion alongside the reference PR
>>>>>> could alleviate some of these complaints?
>>>>> 
>>>>> 
>>>>> What are the complaints ?
>>>>> 
>>>>> 
>>>>>> +1 for "DIP" heh, I think making
>>>>>> them in the form of github issues is probably appropriate, with a dev
>>>>> list
>>>>>> thread to announce them perhaps?
>>>>>> 
>>>>> 
>>>>> I think  github issue with [Proposal] header like
>>>>> https://github.com/apache/incubator-druid/issues/4349 is good to me,
>>>>> 
>>>>> Thanks!
>>>>> 
>>>>> 
>>>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <bs...@apache.org>
>>>> wrote:
>>>>>> 
>>>>>>> Thanks everyone for interacting with this thread.
>>>>>>> 
>>>>>>> The fact that i, Roman, Jihoon  and others in the past (FJ
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ)
>>>>>>> raised this point indicates that PRs without a proposal are indeed an
>>>>>> issue
>>>>>>> and we need to solve it.
>>>>>>> 
>>>>>>> Something Similar to KIP maybe called DIPs is fine with me.
>>>>>>> What i strive to see is the following:
>>>>>>> 
>>>>>>> [Step 1] formalize what is the kind of work that needs a formal
>>>>>> Proposal, I
>>>>>>> think Roman and Jihoon has already covered that pretty well. am +1 on
>>>>>> that.
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
>>>>>>> 
>>>>>>> I am strongly in favor of the separation of Proposal Review and
>>>> (later)
>>>>>>> Code review PRs. My  main reasons:
>>>>>>> Most importantly code reviewing will introduce lot of noise and will
>>>>>>> ultimately make  the GitHub page unreadable.
>>>>>>> Avoid overlapping of work.
>>>>>>> Once code is written hard to think abstract.
>>>>>>> Separate page for Design review later can always be used it as a
>>>> Design
>>>>>>> document that is readable and code free-ish.
>>>>>>> As i said the goal of this first round is to see if the community
>>>> agree
>>>>>>> about such change, then make the process of design more inclusive
>>>> thus
>>>>>>> other contributors can submit a counter proposals.
>>>>>>> 
>>>>>>> [Step 2] IF everybody agree about that point Step 2 is to define
>>>> which
>>>>>>> medium is used to Publish a primitive form of a CODE FREE Abstract
>>>>>> Proposal
>>>>>>> containing at least the following bullet points.
>>>>>>> - The problem description and motivation
>>>>>>> - Overview of the proposed change
>>>>>>> - Operational impact (compatibility/ plans to upgrades) public API
>>>>>> changes,
>>>>>>> configuration changes, algorithm, and so on
>>>>>>> - Expected benefits and drawbacks
>>>>>>> - Rationale and alternatives
>>>>>>> - Estimate Time to Deliver if possible.
>>>>>>> 
>>>>>>> The way i think this can be is a Github issue where member of the
>>>>>> community
>>>>>>> will interact via comments and the author will be updating the
>>>>>> description
>>>>>>> in the light of comments provided by the community.
>>>>>>> 
>>>>>>> During and near the end of the design discussions the author/s can
>>>>> start
>>>>>>> writing POCs to help guide the review process this naturally will be
>>>> a
>>>>>> Pull
>>>>>>> request with actual code.
>>>>>>> 
>>>>>>> *Now the most important thing is that we need to agree that any work
>>>>> that
>>>>>>> does not align with this formal process will be ignored and the
>>>> author
>>>>>> will
>>>>>>> be asked to start with a DIP*
>>>>>>> *That is what i meant with  “If it didn’t happen on the mailing list,
>>>>> it
>>>>>>> didn’t happen.”*
>>>>>>> 
>>>>>>> Thanks and happy coding!
>>>>>>> 
>>>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <gi...@apache.org> wrote:
>>>>>>> 
>>>>>>>> One of the advantages I see with a more formal process is (like
>>>> Kafka
>>>>>>> KIPs)
>>>>>>>> is that it levels the playing field a bit and sets some ground
>>>> rules
>>>>>> for
>>>>>>>> working together. In a way it can help encourage contributions by
>>>>>> making
>>>>>>> it
>>>>>>>> clear what is expected of potential contributors.
>>>>>>>> 
>>>>>>>> We have a design review process today that is not as formal as
>>>> KIPs,
>>>>>> but
>>>>>>> is
>>>>>>>> somewhat heavier than the one you describe. Maybe we could tweak
>>>> our
>>>>>>>> current one by starting to do design reviews separately from PRs.
>>>>> i.e.,
>>>>>>> for
>>>>>>>> anything that meets our 'design review' criteria, do that on the
>>>> dev
>>>>>> list
>>>>>>>> or in a separate issue, and keep the PR focused on code-level
>>>> stuff.
>>>>>> That
>>>>>>>> way we don't end up trying to do both at once. And it makes it
>>>> easier
>>>>>> to
>>>>>>>> start talking about design before the code is ready, which would be
>>>>>>> better.
>>>>>>>> 
>>>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <jh...@apache.org>
>>>> wrote:
>>>>>>>> 
>>>>>>>>> It’s really hard to say no to a contribution when someone has put
>>>>> in
>>>>>> a
>>>>>>>>> significant amount of work.
>>>>>>>>> 
>>>>>>>>> The following approach is simple and works really well: Before
>>>> you
>>>>>>> start
>>>>>>>>> work, log a case, describing the problem. When you have some
>>>> ideas
>>>>>>> about
>>>>>>>>> design, add those to the case. When you have a code branch, add
>>>> its
>>>>>> URL
>>>>>>>> to
>>>>>>>>> the case. And so forth. At any point in the proceedings, people
>>>> can
>>>>>>> chime
>>>>>>>>> in with their opinions.
>>>>>>>>> 
>>>>>>>>> In my opinion, a formal “design review” process is not necessary.
>>>>>> Just
>>>>>>>>> build consensus iteratively, by starting the conversation early
>>>> in
>>>>>> the
>>>>>>>>> process.
>>>>>>>>> 
>>>>>>>>> Julian
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <gi...@apache.org>
>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> In this particular case: please consider the PR as a proposal.
>>>>>> Don't
>>>>>>>> feel
>>>>>>>>>> like just because there is code there that takes a certain
>>>>>> approach,
>>>>>>>> that
>>>>>>>>>> the approach is somehow sacred. I had to implement something to
>>>>>>>>> crystallize
>>>>>>>>>> my own thinking about how the problem could be approached. I
>>>>> won't
>>>>>> be
>>>>>>>>>> disappointed if, as a community, we decide a different
>>>> direction
>>>>> is
>>>>>>>>> better
>>>>>>>>>> and the code all gets thrown away. That's one of the reasons
>>>>> that I
>>>>>>>>> removed
>>>>>>>>>> the 0.14.0 milestone that was added to the patch. (I don't want
>>>>> to
>>>>>>> rush
>>>>>>>>> it,
>>>>>>>>>> nor do I think that's a good idea.)
>>>>>>>>>> 
>>>>>>>>>> In general: Sounds like we could do with some more
>>>> formalization
>>>>>>> around
>>>>>>>>>> what a proposal looks like, which sorts of changes need one,
>>>> and
>>>>>> when
>>>>>>>> in
>>>>>>>>>> the dev cycle it is appropriate. FWIW I think Kafka's process
>>>> is
>>>>>> more
>>>>>>>> or
>>>>>>>>>> less fine, and would be okay with adopting it for Druid if
>>>> people
>>>>>>> like
>>>>>>>>> it.
>>>>>>>>>> Right now our standards for what requires a "design review" are
>>>>>> very
>>>>>>>>>> similar to the Kafka community standards for what requires a
>>>> KIP,
>>>>>> so
>>>>>>> we
>>>>>>>>>> have some familiarity with those concepts. However we don't
>>>>>> separate
>>>>>>> PR
>>>>>>>>>> review and proposal discussion as strictly as they do, which
>>>>> seems
>>>>>> to
>>>>>>>> be
>>>>>>>>>> the foundation for the feeling of exclusion that is being felt
>>>>>> here.
>>>>>>>>>> 
>>>>>>>>>> Separately: I just redid the description on
>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794 to be more
>>>>>>>>> proposal-y.
>>>>>>>>>> I followed the KIP style:
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>>>>>>>>> .
>>>>>>>>>> Please refresh the page and see if it looks more useful.
>>>>>>>>>> 
>>>>>>>>>> Gian
>>>>>>>>>> 
>>>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Slim,
>>>>>>>>>>> 
>>>>>>>>>>> I agree with your points that offline development is bad for
>>>>>>>> community.
>>>>>>>>>>> But I don’t think you need much mentor help. You have raised
>>>>> valid
>>>>>>>>> issues
>>>>>>>>>>> and the Druid community needs to decide what its development
>>>>>>> practices
>>>>>>>>>>> should be.
>>>>>>>>>>> 
>>>>>>>>>>> Julian
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
>>>> bslim@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hello everyone and hope you all have very good holidays.
>>>>>>>>>>>> 
>>>>>>>>>>>> First, this email is not directed on the author or the PR
>>>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794  it
>>>> self,
>>>>>> but
>>>>>>> i
>>>>>>>>> see
>>>>>>>>>>>> this PR as a perfect example.
>>>>>>>>>>>> 
>>>>>>>>>>>> One of the foundation of Apache Way or what i would simply
>>>> call
>>>>>>> open
>>>>>>>>>>> source
>>>>>>>>>>>> community driven development is that "Technical decisions are
>>>>>>>>> discussed,
>>>>>>>>>>>> decided, and archived publicly.
>>>>>>>>>>>> developpement"
>>>>>>>>>>>> Which means that big technical  changes such as the one
>>>> brought
>>>>>> by
>>>>>>>>> #/6794
>>>>>>>>>>>> should have started as a proposal and round of discussions
>>>>> about
>>>>>>> the
>>>>>>>>>>> major
>>>>>>>>>>>> changes designs not as 11K line of code.
>>>>>>>>>>>> I believe such openness will promote a lot of good benefits
>>>>> such
>>>>>>> as:
>>>>>>>>>>>> 
>>>>>>>>>>>> - ensures community health and growth.
>>>>>>>>>>>> - ensures everyone can participate not only the authors and
>>>> his
>>>>>>>>>>> co-workers.
>>>>>>>>>>>> - ensures that the project is driven by the community and
>>>> not a
>>>>>>> given
>>>>>>>>>>>> company or an individual.
>>>>>>>>>>>> - ensures that there is consensus (not saying 100%
>>>> agreement;)
>>>>>>>> however
>>>>>>>>> it
>>>>>>>>>>>> means that all individuals will accept the current progress
>>>> on
>>>>>> the
>>>>>>>>>>> project
>>>>>>>>>>>> until some better proposal is put forth.
>>>>>>>>>>>> 
>>>>>>>>>>>> Personally such BIG offline PR makes me feel excluded and
>>>>> doesn't
>>>>>>>> give
>>>>>>>>>>> me a
>>>>>>>>>>>> sense that i belong to  a community at all.
>>>>>>>>>>>> 
>>>>>>>>>>>> To prevent such off list development i think as a Druid
>>>>> Community
>>>>>>> we
>>>>>>>>> need
>>>>>>>>>>>> to stick to the apache way “If it didn’t happen on the
>>>> mailing
>>>>>>> list,
>>>>>>>> it
>>>>>>>>>>>> didn’t happen.”
>>>>>>>>>>>> 
>>>>>>>>>>>> I would appreciate if some of the Apache mentor help with
>>>> this.
>>>>>>>>>>>> Thanks
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>>>>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> 
>>>>> B-Slim
>>>>> 
>> _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
>>>>> 
>>>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>> For additional commands, e-mail: dev-help@druid.apache.org
>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org

Re: Off list major development

[Gian Merlino <gi...@apache.org>]

I don't think there's a need to raise issues for every change: a small bug
fix or doc fix should just go straight to PR. (GitHub PRs show up as issues
in the issue-search UI/API, so it's not like this means the patch has no
corresponding issue -- in a sense the PR _is_ the issue.)

I do think it makes sense to encourage potential contributors to write to
the dev list or raise an issue if they aren't sure if something would need
to go through a more heavy weight process.

Fwiw we do have a set of 'design review' criteria already (we had a
discussion about this a couple years ago) at:
http://druid.io/community/#getting-your-changes-accepted. So we wouldn't be
starting from zero on defining that. We set it up back when we were trying
to _streamline_ our process -- we used to require two non-author +1s for
_every_ change, even minor ones. The introduction of design review criteria
was meant to classify which PRs need that level of review and which ones
are minor and can be merged with less review. I do think it helped with
getting minor PRs merged more quickly. The list of criteria is,

- Major architectural changes or API changes
- HTTP requests and responses (e. g. a new HTTP endpoint)
- Interfaces for extensions
- Server configuration (e. g. altering the behavior of a config property)
- Emitted metrics
- Other major changes, judged by the discretion of Druid committers

Some of it is subjective, but it has been in place for a while, so it's at
least something we are relatively familiar with.

On Mon, Jan 7, 2019 at 11:32 AM Julian Hyde <jh...@apache.org> wrote:

> Small contributions don’t need any design review, whereas large
> contributions need significant review. I don’t think we should require an
> additional step for those (many) small contributions. But who decides
> whether a contribution fits into the small or large category?
>
> I think the solution is for authors to log a case (or send an email to
> dev) before they start work on any contribution. Then committers can
> request a more heavy-weight process if they think it is needed.
>
> Julian
>
>
> > On Jan 7, 2019, at 11:24 AM, Gian Merlino <gi...@apache.org> wrote:
> >
> > It sounds like splitting design from code review is a common theme in a
> few
> > of the posts here. How does everyone feel about making a point of
> > encouraging design reviews to be done as issues, separate from the pull
> > request, with the expectations that (1) the design review issue
> > ("proposal") should generally appear somewhat _before_ the pull request;
> > (2) pull requests should _not_ have design review happen on them, meaning
> > there should no longer be PRs with design review tags, and we should move
> > the design review approval process to the issue rather than the PR.
> >
> > For (1), even if we encourage design review discussions to start before a
> > pull request appears, I don't see an issue with them running concurrently
> > for a while at some point.
> >
> > On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <jo...@apache.org> wrote:
> >
> >> Thanks for raising these concerns!
> >>
> >> My initial thoughts:
> >> - I agree that separation of design review and code-level review for
> major
> >> changes would be more efficient
> >>
> >> - I agree that a clear, more formalized process for handling major
> changes
> >> would be helpful for contributors:
> >>  - Define what is considered a major change
> >>  - Define a standard proposal structure, KIP-style proposal format
> sounds
> >> good to me
> >>
> >> - I think it's too rigid to have a policy of "no code at all with the
> >> initial proposal"
> >>  - Code samples can be useful references for understanding aspects of a
> >> design
> >>  - In some cases it's necessary to run experiments to fully understand a
> >> problem and determine an appropriate design, or to determine whether
> >> something is even worth doing before committing to the work of fleshing
> out
> >> a proposal, prototype code is a natural outcome of that and I'm not
> against
> >> someone providing such code for reference
> >>  - I tend to view design/code as things that are often developed
> >> simultaneously in an intertwined way
> >>
> >>> Let's not be naive this is very rare that a contributor will accept
> that
> >> his work is to be thrown, usually devs takes coding as personal creation
> >> and they get attached to it.
> >>
> >> If we have a clear review process that emphasizes the need for early
> >> consensus building, with separate design and code review, then I feel
> we've
> >> done enough and don't need a hard rule against having some code linked
> with
> >> the initial proposal. If a potential contributor then still wants to go
> >> ahead and write a lot of code that may be rejected or change
> significantly,
> >> the risks were made clear.
> >>
> >>> Once code is written hard to think abstract.
> >>
> >> I can see the validity of the concern, but I personally don't see it as
> a
> >> huge risk. My impression from the Druid PR reviews I've seen is that our
> >> reviewers are able to keep abstract design vs. implementation details
> >> separate and consider alternate designs when reviewing.
> >>
> >> To summarize I think it's probably enough to have a policy along the
> lines
> >> of:
> >> - Create more formalized guidelines for proposals and what changes
> require
> >> proposals
> >> - Separate design and code review for major changes, with design review
> >> first, code-level review after reaching consensus on the design.
> >> - Code before the design review is completed is just for reference, not
> >> regarded as a candidate for review/merging.
> >>
> >> - Jon
> >>
> >>
> >> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <
> slim.bouguerra@gmail.com>
> >> wrote:
> >>
> >>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <cl...@imply.io>
> >> wrote:
> >>>
> >>>> I am definitely biased in this matter as an owner of another large PR
> >>> that
> >>>> wasn't preceded by a direct proposal or dev list discussion, and in
> >>> general
> >>>> I agree that proposal first is usually better, but I think in some
> >> rarer
> >>>> cases approaching a problem code first *is* the most appropriate way
> to
> >>>> have a discussion.
> >>>
> >>>
> >>> I am wondering here what is the case where code first is better?
> >>> In general when you are writing code you have an idea about what you
> want
> >>> to change, why you want to change and why you want to change it.
> >>> I do not see what is wrong with sharing this primitive ideas and
> thoughts
> >>> as an abstract proposal (at least to avoid overlapping)
> >>>
> >>> I see nothing wrong with it so long as the author
> >>>> accepts that the PR is treated as a combined proposal and proof of
> >>> concept,
> >>>> and fair game to be radically changed via discussion or even rejected,
> >>>> which sounds like Gian's attitude on the matter and is mine as well
> >> with
> >>> my
> >>>> compression stuff.
> >>>
> >>>
> >>> Let's not be naive this is very rare that a contributor will accept
> that
> >>> his work is to be thrown, usually devs takes coding as personal
> creation
> >>> and they get attached to it.
> >>> To my point you can take a look on some old issue in the Druid forum
> >>>
> >>
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> >>> and am sure other communities have similar problems.
> >>> So leaving the door open to some side cases is not a good idea in my
> >>> opinion and will lead to similar issue in the future.
> >>>
> >>> This seems to me especially likely to happen in cases
> >>>> where an approach still needs proven to be a viable idea *to the
> >> author*,
> >>>> so that a much more productive discussion can be had in the first
> >> place.
> >>>>
> >>>> I think there is a trade off, I don't think we want to discourage
> >>>> experimentation by walling it off behind mandatory discussions before
> >> it
> >>>> can even start, but I do think formalizing the process for large
> >> changes
> >>> is
> >>>> a good thing, especially since we probably can't expect the wider
> >>> community
> >>>> to have the same attitude towards a large PR getting discarded as a
> >>>> committer might. I think the Kafka approach is reasonable, a bit more
> >>>> formal than our design review process but not overbearing.
> >>>
> >>>
> >>> Can you please explain what is overbearing ? what can be changed to
> make
> >> it
> >>> easy ?
> >>> Most of the points are kind of the actual questions that you want to
> >>> address before hand anyway isn't ?
> >>>
> >>>
> >>>> Going code first
> >>>> should be in general discouraged, but when it does happen, it seems
> >> like
> >>>> opening DIP/an issue/starting a mailing list thread or whatever we go
> >>> with
> >>>> to have a more high level design discussion alongside the reference PR
> >>>> could alleviate some of these complaints?
> >>>
> >>>
> >>> What are the complaints ?
> >>>
> >>>
> >>>> +1 for "DIP" heh, I think making
> >>>> them in the form of github issues is probably appropriate, with a dev
> >>> list
> >>>> thread to announce them perhaps?
> >>>>
> >>>
> >>> I think  github issue with [Proposal] header like
> >>> https://github.com/apache/incubator-druid/issues/4349 is good to me,
> >>>
> >>> Thanks!
> >>>
> >>>
> >>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <bs...@apache.org>
> >> wrote:
> >>>>
> >>>>> Thanks everyone for interacting with this thread.
> >>>>>
> >>>>> The fact that i, Roman, Jihoon  and others in the past (FJ
> >>>>>
> >>>>
> >>>
> >>
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ)
> >>>>> raised this point indicates that PRs without a proposal are indeed an
> >>>> issue
> >>>>> and we need to solve it.
> >>>>>
> >>>>> Something Similar to KIP maybe called DIPs is fine with me.
> >>>>> What i strive to see is the following:
> >>>>>
> >>>>> [Step 1] formalize what is the kind of work that needs a formal
> >>>> Proposal, I
> >>>>> think Roman and Jihoon has already covered that pretty well. am +1 on
> >>>> that.
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> >>>>>
> >>>>> I am strongly in favor of the separation of Proposal Review and
> >> (later)
> >>>>> Code review PRs. My  main reasons:
> >>>>> Most importantly code reviewing will introduce lot of noise and will
> >>>>> ultimately make  the GitHub page unreadable.
> >>>>> Avoid overlapping of work.
> >>>>> Once code is written hard to think abstract.
> >>>>> Separate page for Design review later can always be used it as a
> >> Design
> >>>>> document that is readable and code free-ish.
> >>>>> As i said the goal of this first round is to see if the community
> >> agree
> >>>>> about such change, then make the process of design more inclusive
> >> thus
> >>>>> other contributors can submit a counter proposals.
> >>>>>
> >>>>> [Step 2] IF everybody agree about that point Step 2 is to define
> >> which
> >>>>> medium is used to Publish a primitive form of a CODE FREE Abstract
> >>>> Proposal
> >>>>> containing at least the following bullet points.
> >>>>> - The problem description and motivation
> >>>>> - Overview of the proposed change
> >>>>> - Operational impact (compatibility/ plans to upgrades) public API
> >>>> changes,
> >>>>> configuration changes, algorithm, and so on
> >>>>> - Expected benefits and drawbacks
> >>>>> - Rationale and alternatives
> >>>>> - Estimate Time to Deliver if possible.
> >>>>>
> >>>>> The way i think this can be is a Github issue where member of the
> >>>> community
> >>>>> will interact via comments and the author will be updating the
> >>>> description
> >>>>> in the light of comments provided by the community.
> >>>>>
> >>>>> During and near the end of the design discussions the author/s can
> >>> start
> >>>>> writing POCs to help guide the review process this naturally will be
> >> a
> >>>> Pull
> >>>>> request with actual code.
> >>>>>
> >>>>> *Now the most important thing is that we need to agree that any work
> >>> that
> >>>>> does not align with this formal process will be ignored and the
> >> author
> >>>> will
> >>>>> be asked to start with a DIP*
> >>>>> *That is what i meant with  “If it didn’t happen on the mailing list,
> >>> it
> >>>>> didn’t happen.”*
> >>>>>
> >>>>> Thanks and happy coding!
> >>>>>
> >>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <gi...@apache.org> wrote:
> >>>>>
> >>>>>> One of the advantages I see with a more formal process is (like
> >> Kafka
> >>>>> KIPs)
> >>>>>> is that it levels the playing field a bit and sets some ground
> >> rules
> >>>> for
> >>>>>> working together. In a way it can help encourage contributions by
> >>>> making
> >>>>> it
> >>>>>> clear what is expected of potential contributors.
> >>>>>>
> >>>>>> We have a design review process today that is not as formal as
> >> KIPs,
> >>>> but
> >>>>> is
> >>>>>> somewhat heavier than the one you describe. Maybe we could tweak
> >> our
> >>>>>> current one by starting to do design reviews separately from PRs.
> >>> i.e.,
> >>>>> for
> >>>>>> anything that meets our 'design review' criteria, do that on the
> >> dev
> >>>> list
> >>>>>> or in a separate issue, and keep the PR focused on code-level
> >> stuff.
> >>>> That
> >>>>>> way we don't end up trying to do both at once. And it makes it
> >> easier
> >>>> to
> >>>>>> start talking about design before the code is ready, which would be
> >>>>> better.
> >>>>>>
> >>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <jh...@apache.org>
> >> wrote:
> >>>>>>
> >>>>>>> It’s really hard to say no to a contribution when someone has put
> >>> in
> >>>> a
> >>>>>>> significant amount of work.
> >>>>>>>
> >>>>>>> The following approach is simple and works really well: Before
> >> you
> >>>>> start
> >>>>>>> work, log a case, describing the problem. When you have some
> >> ideas
> >>>>> about
> >>>>>>> design, add those to the case. When you have a code branch, add
> >> its
> >>>> URL
> >>>>>> to
> >>>>>>> the case. And so forth. At any point in the proceedings, people
> >> can
> >>>>> chime
> >>>>>>> in with their opinions.
> >>>>>>>
> >>>>>>> In my opinion, a formal “design review” process is not necessary.
> >>>> Just
> >>>>>>> build consensus iteratively, by starting the conversation early
> >> in
> >>>> the
> >>>>>>> process.
> >>>>>>>
> >>>>>>> Julian
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <gi...@apache.org>
> >>> wrote:
> >>>>>>>>
> >>>>>>>> In this particular case: please consider the PR as a proposal.
> >>>> Don't
> >>>>>> feel
> >>>>>>>> like just because there is code there that takes a certain
> >>>> approach,
> >>>>>> that
> >>>>>>>> the approach is somehow sacred. I had to implement something to
> >>>>>>> crystallize
> >>>>>>>> my own thinking about how the problem could be approached. I
> >>> won't
> >>>> be
> >>>>>>>> disappointed if, as a community, we decide a different
> >> direction
> >>> is
> >>>>>>> better
> >>>>>>>> and the code all gets thrown away. That's one of the reasons
> >>> that I
> >>>>>>> removed
> >>>>>>>> the 0.14.0 milestone that was added to the patch. (I don't want
> >>> to
> >>>>> rush
> >>>>>>> it,
> >>>>>>>> nor do I think that's a good idea.)
> >>>>>>>>
> >>>>>>>> In general: Sounds like we could do with some more
> >> formalization
> >>>>> around
> >>>>>>>> what a proposal looks like, which sorts of changes need one,
> >> and
> >>>> when
> >>>>>> in
> >>>>>>>> the dev cycle it is appropriate. FWIW I think Kafka's process
> >> is
> >>>> more
> >>>>>> or
> >>>>>>>> less fine, and would be okay with adopting it for Druid if
> >> people
> >>>>> like
> >>>>>>> it.
> >>>>>>>> Right now our standards for what requires a "design review" are
> >>>> very
> >>>>>>>> similar to the Kafka community standards for what requires a
> >> KIP,
> >>>> so
> >>>>> we
> >>>>>>>> have some familiarity with those concepts. However we don't
> >>>> separate
> >>>>> PR
> >>>>>>>> review and proposal discussion as strictly as they do, which
> >>> seems
> >>>> to
> >>>>>> be
> >>>>>>>> the foundation for the feeling of exclusion that is being felt
> >>>> here.
> >>>>>>>>
> >>>>>>>> Separately: I just redid the description on
> >>>>>>>> https://github.com/apache/incubator-druid/pull/6794 to be more
> >>>>>>> proposal-y.
> >>>>>>>> I followed the KIP style:
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >>>>>>> .
> >>>>>>>> Please refresh the page and see if it looks more useful.
> >>>>>>>>
> >>>>>>>> Gian
> >>>>>>>>
> >>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Slim,
> >>>>>>>>>
> >>>>>>>>> I agree with your points that offline development is bad for
> >>>>>> community.
> >>>>>>>>> But I don’t think you need much mentor help. You have raised
> >>> valid
> >>>>>>> issues
> >>>>>>>>> and the Druid community needs to decide what its development
> >>>>> practices
> >>>>>>>>> should be.
> >>>>>>>>>
> >>>>>>>>> Julian
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
> >> bslim@apache.org>
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hello everyone and hope you all have very good holidays.
> >>>>>>>>>>
> >>>>>>>>>> First, this email is not directed on the author or the PR
> >>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794  it
> >> self,
> >>>> but
> >>>>> i
> >>>>>>> see
> >>>>>>>>>> this PR as a perfect example.
> >>>>>>>>>>
> >>>>>>>>>> One of the foundation of Apache Way or what i would simply
> >> call
> >>>>> open
> >>>>>>>>> source
> >>>>>>>>>> community driven development is that "Technical decisions are
> >>>>>>> discussed,
> >>>>>>>>>> decided, and archived publicly.
> >>>>>>>>>> developpement"
> >>>>>>>>>> Which means that big technical  changes such as the one
> >> brought
> >>>> by
> >>>>>>> #/6794
> >>>>>>>>>> should have started as a proposal and round of discussions
> >>> about
> >>>>> the
> >>>>>>>>> major
> >>>>>>>>>> changes designs not as 11K line of code.
> >>>>>>>>>> I believe such openness will promote a lot of good benefits
> >>> such
> >>>>> as:
> >>>>>>>>>>
> >>>>>>>>>> - ensures community health and growth.
> >>>>>>>>>> - ensures everyone can participate not only the authors and
> >> his
> >>>>>>>>> co-workers.
> >>>>>>>>>> - ensures that the project is driven by the community and
> >> not a
> >>>>> given
> >>>>>>>>>> company or an individual.
> >>>>>>>>>> - ensures that there is consensus (not saying 100%
> >> agreement;)
> >>>>>> however
> >>>>>>> it
> >>>>>>>>>> means that all individuals will accept the current progress
> >> on
> >>>> the
> >>>>>>>>> project
> >>>>>>>>>> until some better proposal is put forth.
> >>>>>>>>>>
> >>>>>>>>>> Personally such BIG offline PR makes me feel excluded and
> >>> doesn't
> >>>>>> give
> >>>>>>>>> me a
> >>>>>>>>>> sense that i belong to  a community at all.
> >>>>>>>>>>
> >>>>>>>>>> To prevent such off list development i think as a Druid
> >>> Community
> >>>>> we
> >>>>>>> need
> >>>>>>>>>> to stick to the apache way “If it didn’t happen on the
> >> mailing
> >>>>> list,
> >>>>>> it
> >>>>>>>>>> didn’t happen.”
> >>>>>>>>>>
> >>>>>>>>>> I would appreciate if some of the Apache mentor help with
> >> this.
> >>>>>>>>>> Thanks
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> >>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> >>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>>
> >>> B-Slim
> >>>
> _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> For additional commands, e-mail: dev-help@druid.apache.org
>
>

Re: Off list major development

[Julian Hyde <jh...@apache.org>]

Small contributions don’t need any design review, whereas large contributions need significant review. I don’t think we should require an additional step for those (many) small contributions. But who decides whether a contribution fits into the small or large category?

I think the solution is for authors to log a case (or send an email to dev) before they start work on any contribution. Then committers can request a more heavy-weight process if they think it is needed.

Julian


> On Jan 7, 2019, at 11:24 AM, Gian Merlino <gi...@apache.org> wrote:
> 
> It sounds like splitting design from code review is a common theme in a few
> of the posts here. How does everyone feel about making a point of
> encouraging design reviews to be done as issues, separate from the pull
> request, with the expectations that (1) the design review issue
> ("proposal") should generally appear somewhat _before_ the pull request;
> (2) pull requests should _not_ have design review happen on them, meaning
> there should no longer be PRs with design review tags, and we should move
> the design review approval process to the issue rather than the PR.
> 
> For (1), even if we encourage design review discussions to start before a
> pull request appears, I don't see an issue with them running concurrently
> for a while at some point.
> 
> On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <jo...@apache.org> wrote:
> 
>> Thanks for raising these concerns!
>> 
>> My initial thoughts:
>> - I agree that separation of design review and code-level review for major
>> changes would be more efficient
>> 
>> - I agree that a clear, more formalized process for handling major changes
>> would be helpful for contributors:
>>  - Define what is considered a major change
>>  - Define a standard proposal structure, KIP-style proposal format sounds
>> good to me
>> 
>> - I think it's too rigid to have a policy of "no code at all with the
>> initial proposal"
>>  - Code samples can be useful references for understanding aspects of a
>> design
>>  - In some cases it's necessary to run experiments to fully understand a
>> problem and determine an appropriate design, or to determine whether
>> something is even worth doing before committing to the work of fleshing out
>> a proposal, prototype code is a natural outcome of that and I'm not against
>> someone providing such code for reference
>>  - I tend to view design/code as things that are often developed
>> simultaneously in an intertwined way
>> 
>>> Let's not be naive this is very rare that a contributor will accept that
>> his work is to be thrown, usually devs takes coding as personal creation
>> and they get attached to it.
>> 
>> If we have a clear review process that emphasizes the need for early
>> consensus building, with separate design and code review, then I feel we've
>> done enough and don't need a hard rule against having some code linked with
>> the initial proposal. If a potential contributor then still wants to go
>> ahead and write a lot of code that may be rejected or change significantly,
>> the risks were made clear.
>> 
>>> Once code is written hard to think abstract.
>> 
>> I can see the validity of the concern, but I personally don't see it as a
>> huge risk. My impression from the Druid PR reviews I've seen is that our
>> reviewers are able to keep abstract design vs. implementation details
>> separate and consider alternate designs when reviewing.
>> 
>> To summarize I think it's probably enough to have a policy along the lines
>> of:
>> - Create more formalized guidelines for proposals and what changes require
>> proposals
>> - Separate design and code review for major changes, with design review
>> first, code-level review after reaching consensus on the design.
>> - Code before the design review is completed is just for reference, not
>> regarded as a candidate for review/merging.
>> 
>> - Jon
>> 
>> 
>> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <sl...@gmail.com>
>> wrote:
>> 
>>> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <cl...@imply.io>
>> wrote:
>>> 
>>>> I am definitely biased in this matter as an owner of another large PR
>>> that
>>>> wasn't preceded by a direct proposal or dev list discussion, and in
>>> general
>>>> I agree that proposal first is usually better, but I think in some
>> rarer
>>>> cases approaching a problem code first *is* the most appropriate way to
>>>> have a discussion.
>>> 
>>> 
>>> I am wondering here what is the case where code first is better?
>>> In general when you are writing code you have an idea about what you want
>>> to change, why you want to change and why you want to change it.
>>> I do not see what is wrong with sharing this primitive ideas and thoughts
>>> as an abstract proposal (at least to avoid overlapping)
>>> 
>>> I see nothing wrong with it so long as the author
>>>> accepts that the PR is treated as a combined proposal and proof of
>>> concept,
>>>> and fair game to be radically changed via discussion or even rejected,
>>>> which sounds like Gian's attitude on the matter and is mine as well
>> with
>>> my
>>>> compression stuff.
>>> 
>>> 
>>> Let's not be naive this is very rare that a contributor will accept that
>>> his work is to be thrown, usually devs takes coding as personal creation
>>> and they get attached to it.
>>> To my point you can take a look on some old issue in the Druid forum
>>> 
>> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
>>> and am sure other communities have similar problems.
>>> So leaving the door open to some side cases is not a good idea in my
>>> opinion and will lead to similar issue in the future.
>>> 
>>> This seems to me especially likely to happen in cases
>>>> where an approach still needs proven to be a viable idea *to the
>> author*,
>>>> so that a much more productive discussion can be had in the first
>> place.
>>>> 
>>>> I think there is a trade off, I don't think we want to discourage
>>>> experimentation by walling it off behind mandatory discussions before
>> it
>>>> can even start, but I do think formalizing the process for large
>> changes
>>> is
>>>> a good thing, especially since we probably can't expect the wider
>>> community
>>>> to have the same attitude towards a large PR getting discarded as a
>>>> committer might. I think the Kafka approach is reasonable, a bit more
>>>> formal than our design review process but not overbearing.
>>> 
>>> 
>>> Can you please explain what is overbearing ? what can be changed to make
>> it
>>> easy ?
>>> Most of the points are kind of the actual questions that you want to
>>> address before hand anyway isn't ?
>>> 
>>> 
>>>> Going code first
>>>> should be in general discouraged, but when it does happen, it seems
>> like
>>>> opening DIP/an issue/starting a mailing list thread or whatever we go
>>> with
>>>> to have a more high level design discussion alongside the reference PR
>>>> could alleviate some of these complaints?
>>> 
>>> 
>>> What are the complaints ?
>>> 
>>> 
>>>> +1 for "DIP" heh, I think making
>>>> them in the form of github issues is probably appropriate, with a dev
>>> list
>>>> thread to announce them perhaps?
>>>> 
>>> 
>>> I think  github issue with [Proposal] header like
>>> https://github.com/apache/incubator-druid/issues/4349 is good to me,
>>> 
>>> Thanks!
>>> 
>>> 
>>>> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <bs...@apache.org>
>> wrote:
>>>> 
>>>>> Thanks everyone for interacting with this thread.
>>>>> 
>>>>> The fact that i, Roman, Jihoon  and others in the past (FJ
>>>>> 
>>>> 
>>> 
>> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ)
>>>>> raised this point indicates that PRs without a proposal are indeed an
>>>> issue
>>>>> and we need to solve it.
>>>>> 
>>>>> Something Similar to KIP maybe called DIPs is fine with me.
>>>>> What i strive to see is the following:
>>>>> 
>>>>> [Step 1] formalize what is the kind of work that needs a formal
>>>> Proposal, I
>>>>> think Roman and Jihoon has already covered that pretty well. am +1 on
>>>> that.
>>>>> 
>>>>> 
>>>> 
>>> 
>> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
>>>>> 
>>>>> I am strongly in favor of the separation of Proposal Review and
>> (later)
>>>>> Code review PRs. My  main reasons:
>>>>> Most importantly code reviewing will introduce lot of noise and will
>>>>> ultimately make  the GitHub page unreadable.
>>>>> Avoid overlapping of work.
>>>>> Once code is written hard to think abstract.
>>>>> Separate page for Design review later can always be used it as a
>> Design
>>>>> document that is readable and code free-ish.
>>>>> As i said the goal of this first round is to see if the community
>> agree
>>>>> about such change, then make the process of design more inclusive
>> thus
>>>>> other contributors can submit a counter proposals.
>>>>> 
>>>>> [Step 2] IF everybody agree about that point Step 2 is to define
>> which
>>>>> medium is used to Publish a primitive form of a CODE FREE Abstract
>>>> Proposal
>>>>> containing at least the following bullet points.
>>>>> - The problem description and motivation
>>>>> - Overview of the proposed change
>>>>> - Operational impact (compatibility/ plans to upgrades) public API
>>>> changes,
>>>>> configuration changes, algorithm, and so on
>>>>> - Expected benefits and drawbacks
>>>>> - Rationale and alternatives
>>>>> - Estimate Time to Deliver if possible.
>>>>> 
>>>>> The way i think this can be is a Github issue where member of the
>>>> community
>>>>> will interact via comments and the author will be updating the
>>>> description
>>>>> in the light of comments provided by the community.
>>>>> 
>>>>> During and near the end of the design discussions the author/s can
>>> start
>>>>> writing POCs to help guide the review process this naturally will be
>> a
>>>> Pull
>>>>> request with actual code.
>>>>> 
>>>>> *Now the most important thing is that we need to agree that any work
>>> that
>>>>> does not align with this formal process will be ignored and the
>> author
>>>> will
>>>>> be asked to start with a DIP*
>>>>> *That is what i meant with  “If it didn’t happen on the mailing list,
>>> it
>>>>> didn’t happen.”*
>>>>> 
>>>>> Thanks and happy coding!
>>>>> 
>>>>> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <gi...@apache.org> wrote:
>>>>> 
>>>>>> One of the advantages I see with a more formal process is (like
>> Kafka
>>>>> KIPs)
>>>>>> is that it levels the playing field a bit and sets some ground
>> rules
>>>> for
>>>>>> working together. In a way it can help encourage contributions by
>>>> making
>>>>> it
>>>>>> clear what is expected of potential contributors.
>>>>>> 
>>>>>> We have a design review process today that is not as formal as
>> KIPs,
>>>> but
>>>>> is
>>>>>> somewhat heavier than the one you describe. Maybe we could tweak
>> our
>>>>>> current one by starting to do design reviews separately from PRs.
>>> i.e.,
>>>>> for
>>>>>> anything that meets our 'design review' criteria, do that on the
>> dev
>>>> list
>>>>>> or in a separate issue, and keep the PR focused on code-level
>> stuff.
>>>> That
>>>>>> way we don't end up trying to do both at once. And it makes it
>> easier
>>>> to
>>>>>> start talking about design before the code is ready, which would be
>>>>> better.
>>>>>> 
>>>>>> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <jh...@apache.org>
>> wrote:
>>>>>> 
>>>>>>> It’s really hard to say no to a contribution when someone has put
>>> in
>>>> a
>>>>>>> significant amount of work.
>>>>>>> 
>>>>>>> The following approach is simple and works really well: Before
>> you
>>>>> start
>>>>>>> work, log a case, describing the problem. When you have some
>> ideas
>>>>> about
>>>>>>> design, add those to the case. When you have a code branch, add
>> its
>>>> URL
>>>>>> to
>>>>>>> the case. And so forth. At any point in the proceedings, people
>> can
>>>>> chime
>>>>>>> in with their opinions.
>>>>>>> 
>>>>>>> In my opinion, a formal “design review” process is not necessary.
>>>> Just
>>>>>>> build consensus iteratively, by starting the conversation early
>> in
>>>> the
>>>>>>> process.
>>>>>>> 
>>>>>>> Julian
>>>>>>> 
>>>>>>> 
>>>>>>>> On Jan 2, 2019, at 12:37 PM, Gian Merlino <gi...@apache.org>
>>> wrote:
>>>>>>>> 
>>>>>>>> In this particular case: please consider the PR as a proposal.
>>>> Don't
>>>>>> feel
>>>>>>>> like just because there is code there that takes a certain
>>>> approach,
>>>>>> that
>>>>>>>> the approach is somehow sacred. I had to implement something to
>>>>>>> crystallize
>>>>>>>> my own thinking about how the problem could be approached. I
>>> won't
>>>> be
>>>>>>>> disappointed if, as a community, we decide a different
>> direction
>>> is
>>>>>>> better
>>>>>>>> and the code all gets thrown away. That's one of the reasons
>>> that I
>>>>>>> removed
>>>>>>>> the 0.14.0 milestone that was added to the patch. (I don't want
>>> to
>>>>> rush
>>>>>>> it,
>>>>>>>> nor do I think that's a good idea.)
>>>>>>>> 
>>>>>>>> In general: Sounds like we could do with some more
>> formalization
>>>>> around
>>>>>>>> what a proposal looks like, which sorts of changes need one,
>> and
>>>> when
>>>>>> in
>>>>>>>> the dev cycle it is appropriate. FWIW I think Kafka's process
>> is
>>>> more
>>>>>> or
>>>>>>>> less fine, and would be okay with adopting it for Druid if
>> people
>>>>> like
>>>>>>> it.
>>>>>>>> Right now our standards for what requires a "design review" are
>>>> very
>>>>>>>> similar to the Kafka community standards for what requires a
>> KIP,
>>>> so
>>>>> we
>>>>>>>> have some familiarity with those concepts. However we don't
>>>> separate
>>>>> PR
>>>>>>>> review and proposal discussion as strictly as they do, which
>>> seems
>>>> to
>>>>>> be
>>>>>>>> the foundation for the feeling of exclusion that is being felt
>>>> here.
>>>>>>>> 
>>>>>>>> Separately: I just redid the description on
>>>>>>>> https://github.com/apache/incubator-druid/pull/6794 to be more
>>>>>>> proposal-y.
>>>>>>>> I followed the KIP style:
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>>>>>>> .
>>>>>>>> Please refresh the page and see if it looks more useful.
>>>>>>>> 
>>>>>>>> Gian
>>>>>>>> 
>>>>>>>> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org>
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Slim,
>>>>>>>>> 
>>>>>>>>> I agree with your points that offline development is bad for
>>>>>> community.
>>>>>>>>> But I don’t think you need much mentor help. You have raised
>>> valid
>>>>>>> issues
>>>>>>>>> and the Druid community needs to decide what its development
>>>>> practices
>>>>>>>>> should be.
>>>>>>>>> 
>>>>>>>>> Julian
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
>> bslim@apache.org>
>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hello everyone and hope you all have very good holidays.
>>>>>>>>>> 
>>>>>>>>>> First, this email is not directed on the author or the PR
>>>>>>>>>> https://github.com/apache/incubator-druid/pull/6794  it
>> self,
>>>> but
>>>>> i
>>>>>>> see
>>>>>>>>>> this PR as a perfect example.
>>>>>>>>>> 
>>>>>>>>>> One of the foundation of Apache Way or what i would simply
>> call
>>>>> open
>>>>>>>>> source
>>>>>>>>>> community driven development is that "Technical decisions are
>>>>>>> discussed,
>>>>>>>>>> decided, and archived publicly.
>>>>>>>>>> developpement"
>>>>>>>>>> Which means that big technical  changes such as the one
>> brought
>>>> by
>>>>>>> #/6794
>>>>>>>>>> should have started as a proposal and round of discussions
>>> about
>>>>> the
>>>>>>>>> major
>>>>>>>>>> changes designs not as 11K line of code.
>>>>>>>>>> I believe such openness will promote a lot of good benefits
>>> such
>>>>> as:
>>>>>>>>>> 
>>>>>>>>>> - ensures community health and growth.
>>>>>>>>>> - ensures everyone can participate not only the authors and
>> his
>>>>>>>>> co-workers.
>>>>>>>>>> - ensures that the project is driven by the community and
>> not a
>>>>> given
>>>>>>>>>> company or an individual.
>>>>>>>>>> - ensures that there is consensus (not saying 100%
>> agreement;)
>>>>>> however
>>>>>>> it
>>>>>>>>>> means that all individuals will accept the current progress
>> on
>>>> the
>>>>>>>>> project
>>>>>>>>>> until some better proposal is put forth.
>>>>>>>>>> 
>>>>>>>>>> Personally such BIG offline PR makes me feel excluded and
>>> doesn't
>>>>>> give
>>>>>>>>> me a
>>>>>>>>>> sense that i belong to  a community at all.
>>>>>>>>>> 
>>>>>>>>>> To prevent such off list development i think as a Druid
>>> Community
>>>>> we
>>>>>>> need
>>>>>>>>>> to stick to the apache way “If it didn’t happen on the
>> mailing
>>>>> list,
>>>>>> it
>>>>>>>>>> didn’t happen.”
>>>>>>>>>> 
>>>>>>>>>> I would appreciate if some of the Apache mentor help with
>> this.
>>>>>>>>>> Thanks
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>>>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>>>>>>> For additional commands, e-mail: dev-help@druid.apache.org
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> 
>>> B-Slim
>>> _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
>>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org

Re: Off list major development

[Gian Merlino <gi...@apache.org>]

It sounds like splitting design from code review is a common theme in a few
of the posts here. How does everyone feel about making a point of
encouraging design reviews to be done as issues, separate from the pull
request, with the expectations that (1) the design review issue
("proposal") should generally appear somewhat _before_ the pull request;
(2) pull requests should _not_ have design review happen on them, meaning
there should no longer be PRs with design review tags, and we should move
the design review approval process to the issue rather than the PR.

For (1), even if we encourage design review discussions to start before a
pull request appears, I don't see an issue with them running concurrently
for a while at some point.

On Thu, Jan 3, 2019 at 5:35 PM Jonathan Wei <jo...@apache.org> wrote:

> Thanks for raising these concerns!
>
> My initial thoughts:
> - I agree that separation of design review and code-level review for major
> changes would be more efficient
>
> - I agree that a clear, more formalized process for handling major changes
> would be helpful for contributors:
>   - Define what is considered a major change
>   - Define a standard proposal structure, KIP-style proposal format sounds
> good to me
>
> - I think it's too rigid to have a policy of "no code at all with the
> initial proposal"
>   - Code samples can be useful references for understanding aspects of a
> design
>   - In some cases it's necessary to run experiments to fully understand a
> problem and determine an appropriate design, or to determine whether
> something is even worth doing before committing to the work of fleshing out
> a proposal, prototype code is a natural outcome of that and I'm not against
> someone providing such code for reference
>   - I tend to view design/code as things that are often developed
> simultaneously in an intertwined way
>
> > Let's not be naive this is very rare that a contributor will accept that
> his work is to be thrown, usually devs takes coding as personal creation
> and they get attached to it.
>
> If we have a clear review process that emphasizes the need for early
> consensus building, with separate design and code review, then I feel we've
> done enough and don't need a hard rule against having some code linked with
> the initial proposal. If a potential contributor then still wants to go
> ahead and write a lot of code that may be rejected or change significantly,
> the risks were made clear.
>
> > Once code is written hard to think abstract.
>
> I can see the validity of the concern, but I personally don't see it as a
> huge risk. My impression from the Druid PR reviews I've seen is that our
> reviewers are able to keep abstract design vs. implementation details
> separate and consider alternate designs when reviewing.
>
> To summarize I think it's probably enough to have a policy along the lines
> of:
>  - Create more formalized guidelines for proposals and what changes require
> proposals
>  - Separate design and code review for major changes, with design review
> first, code-level review after reaching consensus on the design.
>  - Code before the design review is completed is just for reference, not
> regarded as a candidate for review/merging.
>
> - Jon
>
>
> On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <sl...@gmail.com>
> wrote:
>
> > On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <cl...@imply.io>
> wrote:
> >
> > > I am definitely biased in this matter as an owner of another large PR
> > that
> > > wasn't preceded by a direct proposal or dev list discussion, and in
> > general
> > > I agree that proposal first is usually better, but I think in some
> rarer
> > > cases approaching a problem code first *is* the most appropriate way to
> > > have a discussion.
> >
> >
> > I am wondering here what is the case where code first is better?
> > In general when you are writing code you have an idea about what you want
> > to change, why you want to change and why you want to change it.
> > I do not see what is wrong with sharing this primitive ideas and thoughts
> > as an abstract proposal (at least to avoid overlapping)
> >
> > I see nothing wrong with it so long as the author
> > > accepts that the PR is treated as a combined proposal and proof of
> > concept,
> > > and fair game to be radically changed via discussion or even rejected,
> > > which sounds like Gian's attitude on the matter and is mine as well
> with
> > my
> > > compression stuff.
> >
> >
> > Let's not be naive this is very rare that a contributor will accept that
> > his work is to be thrown, usually devs takes coding as personal creation
> > and they get attached to it.
> > To my point you can take a look on some old issue in the Druid forum
> >
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
> >  and am sure other communities have similar problems.
> >  So leaving the door open to some side cases is not a good idea in my
> > opinion and will lead to similar issue in the future.
> >
> > This seems to me especially likely to happen in cases
> > > where an approach still needs proven to be a viable idea *to the
> author*,
> > > so that a much more productive discussion can be had in the first
> place.
> > >
> > > I think there is a trade off, I don't think we want to discourage
> > > experimentation by walling it off behind mandatory discussions before
> it
> > > can even start, but I do think formalizing the process for large
> changes
> > is
> > > a good thing, especially since we probably can't expect the wider
> > community
> > > to have the same attitude towards a large PR getting discarded as a
> > > committer might. I think the Kafka approach is reasonable, a bit more
> > > formal than our design review process but not overbearing.
> >
> >
> > Can you please explain what is overbearing ? what can be changed to make
> it
> > easy ?
> > Most of the points are kind of the actual questions that you want to
> > address before hand anyway isn't ?
> >
> >
> > > Going code first
> > > should be in general discouraged, but when it does happen, it seems
> like
> > > opening DIP/an issue/starting a mailing list thread or whatever we go
> > with
> > > to have a more high level design discussion alongside the reference PR
> > > could alleviate some of these complaints?
> >
> >
> > What are the complaints ?
> >
> >
> > > +1 for "DIP" heh, I think making
> > > them in the form of github issues is probably appropriate, with a dev
> > list
> > > thread to announce them perhaps?
> > >
> >
> > I think  github issue with [Proposal] header like
> > https://github.com/apache/incubator-druid/issues/4349 is good to me,
> >
> > Thanks!
> >
> >
> > > On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <bs...@apache.org>
> wrote:
> > >
> > > > Thanks everyone for interacting with this thread.
> > > >
> > > > The fact that i, Roman, Jihoon  and others in the past (FJ
> > > >
> > >
> >
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ)
> > > > raised this point indicates that PRs without a proposal are indeed an
> > > issue
> > > > and we need to solve it.
> > > >
> > > > Something Similar to KIP maybe called DIPs is fine with me.
> > > > What i strive to see is the following:
> > > >
> > > > [Step 1] formalize what is the kind of work that needs a formal
> > > Proposal, I
> > > > think Roman and Jihoon has already covered that pretty well. am +1 on
> > > that.
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> > > >
> > > > I am strongly in favor of the separation of Proposal Review and
> (later)
> > > > Code review PRs. My  main reasons:
> > > > Most importantly code reviewing will introduce lot of noise and will
> > > > ultimately make  the GitHub page unreadable.
> > > > Avoid overlapping of work.
> > > > Once code is written hard to think abstract.
> > > > Separate page for Design review later can always be used it as a
> Design
> > > > document that is readable and code free-ish.
> > > > As i said the goal of this first round is to see if the community
> agree
> > > > about such change, then make the process of design more inclusive
> thus
> > > > other contributors can submit a counter proposals.
> > > >
> > > > [Step 2] IF everybody agree about that point Step 2 is to define
> which
> > > > medium is used to Publish a primitive form of a CODE FREE Abstract
> > > Proposal
> > > > containing at least the following bullet points.
> > > > - The problem description and motivation
> > > > - Overview of the proposed change
> > > > - Operational impact (compatibility/ plans to upgrades) public API
> > > changes,
> > > > configuration changes, algorithm, and so on
> > > > - Expected benefits and drawbacks
> > > > - Rationale and alternatives
> > > > - Estimate Time to Deliver if possible.
> > > >
> > > > The way i think this can be is a Github issue where member of the
> > > community
> > > > will interact via comments and the author will be updating the
> > > description
> > > > in the light of comments provided by the community.
> > > >
> > > > During and near the end of the design discussions the author/s can
> > start
> > > > writing POCs to help guide the review process this naturally will be
> a
> > > Pull
> > > > request with actual code.
> > > >
> > > > *Now the most important thing is that we need to agree that any work
> > that
> > > > does not align with this formal process will be ignored and the
> author
> > > will
> > > > be asked to start with a DIP*
> > > > *That is what i meant with  “If it didn’t happen on the mailing list,
> > it
> > > > didn’t happen.”*
> > > >
> > > > Thanks and happy coding!
> > > >
> > > > On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <gi...@apache.org> wrote:
> > > >
> > > > > One of the advantages I see with a more formal process is (like
> Kafka
> > > > KIPs)
> > > > > is that it levels the playing field a bit and sets some ground
> rules
> > > for
> > > > > working together. In a way it can help encourage contributions by
> > > making
> > > > it
> > > > > clear what is expected of potential contributors.
> > > > >
> > > > > We have a design review process today that is not as formal as
> KIPs,
> > > but
> > > > is
> > > > > somewhat heavier than the one you describe. Maybe we could tweak
> our
> > > > > current one by starting to do design reviews separately from PRs.
> > i.e.,
> > > > for
> > > > > anything that meets our 'design review' criteria, do that on the
> dev
> > > list
> > > > > or in a separate issue, and keep the PR focused on code-level
> stuff.
> > > That
> > > > > way we don't end up trying to do both at once. And it makes it
> easier
> > > to
> > > > > start talking about design before the code is ready, which would be
> > > > better.
> > > > >
> > > > > On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <jh...@apache.org>
> wrote:
> > > > >
> > > > > > It’s really hard to say no to a contribution when someone has put
> > in
> > > a
> > > > > > significant amount of work.
> > > > > >
> > > > > > The following approach is simple and works really well: Before
> you
> > > > start
> > > > > > work, log a case, describing the problem. When you have some
> ideas
> > > > about
> > > > > > design, add those to the case. When you have a code branch, add
> its
> > > URL
> > > > > to
> > > > > > the case. And so forth. At any point in the proceedings, people
> can
> > > > chime
> > > > > > in with their opinions.
> > > > > >
> > > > > > In my opinion, a formal “design review” process is not necessary.
> > > Just
> > > > > > build consensus iteratively, by starting the conversation early
> in
> > > the
> > > > > > process.
> > > > > >
> > > > > > Julian
> > > > > >
> > > > > >
> > > > > > > On Jan 2, 2019, at 12:37 PM, Gian Merlino <gi...@apache.org>
> > wrote:
> > > > > > >
> > > > > > > In this particular case: please consider the PR as a proposal.
> > > Don't
> > > > > feel
> > > > > > > like just because there is code there that takes a certain
> > > approach,
> > > > > that
> > > > > > > the approach is somehow sacred. I had to implement something to
> > > > > > crystallize
> > > > > > > my own thinking about how the problem could be approached. I
> > won't
> > > be
> > > > > > > disappointed if, as a community, we decide a different
> direction
> > is
> > > > > > better
> > > > > > > and the code all gets thrown away. That's one of the reasons
> > that I
> > > > > > removed
> > > > > > > the 0.14.0 milestone that was added to the patch. (I don't want
> > to
> > > > rush
> > > > > > it,
> > > > > > > nor do I think that's a good idea.)
> > > > > > >
> > > > > > > In general: Sounds like we could do with some more
> formalization
> > > > around
> > > > > > > what a proposal looks like, which sorts of changes need one,
> and
> > > when
> > > > > in
> > > > > > > the dev cycle it is appropriate. FWIW I think Kafka's process
> is
> > > more
> > > > > or
> > > > > > > less fine, and would be okay with adopting it for Druid if
> people
> > > > like
> > > > > > it.
> > > > > > > Right now our standards for what requires a "design review" are
> > > very
> > > > > > > similar to the Kafka community standards for what requires a
> KIP,
> > > so
> > > > we
> > > > > > > have some familiarity with those concepts. However we don't
> > > separate
> > > > PR
> > > > > > > review and proposal discussion as strictly as they do, which
> > seems
> > > to
> > > > > be
> > > > > > > the foundation for the feeling of exclusion that is being felt
> > > here.
> > > > > > >
> > > > > > > Separately: I just redid the description on
> > > > > > > https://github.com/apache/incubator-druid/pull/6794 to be more
> > > > > > proposal-y.
> > > > > > > I followed the KIP style:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > .
> > > > > > > Please refresh the page and see if it looks more useful.
> > > > > > >
> > > > > > > Gian
> > > > > > >
> > > > > > > On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > >> Slim,
> > > > > > >>
> > > > > > >> I agree with your points that offline development is bad for
> > > > > community.
> > > > > > >> But I don’t think you need much mentor help. You have raised
> > valid
> > > > > > issues
> > > > > > >> and the Druid community needs to decide what its development
> > > > practices
> > > > > > >> should be.
> > > > > > >>
> > > > > > >> Julian
> > > > > > >>
> > > > > > >>
> > > > > > >>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <
> bslim@apache.org>
> > > > > wrote:
> > > > > > >>>
> > > > > > >>> Hello everyone and hope you all have very good holidays.
> > > > > > >>>
> > > > > > >>> First, this email is not directed on the author or the PR
> > > > > > >>> https://github.com/apache/incubator-druid/pull/6794  it
> self,
> > > but
> > > > i
> > > > > > see
> > > > > > >>> this PR as a perfect example.
> > > > > > >>>
> > > > > > >>> One of the foundation of Apache Way or what i would simply
> call
> > > > open
> > > > > > >> source
> > > > > > >>> community driven development is that "Technical decisions are
> > > > > > discussed,
> > > > > > >>> decided, and archived publicly.
> > > > > > >>> developpement"
> > > > > > >>> Which means that big technical  changes such as the one
> brought
> > > by
> > > > > > #/6794
> > > > > > >>> should have started as a proposal and round of discussions
> > about
> > > > the
> > > > > > >> major
> > > > > > >>> changes designs not as 11K line of code.
> > > > > > >>> I believe such openness will promote a lot of good benefits
> > such
> > > > as:
> > > > > > >>>
> > > > > > >>> - ensures community health and growth.
> > > > > > >>> - ensures everyone can participate not only the authors and
> his
> > > > > > >> co-workers.
> > > > > > >>> - ensures that the project is driven by the community and
> not a
> > > > given
> > > > > > >>> company or an individual.
> > > > > > >>> - ensures that there is consensus (not saying 100%
> agreement;)
> > > > > however
> > > > > > it
> > > > > > >>> means that all individuals will accept the current progress
> on
> > > the
> > > > > > >> project
> > > > > > >>> until some better proposal is put forth.
> > > > > > >>>
> > > > > > >>> Personally such BIG offline PR makes me feel excluded and
> > doesn't
> > > > > give
> > > > > > >> me a
> > > > > > >>> sense that i belong to  a community at all.
> > > > > > >>>
> > > > > > >>> To prevent such off list development i think as a Druid
> > Community
> > > > we
> > > > > > need
> > > > > > >>> to stick to the apache way “If it didn’t happen on the
> mailing
> > > > list,
> > > > > it
> > > > > > >>> didn’t happen.”
> > > > > > >>>
> > > > > > >>> I would appreciate if some of the Apache mentor help with
> this.
> > > > > > >>> Thanks
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > >> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > > > >> For additional commands, e-mail: dev-help@druid.apache.org
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > > > For additional commands, e-mail: dev-help@druid.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > B-Slim
> > _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
> >
>

Re: Off list major development

[Jonathan Wei <jo...@apache.org>]

Thanks for raising these concerns!

My initial thoughts:
- I agree that separation of design review and code-level review for major
changes would be more efficient

- I agree that a clear, more formalized process for handling major changes
would be helpful for contributors:
  - Define what is considered a major change
  - Define a standard proposal structure, KIP-style proposal format sounds
good to me

- I think it's too rigid to have a policy of "no code at all with the
initial proposal"
  - Code samples can be useful references for understanding aspects of a
design
  - In some cases it's necessary to run experiments to fully understand a
problem and determine an appropriate design, or to determine whether
something is even worth doing before committing to the work of fleshing out
a proposal, prototype code is a natural outcome of that and I'm not against
someone providing such code for reference
  - I tend to view design/code as things that are often developed
simultaneously in an intertwined way

> Let's not be naive this is very rare that a contributor will accept that
his work is to be thrown, usually devs takes coding as personal creation
and they get attached to it.

If we have a clear review process that emphasizes the need for early
consensus building, with separate design and code review, then I feel we've
done enough and don't need a hard rule against having some code linked with
the initial proposal. If a potential contributor then still wants to go
ahead and write a lot of code that may be rejected or change significantly,
the risks were made clear.

> Once code is written hard to think abstract.

I can see the validity of the concern, but I personally don't see it as a
huge risk. My impression from the Druid PR reviews I've seen is that our
reviewers are able to keep abstract design vs. implementation details
separate and consider alternate designs when reviewing.

To summarize I think it's probably enough to have a policy along the lines
of:
 - Create more formalized guidelines for proposals and what changes require
proposals
 - Separate design and code review for major changes, with design review
first, code-level review after reaching consensus on the design.
 - Code before the design review is completed is just for reference, not
regarded as a candidate for review/merging.

- Jon


On Thu, Jan 3, 2019 at 12:48 PM Slim Bouguerra <sl...@gmail.com>
wrote:

> On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <cl...@imply.io> wrote:
>
> > I am definitely biased in this matter as an owner of another large PR
> that
> > wasn't preceded by a direct proposal or dev list discussion, and in
> general
> > I agree that proposal first is usually better, but I think in some rarer
> > cases approaching a problem code first *is* the most appropriate way to
> > have a discussion.
>
>
> I am wondering here what is the case where code first is better?
> In general when you are writing code you have an idea about what you want
> to change, why you want to change and why you want to change it.
> I do not see what is wrong with sharing this primitive ideas and thoughts
> as an abstract proposal (at least to avoid overlapping)
>
> I see nothing wrong with it so long as the author
> > accepts that the PR is treated as a combined proposal and proof of
> concept,
> > and fair game to be radically changed via discussion or even rejected,
> > which sounds like Gian's attitude on the matter and is mine as well with
> my
> > compression stuff.
>
>
> Let's not be naive this is very rare that a contributor will accept that
> his work is to be thrown, usually devs takes coding as personal creation
> and they get attached to it.
> To my point you can take a look on some old issue in the Druid forum
> https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
>  and am sure other communities have similar problems.
>  So leaving the door open to some side cases is not a good idea in my
> opinion and will lead to similar issue in the future.
>
> This seems to me especially likely to happen in cases
> > where an approach still needs proven to be a viable idea *to the author*,
> > so that a much more productive discussion can be had in the first place.
> >
> > I think there is a trade off, I don't think we want to discourage
> > experimentation by walling it off behind mandatory discussions before it
> > can even start, but I do think formalizing the process for large changes
> is
> > a good thing, especially since we probably can't expect the wider
> community
> > to have the same attitude towards a large PR getting discarded as a
> > committer might. I think the Kafka approach is reasonable, a bit more
> > formal than our design review process but not overbearing.
>
>
> Can you please explain what is overbearing ? what can be changed to make it
> easy ?
> Most of the points are kind of the actual questions that you want to
> address before hand anyway isn't ?
>
>
> > Going code first
> > should be in general discouraged, but when it does happen, it seems like
> > opening DIP/an issue/starting a mailing list thread or whatever we go
> with
> > to have a more high level design discussion alongside the reference PR
> > could alleviate some of these complaints?
>
>
> What are the complaints ?
>
>
> > +1 for "DIP" heh, I think making
> > them in the form of github issues is probably appropriate, with a dev
> list
> > thread to announce them perhaps?
> >
>
> I think  github issue with [Proposal] header like
> https://github.com/apache/incubator-druid/issues/4349 is good to me,
>
> Thanks!
>
>
> > On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <bs...@apache.org> wrote:
> >
> > > Thanks everyone for interacting with this thread.
> > >
> > > The fact that i, Roman, Jihoon  and others in the past (FJ
> > >
> >
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ)
> > > raised this point indicates that PRs without a proposal are indeed an
> > issue
> > > and we need to solve it.
> > >
> > > Something Similar to KIP maybe called DIPs is fine with me.
> > > What i strive to see is the following:
> > >
> > > [Step 1] formalize what is the kind of work that needs a formal
> > Proposal, I
> > > think Roman and Jihoon has already covered that pretty well. am +1 on
> > that.
> > >
> > >
> >
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> > >
> > > I am strongly in favor of the separation of Proposal Review and (later)
> > > Code review PRs. My  main reasons:
> > > Most importantly code reviewing will introduce lot of noise and will
> > > ultimately make  the GitHub page unreadable.
> > > Avoid overlapping of work.
> > > Once code is written hard to think abstract.
> > > Separate page for Design review later can always be used it as a Design
> > > document that is readable and code free-ish.
> > > As i said the goal of this first round is to see if the community agree
> > > about such change, then make the process of design more inclusive thus
> > > other contributors can submit a counter proposals.
> > >
> > > [Step 2] IF everybody agree about that point Step 2 is to define which
> > > medium is used to Publish a primitive form of a CODE FREE Abstract
> > Proposal
> > > containing at least the following bullet points.
> > > - The problem description and motivation
> > > - Overview of the proposed change
> > > - Operational impact (compatibility/ plans to upgrades) public API
> > changes,
> > > configuration changes, algorithm, and so on
> > > - Expected benefits and drawbacks
> > > - Rationale and alternatives
> > > - Estimate Time to Deliver if possible.
> > >
> > > The way i think this can be is a Github issue where member of the
> > community
> > > will interact via comments and the author will be updating the
> > description
> > > in the light of comments provided by the community.
> > >
> > > During and near the end of the design discussions the author/s can
> start
> > > writing POCs to help guide the review process this naturally will be a
> > Pull
> > > request with actual code.
> > >
> > > *Now the most important thing is that we need to agree that any work
> that
> > > does not align with this formal process will be ignored and the author
> > will
> > > be asked to start with a DIP*
> > > *That is what i meant with  “If it didn’t happen on the mailing list,
> it
> > > didn’t happen.”*
> > >
> > > Thanks and happy coding!
> > >
> > > On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <gi...@apache.org> wrote:
> > >
> > > > One of the advantages I see with a more formal process is (like Kafka
> > > KIPs)
> > > > is that it levels the playing field a bit and sets some ground rules
> > for
> > > > working together. In a way it can help encourage contributions by
> > making
> > > it
> > > > clear what is expected of potential contributors.
> > > >
> > > > We have a design review process today that is not as formal as KIPs,
> > but
> > > is
> > > > somewhat heavier than the one you describe. Maybe we could tweak our
> > > > current one by starting to do design reviews separately from PRs.
> i.e.,
> > > for
> > > > anything that meets our 'design review' criteria, do that on the dev
> > list
> > > > or in a separate issue, and keep the PR focused on code-level stuff.
> > That
> > > > way we don't end up trying to do both at once. And it makes it easier
> > to
> > > > start talking about design before the code is ready, which would be
> > > better.
> > > >
> > > > On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <jh...@apache.org> wrote:
> > > >
> > > > > It’s really hard to say no to a contribution when someone has put
> in
> > a
> > > > > significant amount of work.
> > > > >
> > > > > The following approach is simple and works really well: Before you
> > > start
> > > > > work, log a case, describing the problem. When you have some ideas
> > > about
> > > > > design, add those to the case. When you have a code branch, add its
> > URL
> > > > to
> > > > > the case. And so forth. At any point in the proceedings, people can
> > > chime
> > > > > in with their opinions.
> > > > >
> > > > > In my opinion, a formal “design review” process is not necessary.
> > Just
> > > > > build consensus iteratively, by starting the conversation early in
> > the
> > > > > process.
> > > > >
> > > > > Julian
> > > > >
> > > > >
> > > > > > On Jan 2, 2019, at 12:37 PM, Gian Merlino <gi...@apache.org>
> wrote:
> > > > > >
> > > > > > In this particular case: please consider the PR as a proposal.
> > Don't
> > > > feel
> > > > > > like just because there is code there that takes a certain
> > approach,
> > > > that
> > > > > > the approach is somehow sacred. I had to implement something to
> > > > > crystallize
> > > > > > my own thinking about how the problem could be approached. I
> won't
> > be
> > > > > > disappointed if, as a community, we decide a different direction
> is
> > > > > better
> > > > > > and the code all gets thrown away. That's one of the reasons
> that I
> > > > > removed
> > > > > > the 0.14.0 milestone that was added to the patch. (I don't want
> to
> > > rush
> > > > > it,
> > > > > > nor do I think that's a good idea.)
> > > > > >
> > > > > > In general: Sounds like we could do with some more formalization
> > > around
> > > > > > what a proposal looks like, which sorts of changes need one, and
> > when
> > > > in
> > > > > > the dev cycle it is appropriate. FWIW I think Kafka's process is
> > more
> > > > or
> > > > > > less fine, and would be okay with adopting it for Druid if people
> > > like
> > > > > it.
> > > > > > Right now our standards for what requires a "design review" are
> > very
> > > > > > similar to the Kafka community standards for what requires a KIP,
> > so
> > > we
> > > > > > have some familiarity with those concepts. However we don't
> > separate
> > > PR
> > > > > > review and proposal discussion as strictly as they do, which
> seems
> > to
> > > > be
> > > > > > the foundation for the feeling of exclusion that is being felt
> > here.
> > > > > >
> > > > > > Separately: I just redid the description on
> > > > > > https://github.com/apache/incubator-druid/pull/6794 to be more
> > > > > proposal-y.
> > > > > > I followed the KIP style:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > .
> > > > > > Please refresh the page and see if it looks more useful.
> > > > > >
> > > > > > Gian
> > > > > >
> > > > > > On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org>
> > > wrote:
> > > > > >
> > > > > >> Slim,
> > > > > >>
> > > > > >> I agree with your points that offline development is bad for
> > > > community.
> > > > > >> But I don’t think you need much mentor help. You have raised
> valid
> > > > > issues
> > > > > >> and the Druid community needs to decide what its development
> > > practices
> > > > > >> should be.
> > > > > >>
> > > > > >> Julian
> > > > > >>
> > > > > >>
> > > > > >>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <bs...@apache.org>
> > > > wrote:
> > > > > >>>
> > > > > >>> Hello everyone and hope you all have very good holidays.
> > > > > >>>
> > > > > >>> First, this email is not directed on the author or the PR
> > > > > >>> https://github.com/apache/incubator-druid/pull/6794  it self,
> > but
> > > i
> > > > > see
> > > > > >>> this PR as a perfect example.
> > > > > >>>
> > > > > >>> One of the foundation of Apache Way or what i would simply call
> > > open
> > > > > >> source
> > > > > >>> community driven development is that "Technical decisions are
> > > > > discussed,
> > > > > >>> decided, and archived publicly.
> > > > > >>> developpement"
> > > > > >>> Which means that big technical  changes such as the one brought
> > by
> > > > > #/6794
> > > > > >>> should have started as a proposal and round of discussions
> about
> > > the
> > > > > >> major
> > > > > >>> changes designs not as 11K line of code.
> > > > > >>> I believe such openness will promote a lot of good benefits
> such
> > > as:
> > > > > >>>
> > > > > >>> - ensures community health and growth.
> > > > > >>> - ensures everyone can participate not only the authors and his
> > > > > >> co-workers.
> > > > > >>> - ensures that the project is driven by the community and not a
> > > given
> > > > > >>> company or an individual.
> > > > > >>> - ensures that there is consensus (not saying 100% agreement;)
> > > > however
> > > > > it
> > > > > >>> means that all individuals will accept the current progress on
> > the
> > > > > >> project
> > > > > >>> until some better proposal is put forth.
> > > > > >>>
> > > > > >>> Personally such BIG offline PR makes me feel excluded and
> doesn't
> > > > give
> > > > > >> me a
> > > > > >>> sense that i belong to  a community at all.
> > > > > >>>
> > > > > >>> To prevent such off list development i think as a Druid
> Community
> > > we
> > > > > need
> > > > > >>> to stick to the apache way “If it didn’t happen on the mailing
> > > list,
> > > > it
> > > > > >>> didn’t happen.”
> > > > > >>>
> > > > > >>> I would appreciate if some of the Apache mentor help with this.
> > > > > >>> Thanks
> > > > > >>
> > > > > >>
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > > >> For additional commands, e-mail: dev-help@druid.apache.org
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > > For additional commands, e-mail: dev-help@druid.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
>
> B-Slim
> _______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______
>

Re: Off list major development

[Slim Bouguerra <sl...@gmail.com>]

On Thu, Jan 3, 2019 at 12:16 PM Clint Wylie <cl...@imply.io> wrote:

> I am definitely biased in this matter as an owner of another large PR that
> wasn't preceded by a direct proposal or dev list discussion, and in general
> I agree that proposal first is usually better, but I think in some rarer
> cases approaching a problem code first *is* the most appropriate way to
> have a discussion.


I am wondering here what is the case where code first is better?
In general when you are writing code you have an idea about what you want
to change, why you want to change and why you want to change it.
I do not see what is wrong with sharing this primitive ideas and thoughts
as an abstract proposal (at least to avoid overlapping)

I see nothing wrong with it so long as the author
> accepts that the PR is treated as a combined proposal and proof of concept,
> and fair game to be radically changed via discussion or even rejected,
> which sounds like Gian's attitude on the matter and is mine as well with my
> compression stuff.


Let's not be naive this is very rare that a contributor will accept that
his work is to be thrown, usually devs takes coding as personal creation
and they get attached to it.
To my point you can take a look on some old issue in the Druid forum
https://github.com/apache/incubator-druid/pull/3755#issuecomment-265667690
 and am sure other communities have similar problems.
 So leaving the door open to some side cases is not a good idea in my
opinion and will lead to similar issue in the future.

This seems to me especially likely to happen in cases
> where an approach still needs proven to be a viable idea *to the author*,
> so that a much more productive discussion can be had in the first place.
>
> I think there is a trade off, I don't think we want to discourage
> experimentation by walling it off behind mandatory discussions before it
> can even start, but I do think formalizing the process for large changes is
> a good thing, especially since we probably can't expect the wider community
> to have the same attitude towards a large PR getting discarded as a
> committer might. I think the Kafka approach is reasonable, a bit more
> formal than our design review process but not overbearing.


Can you please explain what is overbearing ? what can be changed to make it
easy ?
Most of the points are kind of the actual questions that you want to
address before hand anyway isn't ?


> Going code first
> should be in general discouraged, but when it does happen, it seems like
> opening DIP/an issue/starting a mailing list thread or whatever we go with
> to have a more high level design discussion alongside the reference PR
> could alleviate some of these complaints?


What are the complaints ?


> +1 for "DIP" heh, I think making
> them in the form of github issues is probably appropriate, with a dev list
> thread to announce them perhaps?
>

I think  github issue with [Proposal] header like
https://github.com/apache/incubator-druid/issues/4349 is good to me,

Thanks!


> On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <bs...@apache.org> wrote:
>
> > Thanks everyone for interacting with this thread.
> >
> > The fact that i, Roman, Jihoon  and others in the past (FJ
> >
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ)
> > raised this point indicates that PRs without a proposal are indeed an
> issue
> > and we need to solve it.
> >
> > Something Similar to KIP maybe called DIPs is fine with me.
> > What i strive to see is the following:
> >
> > [Step 1] formalize what is the kind of work that needs a formal
> Proposal, I
> > think Roman and Jihoon has already covered that pretty well. am +1 on
> that.
> >
> >
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
> >
> > I am strongly in favor of the separation of Proposal Review and (later)
> > Code review PRs. My  main reasons:
> > Most importantly code reviewing will introduce lot of noise and will
> > ultimately make  the GitHub page unreadable.
> > Avoid overlapping of work.
> > Once code is written hard to think abstract.
> > Separate page for Design review later can always be used it as a Design
> > document that is readable and code free-ish.
> > As i said the goal of this first round is to see if the community agree
> > about such change, then make the process of design more inclusive thus
> > other contributors can submit a counter proposals.
> >
> > [Step 2] IF everybody agree about that point Step 2 is to define which
> > medium is used to Publish a primitive form of a CODE FREE Abstract
> Proposal
> > containing at least the following bullet points.
> > - The problem description and motivation
> > - Overview of the proposed change
> > - Operational impact (compatibility/ plans to upgrades) public API
> changes,
> > configuration changes, algorithm, and so on
> > - Expected benefits and drawbacks
> > - Rationale and alternatives
> > - Estimate Time to Deliver if possible.
> >
> > The way i think this can be is a Github issue where member of the
> community
> > will interact via comments and the author will be updating the
> description
> > in the light of comments provided by the community.
> >
> > During and near the end of the design discussions the author/s can start
> > writing POCs to help guide the review process this naturally will be a
> Pull
> > request with actual code.
> >
> > *Now the most important thing is that we need to agree that any work that
> > does not align with this formal process will be ignored and the author
> will
> > be asked to start with a DIP*
> > *That is what i meant with  “If it didn’t happen on the mailing list, it
> > didn’t happen.”*
> >
> > Thanks and happy coding!
> >
> > On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <gi...@apache.org> wrote:
> >
> > > One of the advantages I see with a more formal process is (like Kafka
> > KIPs)
> > > is that it levels the playing field a bit and sets some ground rules
> for
> > > working together. In a way it can help encourage contributions by
> making
> > it
> > > clear what is expected of potential contributors.
> > >
> > > We have a design review process today that is not as formal as KIPs,
> but
> > is
> > > somewhat heavier than the one you describe. Maybe we could tweak our
> > > current one by starting to do design reviews separately from PRs. i.e.,
> > for
> > > anything that meets our 'design review' criteria, do that on the dev
> list
> > > or in a separate issue, and keep the PR focused on code-level stuff.
> That
> > > way we don't end up trying to do both at once. And it makes it easier
> to
> > > start talking about design before the code is ready, which would be
> > better.
> > >
> > > On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <jh...@apache.org> wrote:
> > >
> > > > It’s really hard to say no to a contribution when someone has put in
> a
> > > > significant amount of work.
> > > >
> > > > The following approach is simple and works really well: Before you
> > start
> > > > work, log a case, describing the problem. When you have some ideas
> > about
> > > > design, add those to the case. When you have a code branch, add its
> URL
> > > to
> > > > the case. And so forth. At any point in the proceedings, people can
> > chime
> > > > in with their opinions.
> > > >
> > > > In my opinion, a formal “design review” process is not necessary.
> Just
> > > > build consensus iteratively, by starting the conversation early in
> the
> > > > process.
> > > >
> > > > Julian
> > > >
> > > >
> > > > > On Jan 2, 2019, at 12:37 PM, Gian Merlino <gi...@apache.org> wrote:
> > > > >
> > > > > In this particular case: please consider the PR as a proposal.
> Don't
> > > feel
> > > > > like just because there is code there that takes a certain
> approach,
> > > that
> > > > > the approach is somehow sacred. I had to implement something to
> > > > crystallize
> > > > > my own thinking about how the problem could be approached. I won't
> be
> > > > > disappointed if, as a community, we decide a different direction is
> > > > better
> > > > > and the code all gets thrown away. That's one of the reasons that I
> > > > removed
> > > > > the 0.14.0 milestone that was added to the patch. (I don't want to
> > rush
> > > > it,
> > > > > nor do I think that's a good idea.)
> > > > >
> > > > > In general: Sounds like we could do with some more formalization
> > around
> > > > > what a proposal looks like, which sorts of changes need one, and
> when
> > > in
> > > > > the dev cycle it is appropriate. FWIW I think Kafka's process is
> more
> > > or
> > > > > less fine, and would be okay with adopting it for Druid if people
> > like
> > > > it.
> > > > > Right now our standards for what requires a "design review" are
> very
> > > > > similar to the Kafka community standards for what requires a KIP,
> so
> > we
> > > > > have some familiarity with those concepts. However we don't
> separate
> > PR
> > > > > review and proposal discussion as strictly as they do, which seems
> to
> > > be
> > > > > the foundation for the feeling of exclusion that is being felt
> here.
> > > > >
> > > > > Separately: I just redid the description on
> > > > > https://github.com/apache/incubator-druid/pull/6794 to be more
> > > > proposal-y.
> > > > > I followed the KIP style:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > .
> > > > > Please refresh the page and see if it looks more useful.
> > > > >
> > > > > Gian
> > > > >
> > > > > On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org>
> > wrote:
> > > > >
> > > > >> Slim,
> > > > >>
> > > > >> I agree with your points that offline development is bad for
> > > community.
> > > > >> But I don’t think you need much mentor help. You have raised valid
> > > > issues
> > > > >> and the Druid community needs to decide what its development
> > practices
> > > > >> should be.
> > > > >>
> > > > >> Julian
> > > > >>
> > > > >>
> > > > >>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <bs...@apache.org>
> > > wrote:
> > > > >>>
> > > > >>> Hello everyone and hope you all have very good holidays.
> > > > >>>
> > > > >>> First, this email is not directed on the author or the PR
> > > > >>> https://github.com/apache/incubator-druid/pull/6794  it self,
> but
> > i
> > > > see
> > > > >>> this PR as a perfect example.
> > > > >>>
> > > > >>> One of the foundation of Apache Way or what i would simply call
> > open
> > > > >> source
> > > > >>> community driven development is that "Technical decisions are
> > > > discussed,
> > > > >>> decided, and archived publicly.
> > > > >>> developpement"
> > > > >>> Which means that big technical  changes such as the one brought
> by
> > > > #/6794
> > > > >>> should have started as a proposal and round of discussions about
> > the
> > > > >> major
> > > > >>> changes designs not as 11K line of code.
> > > > >>> I believe such openness will promote a lot of good benefits such
> > as:
> > > > >>>
> > > > >>> - ensures community health and growth.
> > > > >>> - ensures everyone can participate not only the authors and his
> > > > >> co-workers.
> > > > >>> - ensures that the project is driven by the community and not a
> > given
> > > > >>> company or an individual.
> > > > >>> - ensures that there is consensus (not saying 100% agreement;)
> > > however
> > > > it
> > > > >>> means that all individuals will accept the current progress on
> the
> > > > >> project
> > > > >>> until some better proposal is put forth.
> > > > >>>
> > > > >>> Personally such BIG offline PR makes me feel excluded and doesn't
> > > give
> > > > >> me a
> > > > >>> sense that i belong to  a community at all.
> > > > >>>
> > > > >>> To prevent such off list development i think as a Druid Community
> > we
> > > > need
> > > > >>> to stick to the apache way “If it didn’t happen on the mailing
> > list,
> > > it
> > > > >>> didn’t happen.”
> > > > >>>
> > > > >>> I would appreciate if some of the Apache mentor help with this.
> > > > >>> Thanks
> > > > >>
> > > > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > >> For additional commands, e-mail: dev-help@druid.apache.org
> > > > >>
> > > > >>
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > For additional commands, e-mail: dev-help@druid.apache.org
> > > >
> > > >
> > >
> >
>


-- 

B-Slim
_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______

Re: Off list major development

[Clint Wylie <cl...@imply.io>]

I am definitely biased in this matter as an owner of another large PR that
wasn't preceded by a direct proposal or dev list discussion, and in general
I agree that proposal first is usually better, but I think in some rarer
cases approaching a problem code first *is* the most appropriate way to
have a discussion. I see nothing wrong with it so long as the author
accepts that the PR is treated as a combined proposal and proof of concept,
and fair game to be radically changed via discussion or even rejected,
which sounds like Gian's attitude on the matter and is mine as well with my
compression stuff. This seems to me especially likely to happen in cases
where an approach still needs proven to be a viable idea *to the author*,
so that a much more productive discussion can be had in the first place.

I think there is a trade off, I don't think we want to discourage
experimentation by walling it off behind mandatory discussions before it
can even start, but I do think formalizing the process for large changes is
a good thing, especially since we probably can't expect the wider community
to have the same attitude towards a large PR getting discarded as a
committer might. I think the Kafka approach is reasonable, a bit more
formal than our design review process but not overbearing. Going code first
should be in general discouraged, but when it does happen, it seems like
opening DIP/an issue/starting a mailing list thread or whatever we go with
to have a more high level design discussion alongside the reference PR
could alleviate some of these complaints? +1 for "DIP" heh, I think making
them in the form of github issues is probably appropriate, with a dev list
thread to announce them perhaps?

On Thu, Jan 3, 2019 at 10:28 AM Slim Bouguerra <bs...@apache.org> wrote:

> Thanks everyone for interacting with this thread.
>
> The fact that i, Roman, Jihoon  and others in the past (FJ
> https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ)
> raised this point indicates that PRs without a proposal are indeed an issue
> and we need to solve it.
>
> Something Similar to KIP maybe called DIPs is fine with me.
> What i strive to see is the following:
>
> [Step 1] formalize what is the kind of work that needs a formal Proposal, I
> think Roman and Jihoon has already covered that pretty well. am +1 on that.
>
> https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E
>
> I am strongly in favor of the separation of Proposal Review and (later)
> Code review PRs. My  main reasons:
> Most importantly code reviewing will introduce lot of noise and will
> ultimately make  the GitHub page unreadable.
> Avoid overlapping of work.
> Once code is written hard to think abstract.
> Separate page for Design review later can always be used it as a Design
> document that is readable and code free-ish.
> As i said the goal of this first round is to see if the community agree
> about such change, then make the process of design more inclusive thus
> other contributors can submit a counter proposals.
>
> [Step 2] IF everybody agree about that point Step 2 is to define which
> medium is used to Publish a primitive form of a CODE FREE Abstract Proposal
> containing at least the following bullet points.
> - The problem description and motivation
> - Overview of the proposed change
> - Operational impact (compatibility/ plans to upgrades) public API changes,
> configuration changes, algorithm, and so on
> - Expected benefits and drawbacks
> - Rationale and alternatives
> - Estimate Time to Deliver if possible.
>
> The way i think this can be is a Github issue where member of the community
> will interact via comments and the author will be updating the description
> in the light of comments provided by the community.
>
> During and near the end of the design discussions the author/s can start
> writing POCs to help guide the review process this naturally will be a Pull
> request with actual code.
>
> *Now the most important thing is that we need to agree that any work that
> does not align with this formal process will be ignored and the author will
> be asked to start with a DIP*
> *That is what i meant with  “If it didn’t happen on the mailing list, it
> didn’t happen.”*
>
> Thanks and happy coding!
>
> On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <gi...@apache.org> wrote:
>
> > One of the advantages I see with a more formal process is (like Kafka
> KIPs)
> > is that it levels the playing field a bit and sets some ground rules for
> > working together. In a way it can help encourage contributions by making
> it
> > clear what is expected of potential contributors.
> >
> > We have a design review process today that is not as formal as KIPs, but
> is
> > somewhat heavier than the one you describe. Maybe we could tweak our
> > current one by starting to do design reviews separately from PRs. i.e.,
> for
> > anything that meets our 'design review' criteria, do that on the dev list
> > or in a separate issue, and keep the PR focused on code-level stuff. That
> > way we don't end up trying to do both at once. And it makes it easier to
> > start talking about design before the code is ready, which would be
> better.
> >
> > On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <jh...@apache.org> wrote:
> >
> > > It’s really hard to say no to a contribution when someone has put in a
> > > significant amount of work.
> > >
> > > The following approach is simple and works really well: Before you
> start
> > > work, log a case, describing the problem. When you have some ideas
> about
> > > design, add those to the case. When you have a code branch, add its URL
> > to
> > > the case. And so forth. At any point in the proceedings, people can
> chime
> > > in with their opinions.
> > >
> > > In my opinion, a formal “design review” process is not necessary. Just
> > > build consensus iteratively, by starting the conversation early in the
> > > process.
> > >
> > > Julian
> > >
> > >
> > > > On Jan 2, 2019, at 12:37 PM, Gian Merlino <gi...@apache.org> wrote:
> > > >
> > > > In this particular case: please consider the PR as a proposal. Don't
> > feel
> > > > like just because there is code there that takes a certain approach,
> > that
> > > > the approach is somehow sacred. I had to implement something to
> > > crystallize
> > > > my own thinking about how the problem could be approached. I won't be
> > > > disappointed if, as a community, we decide a different direction is
> > > better
> > > > and the code all gets thrown away. That's one of the reasons that I
> > > removed
> > > > the 0.14.0 milestone that was added to the patch. (I don't want to
> rush
> > > it,
> > > > nor do I think that's a good idea.)
> > > >
> > > > In general: Sounds like we could do with some more formalization
> around
> > > > what a proposal looks like, which sorts of changes need one, and when
> > in
> > > > the dev cycle it is appropriate. FWIW I think Kafka's process is more
> > or
> > > > less fine, and would be okay with adopting it for Druid if people
> like
> > > it.
> > > > Right now our standards for what requires a "design review" are very
> > > > similar to the Kafka community standards for what requires a KIP, so
> we
> > > > have some familiarity with those concepts. However we don't separate
> PR
> > > > review and proposal discussion as strictly as they do, which seems to
> > be
> > > > the foundation for the feeling of exclusion that is being felt here.
> > > >
> > > > Separately: I just redid the description on
> > > > https://github.com/apache/incubator-druid/pull/6794 to be more
> > > proposal-y.
> > > > I followed the KIP style:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > .
> > > > Please refresh the page and see if it looks more useful.
> > > >
> > > > Gian
> > > >
> > > > On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org>
> wrote:
> > > >
> > > >> Slim,
> > > >>
> > > >> I agree with your points that offline development is bad for
> > community.
> > > >> But I don’t think you need much mentor help. You have raised valid
> > > issues
> > > >> and the Druid community needs to decide what its development
> practices
> > > >> should be.
> > > >>
> > > >> Julian
> > > >>
> > > >>
> > > >>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <bs...@apache.org>
> > wrote:
> > > >>>
> > > >>> Hello everyone and hope you all have very good holidays.
> > > >>>
> > > >>> First, this email is not directed on the author or the PR
> > > >>> https://github.com/apache/incubator-druid/pull/6794  it self, but
> i
> > > see
> > > >>> this PR as a perfect example.
> > > >>>
> > > >>> One of the foundation of Apache Way or what i would simply call
> open
> > > >> source
> > > >>> community driven development is that "Technical decisions are
> > > discussed,
> > > >>> decided, and archived publicly.
> > > >>> developpement"
> > > >>> Which means that big technical  changes such as the one brought by
> > > #/6794
> > > >>> should have started as a proposal and round of discussions about
> the
> > > >> major
> > > >>> changes designs not as 11K line of code.
> > > >>> I believe such openness will promote a lot of good benefits such
> as:
> > > >>>
> > > >>> - ensures community health and growth.
> > > >>> - ensures everyone can participate not only the authors and his
> > > >> co-workers.
> > > >>> - ensures that the project is driven by the community and not a
> given
> > > >>> company or an individual.
> > > >>> - ensures that there is consensus (not saying 100% agreement;)
> > however
> > > it
> > > >>> means that all individuals will accept the current progress on the
> > > >> project
> > > >>> until some better proposal is put forth.
> > > >>>
> > > >>> Personally such BIG offline PR makes me feel excluded and doesn't
> > give
> > > >> me a
> > > >>> sense that i belong to  a community at all.
> > > >>>
> > > >>> To prevent such off list development i think as a Druid Community
> we
> > > need
> > > >>> to stick to the apache way “If it didn’t happen on the mailing
> list,
> > it
> > > >>> didn’t happen.”
> > > >>>
> > > >>> I would appreciate if some of the Apache mentor help with this.
> > > >>> Thanks
> > > >>
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > >> For additional commands, e-mail: dev-help@druid.apache.org
> > > >>
> > > >>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > For additional commands, e-mail: dev-help@druid.apache.org
> > >
> > >
> >
>

Re: Off list major development

[Slim Bouguerra <bs...@apache.org>]

Thanks everyone for interacting with this thread.

The fact that i, Roman, Jihoon  and others in the past (FJ
https://groups.google.com/forum/#!msg/druid-user/gkUEsAYIfBA/6B2GJdLkAgAJ)
raised this point indicates that PRs without a proposal are indeed an issue
and we need to solve it.

Something Similar to KIP maybe called DIPs is fine with me.
What i strive to see is the following:

[Step 1] formalize what is the kind of work that needs a formal Proposal, I
think Roman and Jihoon has already covered that pretty well. am +1 on that.
https://lists.apache.org/thread.html/9b30d893bdb6bb633cf6a9a700183ffb5b98f115330531a55328ac77@%3Cdev.druid.apache.org%3E

I am strongly in favor of the separation of Proposal Review and (later)
Code review PRs. My  main reasons:
Most importantly code reviewing will introduce lot of noise and will
ultimately make  the GitHub page unreadable.
Avoid overlapping of work.
Once code is written hard to think abstract.
Separate page for Design review later can always be used it as a Design
document that is readable and code free-ish.
As i said the goal of this first round is to see if the community agree
about such change, then make the process of design more inclusive thus
other contributors can submit a counter proposals.

[Step 2] IF everybody agree about that point Step 2 is to define which
medium is used to Publish a primitive form of a CODE FREE Abstract Proposal
containing at least the following bullet points.
- The problem description and motivation
- Overview of the proposed change
- Operational impact (compatibility/ plans to upgrades) public API changes,
configuration changes, algorithm, and so on
- Expected benefits and drawbacks
- Rationale and alternatives
- Estimate Time to Deliver if possible.

The way i think this can be is a Github issue where member of the community
will interact via comments and the author will be updating the description
in the light of comments provided by the community.

During and near the end of the design discussions the author/s can start
writing POCs to help guide the review process this naturally will be a Pull
request with actual code.

*Now the most important thing is that we need to agree that any work that
does not align with this formal process will be ignored and the author will
be asked to start with a DIP*
*That is what i meant with  “If it didn’t happen on the mailing list, it
didn’t happen.”*

Thanks and happy coding!

On Wed, Jan 2, 2019 at 6:47 PM Gian Merlino <gi...@apache.org> wrote:

> One of the advantages I see with a more formal process is (like Kafka KIPs)
> is that it levels the playing field a bit and sets some ground rules for
> working together. In a way it can help encourage contributions by making it
> clear what is expected of potential contributors.
>
> We have a design review process today that is not as formal as KIPs, but is
> somewhat heavier than the one you describe. Maybe we could tweak our
> current one by starting to do design reviews separately from PRs. i.e., for
> anything that meets our 'design review' criteria, do that on the dev list
> or in a separate issue, and keep the PR focused on code-level stuff. That
> way we don't end up trying to do both at once. And it makes it easier to
> start talking about design before the code is ready, which would be better.
>
> On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <jh...@apache.org> wrote:
>
> > It’s really hard to say no to a contribution when someone has put in a
> > significant amount of work.
> >
> > The following approach is simple and works really well: Before you start
> > work, log a case, describing the problem. When you have some ideas about
> > design, add those to the case. When you have a code branch, add its URL
> to
> > the case. And so forth. At any point in the proceedings, people can chime
> > in with their opinions.
> >
> > In my opinion, a formal “design review” process is not necessary. Just
> > build consensus iteratively, by starting the conversation early in the
> > process.
> >
> > Julian
> >
> >
> > > On Jan 2, 2019, at 12:37 PM, Gian Merlino <gi...@apache.org> wrote:
> > >
> > > In this particular case: please consider the PR as a proposal. Don't
> feel
> > > like just because there is code there that takes a certain approach,
> that
> > > the approach is somehow sacred. I had to implement something to
> > crystallize
> > > my own thinking about how the problem could be approached. I won't be
> > > disappointed if, as a community, we decide a different direction is
> > better
> > > and the code all gets thrown away. That's one of the reasons that I
> > removed
> > > the 0.14.0 milestone that was added to the patch. (I don't want to rush
> > it,
> > > nor do I think that's a good idea.)
> > >
> > > In general: Sounds like we could do with some more formalization around
> > > what a proposal looks like, which sorts of changes need one, and when
> in
> > > the dev cycle it is appropriate. FWIW I think Kafka's process is more
> or
> > > less fine, and would be okay with adopting it for Druid if people like
> > it.
> > > Right now our standards for what requires a "design review" are very
> > > similar to the Kafka community standards for what requires a KIP, so we
> > > have some familiarity with those concepts. However we don't separate PR
> > > review and proposal discussion as strictly as they do, which seems to
> be
> > > the foundation for the feeling of exclusion that is being felt here.
> > >
> > > Separately: I just redid the description on
> > > https://github.com/apache/incubator-druid/pull/6794 to be more
> > proposal-y.
> > > I followed the KIP style:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > .
> > > Please refresh the page and see if it looks more useful.
> > >
> > > Gian
> > >
> > > On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org> wrote:
> > >
> > >> Slim,
> > >>
> > >> I agree with your points that offline development is bad for
> community.
> > >> But I don’t think you need much mentor help. You have raised valid
> > issues
> > >> and the Druid community needs to decide what its development practices
> > >> should be.
> > >>
> > >> Julian
> > >>
> > >>
> > >>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <bs...@apache.org>
> wrote:
> > >>>
> > >>> Hello everyone and hope you all have very good holidays.
> > >>>
> > >>> First, this email is not directed on the author or the PR
> > >>> https://github.com/apache/incubator-druid/pull/6794  it self, but i
> > see
> > >>> this PR as a perfect example.
> > >>>
> > >>> One of the foundation of Apache Way or what i would simply call open
> > >> source
> > >>> community driven development is that "Technical decisions are
> > discussed,
> > >>> decided, and archived publicly.
> > >>> developpement"
> > >>> Which means that big technical  changes such as the one brought by
> > #/6794
> > >>> should have started as a proposal and round of discussions about the
> > >> major
> > >>> changes designs not as 11K line of code.
> > >>> I believe such openness will promote a lot of good benefits such as:
> > >>>
> > >>> - ensures community health and growth.
> > >>> - ensures everyone can participate not only the authors and his
> > >> co-workers.
> > >>> - ensures that the project is driven by the community and not a given
> > >>> company or an individual.
> > >>> - ensures that there is consensus (not saying 100% agreement;)
> however
> > it
> > >>> means that all individuals will accept the current progress on the
> > >> project
> > >>> until some better proposal is put forth.
> > >>>
> > >>> Personally such BIG offline PR makes me feel excluded and doesn't
> give
> > >> me a
> > >>> sense that i belong to  a community at all.
> > >>>
> > >>> To prevent such off list development i think as a Druid Community we
> > need
> > >>> to stick to the apache way “If it didn’t happen on the mailing list,
> it
> > >>> didn’t happen.”
> > >>>
> > >>> I would appreciate if some of the Apache mentor help with this.
> > >>> Thanks
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > >> For additional commands, e-mail: dev-help@druid.apache.org
> > >>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > For additional commands, e-mail: dev-help@druid.apache.org
> >
> >
>

Re: Off list major development

[Gian Merlino <gi...@apache.org>]

One of the advantages I see with a more formal process is (like Kafka KIPs)
is that it levels the playing field a bit and sets some ground rules for
working together. In a way it can help encourage contributions by making it
clear what is expected of potential contributors.

We have a design review process today that is not as formal as KIPs, but is
somewhat heavier than the one you describe. Maybe we could tweak our
current one by starting to do design reviews separately from PRs. i.e., for
anything that meets our 'design review' criteria, do that on the dev list
or in a separate issue, and keep the PR focused on code-level stuff. That
way we don't end up trying to do both at once. And it makes it easier to
start talking about design before the code is ready, which would be better.

On Wed, Jan 2, 2019 at 6:10 PM Julian Hyde <jh...@apache.org> wrote:

> It’s really hard to say no to a contribution when someone has put in a
> significant amount of work.
>
> The following approach is simple and works really well: Before you start
> work, log a case, describing the problem. When you have some ideas about
> design, add those to the case. When you have a code branch, add its URL to
> the case. And so forth. At any point in the proceedings, people can chime
> in with their opinions.
>
> In my opinion, a formal “design review” process is not necessary. Just
> build consensus iteratively, by starting the conversation early in the
> process.
>
> Julian
>
>
> > On Jan 2, 2019, at 12:37 PM, Gian Merlino <gi...@apache.org> wrote:
> >
> > In this particular case: please consider the PR as a proposal. Don't feel
> > like just because there is code there that takes a certain approach, that
> > the approach is somehow sacred. I had to implement something to
> crystallize
> > my own thinking about how the problem could be approached. I won't be
> > disappointed if, as a community, we decide a different direction is
> better
> > and the code all gets thrown away. That's one of the reasons that I
> removed
> > the 0.14.0 milestone that was added to the patch. (I don't want to rush
> it,
> > nor do I think that's a good idea.)
> >
> > In general: Sounds like we could do with some more formalization around
> > what a proposal looks like, which sorts of changes need one, and when in
> > the dev cycle it is appropriate. FWIW I think Kafka's process is more or
> > less fine, and would be okay with adopting it for Druid if people like
> it.
> > Right now our standards for what requires a "design review" are very
> > similar to the Kafka community standards for what requires a KIP, so we
> > have some familiarity with those concepts. However we don't separate PR
> > review and proposal discussion as strictly as they do, which seems to be
> > the foundation for the feeling of exclusion that is being felt here.
> >
> > Separately: I just redid the description on
> > https://github.com/apache/incubator-druid/pull/6794 to be more
> proposal-y.
> > I followed the KIP style:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> .
> > Please refresh the page and see if it looks more useful.
> >
> > Gian
> >
> > On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org> wrote:
> >
> >> Slim,
> >>
> >> I agree with your points that offline development is bad for community.
> >> But I don’t think you need much mentor help. You have raised valid
> issues
> >> and the Druid community needs to decide what its development practices
> >> should be.
> >>
> >> Julian
> >>
> >>
> >>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <bs...@apache.org> wrote:
> >>>
> >>> Hello everyone and hope you all have very good holidays.
> >>>
> >>> First, this email is not directed on the author or the PR
> >>> https://github.com/apache/incubator-druid/pull/6794  it self, but i
> see
> >>> this PR as a perfect example.
> >>>
> >>> One of the foundation of Apache Way or what i would simply call open
> >> source
> >>> community driven development is that "Technical decisions are
> discussed,
> >>> decided, and archived publicly.
> >>> developpement"
> >>> Which means that big technical  changes such as the one brought by
> #/6794
> >>> should have started as a proposal and round of discussions about the
> >> major
> >>> changes designs not as 11K line of code.
> >>> I believe such openness will promote a lot of good benefits such as:
> >>>
> >>> - ensures community health and growth.
> >>> - ensures everyone can participate not only the authors and his
> >> co-workers.
> >>> - ensures that the project is driven by the community and not a given
> >>> company or an individual.
> >>> - ensures that there is consensus (not saying 100% agreement;) however
> it
> >>> means that all individuals will accept the current progress on the
> >> project
> >>> until some better proposal is put forth.
> >>>
> >>> Personally such BIG offline PR makes me feel excluded and doesn't give
> >> me a
> >>> sense that i belong to  a community at all.
> >>>
> >>> To prevent such off list development i think as a Druid Community we
> need
> >>> to stick to the apache way “If it didn’t happen on the mailing list, it
> >>> didn’t happen.”
> >>>
> >>> I would appreciate if some of the Apache mentor help with this.
> >>> Thanks
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> >> For additional commands, e-mail: dev-help@druid.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> For additional commands, e-mail: dev-help@druid.apache.org
>
>

Re: Off list major development

[Julian Hyde <jh...@apache.org>]

It’s really hard to say no to a contribution when someone has put in a significant amount of work. 

The following approach is simple and works really well: Before you start work, log a case, describing the problem. When you have some ideas about design, add those to the case. When you have a code branch, add its URL to the case. And so forth. At any point in the proceedings, people can chime in with their opinions.

In my opinion, a formal “design review” process is not necessary. Just build consensus iteratively, by starting the conversation early in the process.

Julian


> On Jan 2, 2019, at 12:37 PM, Gian Merlino <gi...@apache.org> wrote:
> 
> In this particular case: please consider the PR as a proposal. Don't feel
> like just because there is code there that takes a certain approach, that
> the approach is somehow sacred. I had to implement something to crystallize
> my own thinking about how the problem could be approached. I won't be
> disappointed if, as a community, we decide a different direction is better
> and the code all gets thrown away. That's one of the reasons that I removed
> the 0.14.0 milestone that was added to the patch. (I don't want to rush it,
> nor do I think that's a good idea.)
> 
> In general: Sounds like we could do with some more formalization around
> what a proposal looks like, which sorts of changes need one, and when in
> the dev cycle it is appropriate. FWIW I think Kafka's process is more or
> less fine, and would be okay with adopting it for Druid if people like it.
> Right now our standards for what requires a "design review" are very
> similar to the Kafka community standards for what requires a KIP, so we
> have some familiarity with those concepts. However we don't separate PR
> review and proposal discussion as strictly as they do, which seems to be
> the foundation for the feeling of exclusion that is being felt here.
> 
> Separately: I just redid the description on
> https://github.com/apache/incubator-druid/pull/6794 to be more proposal-y.
> I followed the KIP style:
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals.
> Please refresh the page and see if it looks more useful.
> 
> Gian
> 
> On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org> wrote:
> 
>> Slim,
>> 
>> I agree with your points that offline development is bad for community.
>> But I don’t think you need much mentor help. You have raised valid issues
>> and the Druid community needs to decide what its development practices
>> should be.
>> 
>> Julian
>> 
>> 
>>> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <bs...@apache.org> wrote:
>>> 
>>> Hello everyone and hope you all have very good holidays.
>>> 
>>> First, this email is not directed on the author or the PR
>>> https://github.com/apache/incubator-druid/pull/6794  it self, but i see
>>> this PR as a perfect example.
>>> 
>>> One of the foundation of Apache Way or what i would simply call open
>> source
>>> community driven development is that "Technical decisions are discussed,
>>> decided, and archived publicly.
>>> developpement"
>>> Which means that big technical  changes such as the one brought by #/6794
>>> should have started as a proposal and round of discussions about the
>> major
>>> changes designs not as 11K line of code.
>>> I believe such openness will promote a lot of good benefits such as:
>>> 
>>> - ensures community health and growth.
>>> - ensures everyone can participate not only the authors and his
>> co-workers.
>>> - ensures that the project is driven by the community and not a given
>>> company or an individual.
>>> - ensures that there is consensus (not saying 100% agreement;) however it
>>> means that all individuals will accept the current progress on the
>> project
>>> until some better proposal is put forth.
>>> 
>>> Personally such BIG offline PR makes me feel excluded and doesn't give
>> me a
>>> sense that i belong to  a community at all.
>>> 
>>> To prevent such off list development i think as a Druid Community we need
>>> to stick to the apache way “If it didn’t happen on the mailing list, it
>>> didn’t happen.”
>>> 
>>> I would appreciate if some of the Apache mentor help with this.
>>> Thanks
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>> For additional commands, e-mail: dev-help@druid.apache.org
>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org

Re: Off list major development

[Gian Merlino <gi...@apache.org>]

In this particular case: please consider the PR as a proposal. Don't feel
like just because there is code there that takes a certain approach, that
the approach is somehow sacred. I had to implement something to crystallize
my own thinking about how the problem could be approached. I won't be
disappointed if, as a community, we decide a different direction is better
and the code all gets thrown away. That's one of the reasons that I removed
the 0.14.0 milestone that was added to the patch. (I don't want to rush it,
nor do I think that's a good idea.)

In general: Sounds like we could do with some more formalization around
what a proposal looks like, which sorts of changes need one, and when in
the dev cycle it is appropriate. FWIW I think Kafka's process is more or
less fine, and would be okay with adopting it for Druid if people like it.
Right now our standards for what requires a "design review" are very
similar to the Kafka community standards for what requires a KIP, so we
have some familiarity with those concepts. However we don't separate PR
review and proposal discussion as strictly as they do, which seems to be
the foundation for the feeling of exclusion that is being felt here.

Separately: I just redid the description on
https://github.com/apache/incubator-druid/pull/6794 to be more proposal-y.
I followed the KIP style:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals.
Please refresh the page and see if it looks more useful.

Gian

On Wed, Jan 2, 2019 at 10:52 AM Julian Hyde <jh...@apache.org> wrote:

> Slim,
>
> I agree with your points that offline development is bad for community.
> But I don’t think you need much mentor help. You have raised valid issues
> and the Druid community needs to decide what its development practices
> should be.
>
> Julian
>
>
> > On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <bs...@apache.org> wrote:
> >
> > Hello everyone and hope you all have very good holidays.
> >
> > First, this email is not directed on the author or the PR
> > https://github.com/apache/incubator-druid/pull/6794  it self, but i see
> > this PR as a perfect example.
> >
> > One of the foundation of Apache Way or what i would simply call open
> source
> > community driven development is that "Technical decisions are discussed,
> > decided, and archived publicly.
> > developpement"
> > Which means that big technical  changes such as the one brought by #/6794
> > should have started as a proposal and round of discussions about the
> major
> > changes designs not as 11K line of code.
> > I believe such openness will promote a lot of good benefits such as:
> >
> > - ensures community health and growth.
> > - ensures everyone can participate not only the authors and his
> co-workers.
> > - ensures that the project is driven by the community and not a given
> > company or an individual.
> > - ensures that there is consensus (not saying 100% agreement;) however it
> > means that all individuals will accept the current progress on the
> project
> > until some better proposal is put forth.
> >
> > Personally such BIG offline PR makes me feel excluded and doesn't give
> me a
> > sense that i belong to  a community at all.
> >
> > To prevent such off list development i think as a Druid Community we need
> > to stick to the apache way “If it didn’t happen on the mailing list, it
> > didn’t happen.”
> >
> > I would appreciate if some of the Apache mentor help with this.
> > Thanks
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> For additional commands, e-mail: dev-help@druid.apache.org
>
>

Re: Off list major development

[Julian Hyde <jh...@apache.org>]

Slim,

I agree with your points that offline development is bad for community. But I don’t think you need much mentor help. You have raised valid issues and the Druid community needs to decide what its development practices should be.

Julian


> On Jan 2, 2019, at 10:29 AM, Slim Bouguerra <bs...@apache.org> wrote:
> 
> Hello everyone and hope you all have very good holidays.
> 
> First, this email is not directed on the author or the PR
> https://github.com/apache/incubator-druid/pull/6794  it self, but i see
> this PR as a perfect example.
> 
> One of the foundation of Apache Way or what i would simply call open source
> community driven development is that "Technical decisions are discussed,
> decided, and archived publicly.
> developpement"
> Which means that big technical  changes such as the one brought by #/6794
> should have started as a proposal and round of discussions about the major
> changes designs not as 11K line of code.
> I believe such openness will promote a lot of good benefits such as:
> 
> - ensures community health and growth.
> - ensures everyone can participate not only the authors and his co-workers.
> - ensures that the project is driven by the community and not a given
> company or an individual.
> - ensures that there is consensus (not saying 100% agreement;) however it
> means that all individuals will accept the current progress on the project
> until some better proposal is put forth.
> 
> Personally such BIG offline PR makes me feel excluded and doesn't give me a
> sense that i belong to  a community at all.
> 
> To prevent such off list development i think as a Druid Community we need
> to stick to the apache way “If it didn’t happen on the mailing list, it
> didn’t happen.”
> 
> I would appreciate if some of the Apache mentor help with this.
> Thanks


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org